Terraform Operator: running Terraform apply from inside Kubernetes

Published: 2026-04-24

Yandex Cloud infrastructure (VMs, managed Kubernetes clusters, VPCs) is managed by Terraform. Running terraform apply from a GitLab CI job works, but requires passing credentials as variables and managing state externally. The GalleyBytes terraform-operator runs Terraform as a Kubernetes workload — the state, credentials, and execution all live in the cluster.

The operator

GalleyBytes terraform-operator defines a Terraform CRD. Each object describes a Terraform module to run: the source git repo, the backend config, and how often to reconcile.

It's deployed from custom/apps/terraform-operator/ via a HelmRelease on the infra cluster.

Install:

bashcopykubectl apply -f https://raw.githubusercontent.com/GalleyBytes/terraform-operator/master/deploy/bundles/crd-bundle.yaml

Or as a HelmRelease via Flux:

yamlcopyapiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: terraform-operator
  namespace: terraform-operator
spec:
  chart:
    spec:
      chart: terraform-operator
      sourceRef:
        kind: HelmRepository
        name: galleybytes
      version: ">=0.1.0"
  interval: 1h

The Terraform CRD object

yamlcopyapiVersion: tf.galleybytes.com/v1beta1
kind: Terraform
metadata:
  name: infra-terraform
  namespace: terraform-operator
spec:
  terraformVersion: "1.5.5"
  terraformModule:
    source: "git::ssh://git@gitlab.example.com/docker/k8s/infra.git//terraform/infra?ref=main"
  scmAuthMethods:
    - host: gitlab.example.com
      git:
        ssh:
          sshKeySecretRef:
            name: flux-system
            namespace: flux-system
            key: identity
  backend: |
    terraform {
      backend "local" {}
    }
  storageClassName: local-path
  keepLatestPodsOnly: true
  writeOutputsToStatus: true
  env:
    - name: YC_TOKEN
      valueFrom:
        secretKeyRef:
          name: yc-token
          key: YC_TOKEN

source — the module comes from the same infra git repo, path terraform/infra, branch main. The operator clones it before each run.

scmAuthMethods — reuses the flux-system SSH key secret that Flux creates during bootstrap. No separate credential management needed.

backend: local — Terraform state is stored on a PVC (created by the operator using storageClassName: local-path). Not in S3 or Terraform Cloud. For production use, switch to backend "s3" pointing to an object storage bucket for state durability.

writeOutputsToStatus: true — Terraform outputs (cluster endpoint IDs, VPC IDs) are written to the Terraform object's .status field, making them available to other Kubernetes controllers via kubectl get or ExternalSecrets.

Vault token for Terraform

The Yandex Cloud provider requires a token. Stored as a SealedSecret:

yamlcopyapiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: yc-token
  namespace: terraform-operator
spec:
  encryptedData:
    YC_TOKEN: AgBx...

Referenced in the Terraform object as an env var for the runner pod.

RBAC

The operator needs RBAC to create pods and PVCs:

yamlcopyapiVersion: v1
kind: ServiceAccount
metadata:
  name: terraform-operator
  namespace: terraform-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: terraform-operator
rules:
  - apiGroups: ["tf.galleybytes.com"]
    resources: ["*"]
    verbs: ["*"]
  - apiGroups: [""]
    resources: ["pods", "pods/log", "persistentvolumeclaims", "secrets", "serviceaccounts"]
    verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: terraform-operator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: terraform-operator
subjects:
  - kind: ServiceAccount
    name: terraform-operator
    namespace: terraform-operator

Terraform directory structure

copyterraform/
├── infra/    ← infra cluster YC resources (VMs, LB, VPC)
├── dev/      ← dev cluster
├── test/     ← test cluster
├── sre/      ← sre YC cluster
├── loadgds/  ← loadgds YC cluster
└── demo/     ← demo YC cluster

Each directory is an independent Terraform root module. A separate Terraform CRD object manages each one independently.

Reconcile behavior

The operator runs terraform plan and terraform apply on a schedule (default: every 5 minutes). If the plan shows no changes — no-op. If it detects drift (from manual edits or new resources in .tf files) — it applies automatically.

keepLatestPodsOnly: true — the runner pod from each reconcile is kept for log inspection, but only the most recent one. Old pods are cleaned up automatically.

To trigger an immediate reconcile:

bashcopykubectl annotate terraform infra-terraform \
  -n terraform-operator \
  tf.galleybytes.com/sync="true" \
  --overwrite \
  --context=infra-k8s

Viewing status

bashcopy# Terraform object status
kubectl get terraform -n terraform-operator --context=infra-k8s

# Detailed status with conditions
kubectl describe terraform infra-terraform -n terraform-operator --context=infra-k8s

# Runner pod logs for the last apply
kubectl logs \
  -n terraform-operator \
  -l terraform.galleybytes.com/name=infra-terraform \
  --context=infra-k8s

# Terraform outputs written to status
kubectl get terraform infra-terraform \
  -n terraform-operator \
  -o jsonpath='{.status.outputs}' \
  --context=infra-k8s | jq .

Comparison: CI job vs operator

The operator is better suited for infrastructure that rarely changes but needs drift detection. CI pipelines are better for infrastructure that changes on every PR with approval gates.

Troubleshooting

Runner pod stuck in Init state — check operator logs:

bashcopykubectl logs -n terraform-operator -l app=terraform-operator

git clone fails — SSH key not found or wrong namespace. Verify:

bashcopykubectl get secret flux-system -n flux-system -o yaml

State lock error — a previous runner pod died mid-apply and left a lock. Delete the state lock file in the PVC or force-unlock:

bashcopy# Exec into a debug pod with the PVC mounted
terraform force-unlock <LOCK_ID>

Summary

• Terraform Operator runs terraform apply as a Kubernetes workload — state, credentials, and execution all live in the cluster
• Workspaces (Workspace CRD) define the git source, Vault credentials, and variable overrides per-environment
• State is stored in a PVC mounted to the runner pod, not in a remote backend — keep the PVC on a reliable StorageClass
• A failed runner pod mid-apply leaves a state lock; use terraform force-unlock to recover
• Operator fits scenarios where you want GitOps-native Terraform without giving CI runners direct access to cloud credentials