HashiCorp Vault: bootstrap, unseal, and Kubernetes auth

Published: 2026-02-10

Setting up Vault for the first time in a Kubernetes cluster is not just helm install. There's an initialization ceremony, an unsealing strategy, and a Kubernetes auth method to configure before External Secrets Operator can pull anything.

This post covers the full bootstrap process: HA Raft deployment, the unseal ceremony, Kubernetes auth setup, policies, ServiceAccount bindings, and the operational patterns for day-2 operations.

Why Vault over Kubernetes Secrets + SealedSecrets

Kubernetes Secrets are base64-encoded, not encrypted at rest by default (unless you've configured etcd encryption). SealedSecrets solve the "encrypt before committing to Git" problem but don't solve:

• Secret rotation — you need to re-seal and redeploy every time a password changes
• Audit trail — who read which secret, when
• Dynamic secrets — database credentials valid for 1 hour, automatically revoked after use
• Fine-grained access control — which pod can read which secret path

Vault addresses all of these. The operational overhead of running Vault is real (unsealing, key management), but for production environments with multiple teams and compliance requirements, it's worth it.

Deployment

Vault runs as a StatefulSet via the official HashiCorp Helm chart. HA mode with integrated Raft storage — no external Consul dependency:

yamlcopyserver:
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      config: |
        ui = true
        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }
        storage "raft" {
          path = "/vault/data"
          retry_join {
            leader_api_addr = "http://vault-0.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-1.vault-internal:8200"
          }
          retry_join {
            leader_api_addr = "http://vault-2.vault-internal:8200"
          }
        }
        service_registration "kubernetes" {}
  dataStorage:
    storageClass: local-path
    size: 10Gi

tls_disable = 1 in the listener — TLS termination is done by the ingress (Traefik/APISIX), not Vault itself. In a production deployment with direct pod-to-pod communication you'd enable TLS here.

service_registration "kubernetes" {} allows Vault to register itself as a Kubernetes service. When the active Vault node changes (leader election), Kubernetes routes traffic correctly.

retry_join — all three Raft nodes try to join each other at startup. Only one will become leader, others join as followers. If you restart vault-0, it rejoins automatically.

Storage sizing

10Gi is adequate for secret storage but grow quickly with audit logging enabled. Enable audit to a separate volume:

yamlcopy# In your Vault config
audit_backend "file" {
  file_path = "/vault/audit/audit.log"
}

Initialization

After the pods start, only one node needs initializing. All other pods will join the Raft cluster automatically:

bashcopykubectl exec -n vault vault-0 -- vault operator init \
  -key-shares=5 \
  -key-threshold=3 \
  -format=json > vault-init.json

This produces 5 unseal keys and a root token. The threshold of 3 means any 3 of the 5 keys can unseal.

Key security: Store the root token offline — it's only needed for initial setup and emergency access. The unseal keys should be distributed: in a serious deployment, each key goes to a different person/team, so unsealing requires consensus.

Key storage options

• Password manager with 5 separate holders — most secure for multi-team environments
• SealedSecret in the infra repo — convenient for single-team setups; the infra cluster's Sealed Secrets key protects Vault's unseal keys (recursive protection)
• YC KMS / AWS KMS auto-unseal — fully automated, removes the human ceremony at restart cost

For single-team homelab/startup environments, storing in a SealedSecret is pragmatic. For anything regulated (SOC2, PCI, GDPR audits), you want the distributed key ceremony.

Unsealing

Each Vault pod must be unsealed independently after every restart. Vault starts in sealed state and refuses all operations until unsealed:

bashcopyfor pod in vault-0 vault-1 vault-2; do
  for key in $(cat vault-init.json | jq -r '.unseal_keys_b64[:3][]'); do
    kubectl exec -n vault $pod -- vault operator unseal $key
  done
done

Check unseal status:

bashcopykubectl exec -n vault vault-0 -- vault status

Look for Sealed: false and HA Enabled: true. The active node shows HA Mode: active, others show HA Mode: standby.

Auto-unseal with Yandex Cloud KMS

For automated unsealing (eliminating the manual ceremony after each restart):

hclcopyseal "transit" {
  address = "https://kms.yandex"
  key_name = "vault-unseal-key"
  # Uses instance metadata for auth in YC
}

With auto-unseal, Vault pods unseal automatically on restart. The tradeoff: if your KMS key is deleted or rotated incorrectly, all Vault data becomes permanently inaccessible. Maintain a migration seal for emergency recovery.

Auto-unseal with AWS KMS

hclcopyseal "awskms" {
  region     = "us-east-1"
  kms_key_id = "alias/vault-unseal"
}

Configuring Kubernetes auth

After unsealing, login with the root token and set up Kubernetes auth so External Secrets Operator can authenticate:

bashcopyexport VAULT_ADDR="http://localhost:8200"  # port-forward or ingress
export VAULT_TOKEN=$(cat vault-init.json | jq -r '.root_token')

vault auth enable kubernetes

vault write auth/kubernetes/config \
  kubernetes_host="https://$KUBE_HOST" \
  token_reviewer_jwt="$(kubectl get secret -n vault vault-sa-token -o jsonpath='{.data.token}' | base64 -d)" \
  kubernetes_ca_cert="$(kubectl config view --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d)"

kubernetes_host — the API server URL. Inside the cluster: https://kubernetes.default.svc.

token_reviewer_jwt — a ServiceAccount token that Vault uses to call the Kubernetes TokenReview API to validate incoming JWT tokens from pods. Create a dedicated ServiceAccount for this:

bashcopykubectl create serviceaccount vault-auth -n vault
kubectl create clusterrolebinding vault-auth-binding \
  --clusterrole=system:auth-delegator \
  --serviceaccount=vault:vault-auth

The system:auth-delegator ClusterRole gives the ServiceAccount permission to call TokenReview, which is exactly what Vault needs.

Verify Kubernetes auth is configured

bashcopyvault auth list
# Should show kubernetes/ in the list

vault read auth/kubernetes/config

KV engine and policies

Enable KV v2 (supports versioning, soft-delete, metadata):

bashcopyvault secrets enable -path=secret kv-v2

Create a policy for the dev environment:

hclcopy# dev-read.hcl
path "secret/data/dev/*" {
  capabilities = ["read"]
}
path "secret/metadata/dev/*" {
  capabilities = ["read", "list"]
}

bashcopyvault policy write dev-read dev-read.hcl

The secret/data/dev/* pattern allows reading any secret under dev/. Use more specific paths if you need per-service isolation:

hclcopy# eso-postgres.hcl — only postgres secrets for the payments team
path "secret/data/dev/postgres" {
  capabilities = ["read"]
}

Policy for multiple namespaces

If you have separate teams using separate Kubernetes namespaces, create separate policies and separate roles:

bashcopy# Prod policy — same paths, more restrictive (no list)
vault policy write prod-read - <<EOF
path "secret/data/prod/*" {
  capabilities = ["read"]
}
EOF

Binding a Kubernetes ServiceAccount to the policy

bashcopyvault write auth/kubernetes/role/eso-dev \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets \
  policies=dev-read \
  ttl=1h

The external-secrets ServiceAccount in the external-secrets namespace (created by the ESO chart) gets a 1-hour token with dev-read permissions. ESO uses this to fetch secrets for all ExternalSecret objects.

ttl=1h means the Vault token ESO receives expires after one hour. ESO renews it automatically before expiry. If the token expires (e.g., Vault was unreachable during renewal), ESO logs an authentication error and stops syncing until it re-authenticates.

Multiple roles for multiple clusters

In a hub-and-spoke setup, ESO on the hub cluster might need to fetch secrets for multiple environments. Create separate roles:

bashcopyvault write auth/kubernetes/role/eso-infra \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets \
  policies=infra-read \
  ttl=1h

vault write auth/kubernetes/role/eso-dev \
  bound_service_account_names=external-secrets \
  bound_service_account_namespaces=external-secrets \
  policies=dev-read \
  ttl=1h

The roles are differentiated by the policies they grant, not by the ServiceAccount (which is the same ESO SA in both cases).

Adding secrets

bashcopyvault kv put secret/dev/postgres \
  username=app \
  password=s3cr3t

vault kv put secret/dev/gitlab-registry \
  dockerconfig='{"auths":{"registry.example.com":{"auth":"..."}}}'

Versioning

KV v2 keeps up to 10 versions by default. Roll back to a previous version:

bashcopy# See versions
vault kv metadata get secret/dev/postgres

# Roll back to version 2
vault kv rollback -version=2 secret/dev/postgres

Checking what ESO actually pulls

If ESO pulls a secret and the result looks wrong, check the secret value directly:

bashcopyvault kv get -format=json secret/dev/postgres | jq '.data.data'

Note the double .data.data — in KV v2, the actual values are nested under .data.data in the JSON response.

Day-2 operations

Rotating the root token

After initial setup, revoke the root token and generate a new one only when needed:

bashcopyvault token revoke $ROOT_TOKEN

# Generate a new root token when needed (requires unseal quorum)
vault operator generate-root -init
# ... ceremony with unseal key holders ...

Audit logging

Enable audit log before any production use:

bashcopyvault audit enable file file_path=/vault/audit/audit.log

Audit log records every request and response (with secrets redacted). Essential for compliance and incident investigation.

Monitoring Vault with Prometheus

The official Vault Helm chart includes a serviceMonitor for Prometheus:

yamlcopyserver:
  serviceAccount:
    create: true
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      labels:
        release: kube-prom-stack

Key metrics to alert on:

• vault_core_unsealed — 0 means Vault is sealed, all secret reads will fail
• vault_core_active — number of active (leader) nodes, should be 1
• vault_runtime_total_gc_pause_ns — high GC pause times indicate memory pressure