consul-template in the GitLab CI deploy job

Published: 2026-05-22

consul-template is usually described as a daemon that watches Vault and rewrites config files in real time. We use it differently: as a one-shot renderer during helm upgrade. The pattern is: pull the chart from Artifactory, render secrets from Vault into the chart's config/ directory using HCL templates, then run helm upgrade so the rendered files end up in ConfigMaps and Secrets inside the cluster.

Why not ESO or Vault Agent?

For this project ESO wasn't available at the time we set this up. Vault Agent sidecar injects secrets at pod startup — fine for long-running services but adds a sidecar to every pod. The CI-side rendering approach keeps the cluster dumb: the Helm chart just consumes files from config/; no Vault dependency at runtime.

The tradeoff is that secrets are rendered into the chart during deploy and baked into ConfigMaps (for non-sensitive config) or Kubernetes Secrets (for credentials). If a Vault value changes, you need to re-run the pipeline to pick it up.

Chart layout

Every application chart has this structure:

my-service/
  Chart.yaml
  values.yaml
  values-Development.yaml
  values-Production.yaml
  config/
    appsettings.json        # consul-template HCL — rendered at deploy time
    appsettings.ENV.json    # per-environment overrides
  crt/
    YC-CA.pem               # rendered from Vault
  files/
    config.hcl              # consul-template source template for appsettings.json
    config-user.hcl         # per-env overrides template
    config-crt.hcl          # CA certificate template
  templates/
    configmap.yaml
    deployment.yaml
    ...

The config/ and crt/ files are outputs — they don't exist in git, they are created by consul-template -once during the CI job and then embedded into ConfigMaps by helm upgrade.

HCL templates

files/config.hcl is a consul-template file that reads from Vault KV v2 and renders valid JSON. The double-curly syntax is consul-template, not Go templates:

{{ with $env := env "APP_ENV" }}
{{ $prj := "my-service" }}
{
  "ConnectionStrings": {
    "DefaultConnection": "{{ $path := printf "Microservices/%s/%s/system" $prj $env }}{{ with secret $path }}{{ .Data.data.connection_string }}{{ end }}"
  },
  "RabbitMq": {
    "Host":     "{{ with secret (printf "common-secret/%s/" $env) }}{{ .Data.data.rabbit_hostname }}{{ end }}",
    "Port":     "{{ with secret (printf "common-secret/%s/" $env) }}{{ .Data.data.rabbit_port }}{{ end }}",
    "Username": "{{ with secret (printf "common-secret/%s/" $env) }}{{ .Data.data.rabbit_username }}{{ end }}",
    "Password": "{{ with secret (printf "common-secret/%s/" $env) }}{{ .Data.data.rabbit_password }}{{ end }}"
  },
  "Logging": {
    "MinimumLevel": {
      {{- if eq $env "Production" }}
      "Default": "Warning"
      {{- else }}
      "Default": "Debug"
      {{- end }}
    }
  }
}
{{ end }}

Two Vault path patterns in use:

Microservices/<service>/<env>/system — service-specific secrets (DB URL, API keys)
common-secret/<env>/ — shared infrastructure (RabbitMQ, Elastic, APM endpoint)

files/config-user.hcl dumps the entire user KV secret as pretty JSON:

{{ with $env := env "APP_ENV" }}
{{ $prj := "my-service" }}
{{ $path := printf "Microservices/%s/%s/user" $prj $env }}{{ with secret $path }}{{ .Data.data | toJSONPretty }}{{ end }}
{{ end }}

files/config-crt.hcl pulls the YC root CA from the shared Vault path:

{{ with $env := env "APP_ENV" }}
{{ with secret (printf "common-secret/%s/" $env) }}{{ .Data.data.yc_ca }}{{ end }}
{{ end }}

Deploy job

yaml.deploy:
  image: registry.example.com/ci-images/vault-cli:1.0.1
  stage: deploy
  script:
    # 1. Pull chart from Helm registry
    - helm repo add charts https://artifacts.example.com/repository/helm/
    - helm repo update
    - helm fetch charts/${CHART_NAME} --untar

    # 2. Authenticate to Vault via GitLab JWT
    - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login
        role=access-${APP_ENV} jwt=$CI_JOB_JWT)"
    - vault kv get -field=kubeconfig_${NAMESPACE} common-secret/${APP_ENV} > kube_config
    - chmod 600 kube_config

    # 3. Render secrets into the chart directory
    - consul-template -template="${CHART_NAME}/files/config-crt.hcl:${CHART_NAME}/crt/YC-CA.pem" -once
    - consul-template -template="${CHART_NAME}/files/config-user.hcl:${CHART_NAME}/config/appsettings.ENV.json" -once
    - consul-template -template="${CHART_NAME}/files/config.hcl:${CHART_NAME}/config/appsettings.json" -once

    # 4. Deploy with rendered files now present in chart/config/
    - helm upgrade --install --wait \
        --kubeconfig kube_config \
        ${INSTANCE} ${CHART_NAME} \
        -f ${CHART_NAME}/values-${APP_ENV}.yaml \
        -n ${NAMESPACE} \
        --set aspnetcoreEnvironment=${APP_ENV} \
        --set image.tag=${CI_COMMIT_SHORT_SHA}

The -template src:dst flag of consul-template -once takes the HCL source and writes the rendered output to the destination path. The Vault token is read from $VAULT_TOKEN env var automatically.

How the chart consumes the rendered files

In templates/configmap.yaml the chart reads the rendered config/appsettings.json using Files.Get:

yamlapiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "my-service.fullname" . }}-app-settings
data:
  appsettings.json: |
{{ tpl (.Files.Get "config/appsettings.json") . | indent 4 }}
  appsettings.{{ .Values.aspnetcoreEnvironment }}.json: |
{{ tpl (.Files.Get "config/appsettings.ENV.json") . | indent 4 }}

The CA cert ends up in a separate ConfigMap mounted at /etc/ssl/certs/:

yamlapiVersion: v1
kind: ConfigMap
metadata:
  name: {{ include "my-service.fullname" . }}-ca-pemstore
data:
  YC-CA.pem: |
{{ .Files.Get "crt/YC-CA.pem" | indent 4 }}

Rolling update on config change

The Deployment template includes a checksum annotation so pods restart whenever the ConfigMap content changes:

yamlspec:
  template:
    metadata:
      annotations:
        checksum/configmap: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
        checksum/configmap-ca: {{ include (print $.Template.BasePath "/capem-configmap.yaml") . | sha256sum }}

Helm computes sha256sum of the rendered ConfigMap content. If the content changes between two helm upgrade calls, the Pod template hash changes and Kubernetes triggers a rolling restart.

Debugging render failures

bash# Test consul-template rendering locally
export VAULT_ADDR=https://vault.test.example.com
export VAULT_TOKEN=<your-dev-token>
export APP_ENV=Development

consul-template -template="files/config.hcl:/tmp/appsettings.json" -once -log-level=debug
cat /tmp/appsettings.json

If consul-template hangs (doesn't exit with -once), it means a secret path doesn't exist — the {{ with secret }} block blocks until it gets data. Check the Vault path exists:

bashvault kv get Microservices/my-service/Development/system

Limitations

If a Vault secret changes without a new deploy, pods keep the old value until the next pipeline run. For frequently rotating credentials, pair this with External Secrets Operator or Vault Agent instead.
The rendered config/appsettings.json is never committed to git — it exists only in the ephemeral CI runner filesystem during the deploy job.
consul-template -once exits non-zero if Vault is unreachable or the token lacks permission. This is desirable: the helm upgrade never runs with incomplete config.

Summary

consul-template -once renders Vault secrets into chart files during CI deploy — no Vault dependency at runtime
Rendered files live only in the ephemeral CI runner filesystem; they are never committed to git
checksum/configmap annotation on the Deployment triggers rolling restart whenever config changes
If Vault is unreachable or the token lacks permissions, consul-template -once exits non-zero and helm upgrade never runs
For frequently rotating credentials, consider External Secrets Operator or Vault Agent instead

weblog