Kustomize base/custom/patch: DRY Helm values across environments

Published: 2026-02-07

Six environments, one git repo, no repeated Helm values. That's the goal. The three-tier Kustomize layout achieves it. This post walks through the full structure, what belongs at each tier, how patches work, common mistakes, and how to validate everything locally before pushing.

The three tiers

copyfluxcd/base/          ← Tier 1: shared HelmRelease, all common values
fluxcd/custom/        ← Tier 2: optional components (not for every env)
projects/{env}/hub/   ← Tier 3: env overlay = base+custom + delta patches

A HelmRelease object starts in base/. It carries every value that is the same across all environments: image repository, tag, resource requests/limits, probes, serviceMonitor settings, TLS config, feature flags.

The value in this structure: when you upgrade a chart version, you change one line in base/. When you tune resource limits, you change one block in base/. Patches never touch chart versions or common values — they only ever contain environment-specific deltas.

What goes in base

Everything stable and shared. Example — base/monitoring/kube-prom-stack/helmrelease.yaml:

yamlcopyapiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kube-prom-stack
spec:
  interval: 30m
  timeout: 15m
  chart:
    spec:
      chart: kube-prometheus-stack
      version: "85.2.0"
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
        namespace: flux-system
  values:
    grafana:
      enabled: false      # Grafana lives only on infra, never on spokes
    defaultRules:
      create: false
    kubeScheduler:
      enabled: false
    kubeProxy:
      enabled: false
    prometheus:
      prometheusSpec:
        retention: 7d
        storageSpec:
          volumeClaimTemplate:
            spec:
              resources:
                requests:
                  storage: 20Gi

Note grafana.enabled: false. Grafana lives only on infra (the hub). All spokes send metrics to hub's Prometheus via remote_write. The spoke only runs the stack, not the visualization layer.

The chart version is in base/. Upgrading kube-prometheus-stack from 85.2.0 to 86.0.0 is one commit, one line, and automatically applies to all environments.

What goes in custom

Components not needed by all environments. The custom/ tree mirrors base/:

copycustom/
├── apps/      ← allure, sonarqube, vault, trivy-operator, terraform-operator
├── db/        ← cassandra, clickhouse, minio, mongodb, redis
├── elk/       ← apm-server, kibana, otel-collector
├── monitoring/← abot, cctp, exporters, apps-rules
├── queue/     ← kafka, kafka-ui, rabbitmq
└── web/       ← apisix-ext (demo only)

dev and test pull in the full database stack. sre and loadgds are lightweight — no Kafka, Mongo, Cassandra, ClickHouse.

Each custom/ subdirectory has its own kustomization.yaml that lists the HelmRelease files inside it:

yamlcopy# custom/db/kustomization.yaml
resources:
  - redis/
  - mongodb/
  - cassandra/
  - clickhouse/
  - minio/

This means custom/db/ is a valid Kustomization target. The hub overlay can reference it as a directory, which the load-restrictor allows.

What goes in patches (only the delta)

Patches contain only what differs between environments. Never duplicate base values in patches:

A minimal patch:

yamlcopy# projects/dev/kustomization/hub/patches/elasticsearch.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: elasticsearch
  namespace: dev
spec:
  values:
    master:
      persistence:
        storageClass: local-path
    esConfig:
      elasticsearch.yml: |
        xpack.security.enabled: true
    esJavaOpts: "-Xmx2g -Xms2g"

The namespace: dev line in metadata is required — Kustomize uses it to match the correct object. Without it, Kustomize doesn't know which HelmRelease named elasticsearch to patch (there could be one per environment in the same kustomization).

Compare this to the same chart on sre (Yandex Cloud, more resources):

yamlcopy# projects/sre/kustomization/hub/patches/elasticsearch.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: elasticsearch
  namespace: sre
spec:
  values:
    master:
      persistence:
        storageClass: yc-network-ssd
      resources:
        requests:
          memory: 8Gi
          cpu: 2
    esJavaOpts: "-Xmx6g -Xms6g"
    replicas: 3

The patches are different; base/ is identical. A chart version bump in base/ hits both environments automatically.

Hub kustomization.yaml: assembling the overlay

yamlcopy# projects/dev/kustomization/hub/kustomization.yaml
namespace: dev
resources:
  - ../../../../base/
  - ../../../../custom/db/
  - ../../../../custom/elk/
  - ../../../../custom/monitoring/
  - spoke.yaml        # Flux Kustomization object for spoke layer
  - monitoring.yaml   # Flux Kustomization object for monitoring layer

patches:
  - path: patches/elasticsearch.yaml
  - path: patches/kube-prom-stack.yaml
  - path: patches/redis.yaml
  # kubeConfig injection for ALL HelmRelease
  - target:
      kind: HelmRelease
    patch: |-
      - op: add
        path: /spec/kubeConfig
        value:
          secretRef:
            name: dev-kubeconfig

The kubeConfig JSON patch is appended to all HelmRelease automatically — no per-chart duplication. This is the only place where dev-kubeconfig is mentioned in the hub overlay.

The namespace: dev at the top sets the namespace for all resources in this overlay, unless they specify their own. This is how all HelmRelease end up in the dev namespace on hub without repeating namespace: dev in every file.

The directory-only rule

Kustomize's load-restrictor blocks references outside the current directory when running kubectl kustomize locally. Always reference directories with a kustomization.yaml inside, never individual files:

yamlcopy# Correct
resources:
  - ../../../../base/
  - ../../../../custom/db/

# Wrong — breaks kubectl kustomize locally
resources:
  - ../../../../custom/db/redis/helmrelease.yaml

This also makes the structure self-documenting: you can look at any kustomization.yaml and immediately understand what components are included, without counting YAML files.

Patch strategies: strategic merge vs JSON patch

Kustomize supports two patch formats:

Strategic merge patch (what we use for most patches): looks like a partial YAML object, merges with the base at the same path:

yamlcopy# Merge: only the listed keys are changed, everything else from base stays
spec:
  values:
    master:
      replicas: 3

JSON patch (what we use for structural operations like adding kubeConfig):

yamlcopy- op: add
  path: /spec/kubeConfig
  value:
    secretRef:
      name: dev-kubeconfig

Use strategic merge for value overrides. Use JSON patch when you need add, remove, or replace on a path that might not exist.

One gotcha with strategic merge on the values: block of a HelmRelease: Kustomize merges at the YAML key level, not at the Helm values level. If base has:

yamlcopyvalues:
  redis:
    password: placeholder
    replicas: 1

And your patch has:

yamlcopyvalues:
  redis:
    password: "${REDIS_PASSWORD}"

The result is:

yamlcopyvalues:
  redis:
    password: "${REDIS_PASSWORD}"
    replicas: 1   # preserved from base

This works as expected. But if the patch introduces a completely new top-level key under values:, it's added. If it removes a key, it's... not removed (strategic merge doesn't remove keys unless you use $patch: delete directive). For key removal, use JSON patch with op: remove.

Validating the render locally

bashcopy# Full hub overlay
kubectl kustomize fluxcd/projects/dev/kustomization/hub/

# Check kubeConfig was injected in all HelmRelease
kubectl kustomize fluxcd/projects/dev/kustomization/hub/ | grep -A 5 "kubeConfig"

# Check a specific chart values after patching
kubectl kustomize fluxcd/projects/dev/kustomization/hub/ | \
  yq '. | select(.metadata.name == "elasticsearch") | .spec.values'

# Diff between two environments
diff \
  <(kubectl kustomize fluxcd/projects/dev/kustomization/hub/) \
  <(kubectl kustomize fluxcd/projects/test/kustomization/hub/)

Run this before every commit that touches patches. It catches YAML errors and merge conflicts before they reach Flux.

For CI, add this to your GitLab pipeline:

yamlcopykustomize-validate:
  stage: validate
  image: bitnami/kubectl:latest
  script:
    - for env in dev test sre loadgds demo; do
        echo "Validating $env...";
        kubectl kustomize fluxcd/projects/${env}/kustomization/hub/ > /dev/null;
      done

This runs in ~5 seconds and catches 90% of patch errors before merge.

Adding a new environment

1. Copy an existing environment's projects/{env}/ directory
2. Update the namespace: in kustomization.yaml
3. Update the secret name in the kubeConfig patch
4. Update hostnames and StorageClass in patches
5. Add the new SealedSecret kubeconfig
6. Commit and let Flux pick it up

The entire process takes 20-30 minutes, and the new environment gets the same Helm chart versions and values as all existing ones. No manual Helm installs, no copy-pasted values files.