Adding a new environment to a FluxCD hub-and-spoke cluster

Published: 2026-04-30

The full checklist for onboarding a new environment — from provisioning the cluster to the first successful Flux reconcile. Uses a hypothetical staging environment on Yandex Cloud (light stack, no Kafka/Mongo/Cassandra/ClickHouse).

Overview

In the hub-and-spoke model, the hub cluster (infra) runs Flux and manages spoke clusters via kubeConfig in each Kustomization object. Adding a new environment means:

Creating a Kubernetes cluster
Creating the FluxCD project directory structure
Creating the top-level kustomization file
Writing per-chart patches
Sealing the kubeconfig as a SealedSecret
Wiring up namespaces and secrets
Registering in the root kustomization

Step 1: provision the cluster

For Yandex Cloud:

bash# Provision via Terraform (or manually in YC console)
# Get credentials
yc managed-kubernetes cluster get-credentials staging --external
# Rename context to match convention
kubectl config rename-context yc-staging staging-k8s
# Copy to local kubeconfigs directory (used for sealing)
cp ~/.kube/config .kubeconfigs/staging.yaml

For on-prem k3s:

bash# Add hosts to ansible/inventory/staging/hosts.yaml
# Add vars to ansible/group_vars/staging.yaml
ansible-playbook ansible/install-k3s.yaml -i ansible/inventory/staging/
ansible-playbook ansible/get-kubeconfig.yaml -i ansible/inventory/staging/
ansible-playbook ansible/upgrade-cilium.yaml -i ansible/inventory/staging/
cp ~/.kube/config .kubeconfigs/staging.yaml

Step 2: create the project structure

bashmkdir -p fluxcd/projects/staging/kustomization/{hub/patches,spoke/secrets,spoke/monitoring,routes}

Copy the closest existing environment as a starting point. For a YC light stack, sre or loadgds is the best template:

bashcp fluxcd/projects/sre/kustomization/hub/kustomization.yaml \
   fluxcd/projects/staging/kustomization/hub/kustomization.yaml

cp -r fluxcd/projects/sre/kustomization/spoke/ \
      fluxcd/projects/staging/kustomization/spoke/

cp fluxcd/projects/sre/kustomization/routes/kustomization.yaml \
   fluxcd/projects/staging/kustomization/routes/kustomization.yaml

Step 3: create staging-kustomization.yaml

Copy fluxcd/sre-kustomization.yaml → fluxcd/staging-kustomization.yaml and edit:

yaml# Namespace for hub-side objects
apiVersion: v1
kind: Namespace
metadata:
  name: staging
---
# SealedSecret placeholder — fill in step 5
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: staging-kubeconfig
  namespace: staging
spec:
  encryptedData:
    value: PLACEHOLDER
---
# Hub: deploy apps to the staging cluster
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: staging-apps
  namespace: staging
spec:
  interval: 10m
  path: ./fluxcd/projects/staging/kustomization/hub/
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  kubeConfig:
    secretRef:
      name: staging-kubeconfig
---
# Spoke: deploy cluster-level resources
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: staging-spoke
  namespace: staging
spec:
  interval: 10m
  path: ./fluxcd/projects/staging/kustomization/spoke/
  prune: true
  dependsOn:
    - name: staging-apps
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  kubeConfig:
    secretRef:
      name: staging-kubeconfig
---
# Routes: ApisixRoutes and TLS config
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: staging-routes
  namespace: staging
spec:
  interval: 10m
  path: ./fluxcd/projects/staging/kustomization/routes/
  prune: true
  dependsOn:
    - name: staging-spoke
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  kubeConfig:
    secretRef:
      name: staging-kubeconfig

Step 4: create patches

One patch file per Helm chart, containing only the delta from base.

kube-prom-stack.yaml — storageClass, retention, external labels:

yamlspec:
  values:
    prometheus:
      prometheusSpec:
        retention: 7d
        externalLabels:
          cluster: staging    # MUST be unique across all clusters
        storageSpec:
          volumeClaimTemplate:
            spec:
              storageClassName: yc-network-ssd
              resources:
                requests:
                  storage: 20Gi
    grafana:
      ingress:
        hosts:
          - grafana.staging.test.example.com

elasticsearch.yaml — hostname and storage:

yamlspec:
  values:
    master:
      persistence:
        storageClass: yc-network-ssd
    ingress:
      hosts:
        - host: elasticsearch.staging.test.example.com

apisix-int.yaml — external IP (from YC LB, known after cluster provision):

yamlspec:
  values:
    service:
      type: LoadBalancer
      # loadBalancerIP: "X.X.X.X"  # Leave empty for YC to auto-assign

Step 5: seal the kubeconfig

bashkubectl create secret generic staging-kubeconfig \
  --from-file=value=.kubeconfigs/staging.yaml \
  --namespace staging \
  --dry-run=client -o yaml | \
  kubeseal \
    --format yaml \
    --controller-namespace flux-system \
    --kubeconfig .kubeconfigs/infra.yaml \
  > /tmp/staging-sealed.yaml

# Copy encryptedData.value from /tmp/staging-sealed.yaml
# into fluxcd/staging-kustomization.yaml
# Then clean up
rm /tmp/staging-sealed.yaml

The kubeconfig is sealed with the infra cluster's Sealed Secrets key. If the infra cluster is ever rebuilt, reseal all kubeconfigs with the new key.

Step 6: add to spoke overlays

Create fluxcd/projects/staging/kustomization/spoke/namespaces.yaml listing all namespaces to create. For light stack, exclude kafka, mongodb, rabbitmq, cassandra, clickhouse.

Add docker-registry pull secrets to spoke/secrets/ — copy from an existing environment and reseal with the staging namespace.

For YC clusters: skip cilium-lb-pool.yaml and cilium-l2-announcement.yaml — not needed.

Step 7: create ApisixTls route

yaml# routes/apisix-tls-staging.yaml
apiVersion: apisix.apache.org/v2
kind: ApisixTls
metadata:
  name: staging-example-com
  namespace: ingress-apisix
spec:
  hosts:
    - "*.staging.test.example.com"
  secret:
    name: staging-example-com-tls
    namespace: ingress-apisix

Add the TLS cert and key as a SealedSecret in spoke/secrets/.

Step 8: register in the root kustomization

yaml# fluxcd/kustomization.yaml
resources:
  - flux-system/
  - infra-kustomization.yaml
  - dev-kustomization.yaml
  - test-kustomization.yaml
  - sre-kustomization.yaml
  - loadgds-kustomization.yaml
  - staging-kustomization.yaml   # ← add this

Step 9: commit, push, verify

bashgit add fluxcd/
git commit -m "feat: add staging environment"
git push

# Wait for Flux to pick it up (usually 2-3 minutes), then check
flux get kustomizations -A --context=infra-k8s | grep staging

# Force reconcile if needed
flux reconcile ks staging-apps -n staging --context=infra-k8s --timeout=10m

# Check pods on the new cluster
kubectl get pods -A --context=staging-k8s

Common pitfalls

Forgetting externalLabels.cluster in kube-prom-stack — Grafana can't distinguish staging metrics from dev. Every cluster MUST have a unique value.
Missing namespace field in patch — Kustomize won't find the target HelmRelease. The patch target must include namespace.
dependsOn pointing to non-existent object — check that spoke.yaml and monitoring.yaml Kustomization object names match exactly.
TLS cert missing — if APISIX starts before the TLS SealedSecret decrypts, routes will fail. The dependsOn chain ensures ordering.
Wrong context in kubeconfig — the sealed kubeconfig must use the full path to the server, not a cluster alias that only exists locally.
Copying patches from a cluster with Cilium L2 to a YC cluster — remove the loadBalancerIP field; YC auto-assigns.

Summary

Nine-step checklist from cluster provisioning to first successful Flux reconcile
Every environment needs a unique externalLabels.cluster in kube-prom-stack — missing it means Grafana can't distinguish metrics from different clusters
The kubeconfig sealed secret and the kubeConfig reference in the hub Kustomization are the critical link between hub and spoke
dependsOn chains ensure ordering: hub waits for the spoke layer, monitoring waits for both
Test locally with kubectl kustomize fluxcd/projects/staging/ --dry-run=client before pushing to avoid breaking the root Kustomization

weblog