FluxCD hub-and-spoke: managing multiple Kubernetes clusters from one git repo

Published: 2026-02-02

At work we run six Kubernetes clusters: an on-prem infra hub, on-prem dev and test, and three Yandex Cloud managed clusters (sre, loadgds, demo). FluxCD runs only on infra. Everything else is a spoke — Flux never touches it directly.

Here's how the setup works, why we chose it over per-cluster Flux installs, and the operational patterns we've settled on after running it in production.

Why hub-and-spoke instead of Flux on every cluster

The most obvious alternative is installing FluxCD on every cluster and pointing each instance at a different path in the same git repo. It works, but it multiplies operational overhead:

• Flux upgrades become a per-cluster operation. Six clusters mean six flux bootstrap runs or six HelmRelease bumps.
• Debugging requires switching kubectl contexts to check Flux state on each cluster individually.
• Secret management for GitLab CI tokens, container registry credentials, and alerting hooks must be replicated everywhere.
• RBAC for Flux controllers must be configured and audited on each cluster separately.

With hub-and-spoke, you install Flux once. The hub's controllers reconcile everything. You kubectl and flux exclusively against infra. The spokes are managed surfaces, not Flux hosts.

The trade-off is availability: if infra goes down, reconciliation stops for all spokes. Existing workloads keep running — Kubernetes doesn't need Flux to keep pods alive — but new git commits won't be applied until hub recovers. For our workloads this is acceptable. If it isn't for yours, per-cluster Flux with a shared config source is the right call.

What hub-and-spoke means in FluxCD terms

FluxCD v2 lets a HelmRelease or Kustomization specify a kubeConfig secret. When it does, the Flux controllers on hub will apply the manifests to the remote cluster referenced by that secret — not to hub itself.

The pattern:

copyhub (infra cluster)
  └── Flux controllers
        ├── HelmRelease redis → kubeConfig: dev-kubeconfig → dev cluster
        ├── HelmRelease redis → kubeConfig: test-kubeconfig → test cluster
        └── HelmRelease redis → kubeConfig: sre-kubeconfig  → sre cluster

All Flux state lives in one place. flux get helmreleases -A on hub shows the status of every chart on every spoke.

Internally, Flux uses the kubeconfig to create a REST client pointed at the remote API server. The Helm controller runs helm upgrade --install against that client. From the spoke's perspective it looks exactly like a normal Helm install — the CRDs and Pods land in the cluster, but the driver lives elsewhere.

Repository structure

copyfluxcd/
├── kustomization.yaml            ← root Flux entry point
├── flux-system/                  ← Flux CRDs + gotk-components
├── base/                         ← shared HelmRelease (no kubeConfig)
│   ├── elk/
│   ├── monitoring/
│   └── web/
├── custom/                       ← optional components per env
│   ├── apps/
│   ├── db/
│   ├── elk/
│   └── monitoring/
└── projects/
    └── {env}/
        └── kustomization/
            ├── hub/              ← combines base+custom, adds kubeConfig via JSON patch
            ├── spoke/            ← plain k8s objects applied directly to spoke
            ├── spoke/monitoring/ ← Probes + PrometheusRules (after spoke)
            └── routes/           ← ApisixTls

The key design decision: base/ contains complete, working HelmRelease manifests with chart versions, values, and target namespaces. They're environment-agnostic. The only thing missing is spec.kubeConfig — that gets injected by the per-env overlay.

This means chart upgrades happen in one place (base/) and are automatically promoted across all environments that include that base component.

The kubeConfig injection trick

base/ HelmReleases have no kubeConfig field — they're environment-agnostic. The per-env hub overlay adds it via a Kustomize JSON patch:

yamlcopy# projects/dev/kustomization/hub/kustomization.yaml
patches:
  - target:
      kind: HelmRelease
    patch: |-
      - op: add
        path: /spec/kubeConfig
        value:
          secretRef:
            name: dev-kubeconfig

One patch covers every HelmRelease in the overlay. No per-chart repetition.

The patch uses op: add rather than op: replace specifically because the field doesn't exist in base — replace would fail if the path is absent. This is worth knowing if you ever refactor and accidentally put a placeholder kubeConfig in base: the patch would silently do the wrong thing with add (it would replace the value anyway since the path exists), but your intent is unclear. Keep base kubeConfig-free.

The four Kustomization objects per environment

Each environment gets four Flux Kustomization CRDs applied on hub:

dev-monitoring-spoke has dependsOn: dev-spoke because Probe CRDs come from kube-prom-stack, which is installed by dev-apps. Order matters.

Full Kustomization definition for reference:

yamlcopyapiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: dev-apps
  namespace: dev
spec:
  interval: 10m
  path: ./projects/dev/kustomization/hub
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
    namespace: flux-system
  kubeConfig:
    secretRef:
      name: dev-kubeconfig
  timeout: 5m
  wait: true

Note wait: true — Flux will poll the spoke until all applied resources reach a ready state before marking the Kustomization as reconciled. Without it, dependsOn chains can fire before the dependent CRDs are actually installed.

The kubeconfig secret

Each spoke's kubeconfig is stored as a SealedSecret directly inside the {env}-kustomization.yaml file. Flux decrypts it to a plain Secret; HelmRelease references that Secret.

yamlcopy# fluxcd/dev-kustomization.yaml (excerpt)
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: dev-kubeconfig
  namespace: dev
spec:
  encryptedData:
    value: AgB3...long-base64...

The public cert for sealing is in the repo. The private key never leaves the hub cluster.

Generating the kubeconfig

The kubeconfig used as dev-kubeconfig should use a dedicated service account with the minimum permissions Flux needs. For HelmRelease this is typically cluster-admin in practice, though you can scope it to specific namespaces if all charts install into known namespaces.

bashcopy# Create a service account on the spoke for Flux
kubectl --context=dev-k8s create serviceaccount flux-remote -n kube-system
kubectl --context=dev-k8s create clusterrolebinding flux-remote \
  --clusterrole=cluster-admin \
  --serviceaccount=kube-system:flux-remote

# Generate a token (k8s 1.24+)
TOKEN=$(kubectl --context=dev-k8s create token flux-remote -n kube-system --duration=8760h)
SERVER=$(kubectl --context=dev-k8s config view --minify -o jsonpath='{.clusters[0].cluster.server}')
CA=$(kubectl --context=dev-k8s config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}')

# Build kubeconfig
cat > /tmp/dev-flux-kubeconfig.yaml <<EOF
apiVersion: v1
kind: Config
clusters:
- cluster:
    certificate-authority-data: ${CA}
    server: ${SERVER}
  name: dev
contexts:
- context:
    cluster: dev
    user: flux-remote
  name: dev
current-context: dev
users:
- name: flux-remote
  user:
    token: ${TOKEN}
EOF

# Seal it
kubectl create secret generic dev-kubeconfig \
  --from-file=value=/tmp/dev-flux-kubeconfig.yaml \
  --dry-run=client -o yaml | \
  kubeseal --cert flux-sealed-secrets.pem --format yaml

The sealed output goes into the repo. Rotate it before the token expires — 8760h is one year.

Infra is both hub and spoke

infra is the only environment where Flux deploys directly to the same cluster it runs on. The infra Kustomizations have no kubeConfig — no JSON patch is applied. This is the self-managing hub.

This means Flux upgrades itself: when you bump the gotk-components version in flux-system/, Flux will apply the new manifests to its own namespace and restart its own pods. It works reliably but can cause a brief reconciliation pause during the controller rollout.

One gotcha: if the new Flux version has a CRD schema change that breaks existing resources (rare but has happened between major versions), the self-upgrade can leave Flux controllers crashlooping. Always check the FluxCD release notes before bumping the version in flux-system/gotk-components.yaml.

Drift detection and prune: true

All our Kustomizations use prune: true. This means Flux will delete resources from the spoke if they're removed from git. It's what makes git the source of truth rather than just a suggestion.

The implication: if someone manually applies a resource to a spoke with kubectl apply, Flux will delete it on the next reconciliation cycle (default interval: 10 minutes). This is intentional. Spokes should not have out-of-band state.

For the rare case where you need a temporary manual resource on a spoke (debugging, incident response), annotate it:

yamlcopymetadata:
  annotations:
    kustomize.toolkit.fluxcd.io/prune: disabled

Flux will ignore this resource during pruning. Remove the annotation (or the resource) when you're done.

Suspending and resuming

When you need to stop Flux from touching a specific environment — say, during a manual incident mitigation or a deployment freeze — suspend the Kustomization:

bashcopy# Freeze all Flux activity for the dev environment
flux suspend kustomization dev-apps dev-spoke dev-monitoring-spoke dev-routes -n dev --context=infra-k8s

# Resume after the freeze lifts
flux resume kustomization dev-apps dev-spoke dev-monitoring-spoke dev-routes -n dev --context=infra-k8s

Suspending a Kustomization does not affect running workloads on the spoke. Pods keep running. It only stops Flux from applying new changes.

To suspend a single chart without freezing the whole environment:

bashcopyflux suspend helmrelease redis -n dev --context=infra-k8s

Day-to-day operations

bashcopy# Status of all Kustomizations
flux get kustomizations -A --context=infra-k8s

# Status of all HelmRelease on all spokes
flux get helmreleases -A --context=infra-k8s

# Force reconcile after a push
flux reconcile source git flux-system -n flux-system --context=infra-k8s
flux reconcile kustomization dev-apps -n dev --context=infra-k8s

# Correct reconcile order for a full spoke refresh
ENV=dev
flux reconcile ks ${ENV}-apps             -n ${ENV} --context=infra-k8s --timeout=5m
flux reconcile ks ${ENV}-spoke            -n ${ENV} --context=infra-k8s --timeout=5m
flux reconcile ks ${ENV}-monitoring-spoke -n ${ENV} --context=infra-k8s --timeout=5m
flux reconcile ks ${ENV}-routes           -n ${ENV} --context=infra-k8s --timeout=5m

Always pass --context explicitly. Never rely on the current kubectl context when operating a multi-cluster setup.

Checking spoke health from hub

bashcopy# All pods on a spoke without switching context
kubectl --context=dev-k8s get pods -A

# Or if you're working through hub and don't have direct spoke access,
# check Flux's view of the spoke's HelmRelease
flux get helmreleases -n dev --context=infra-k8s

# Events on a failing HelmRelease
flux events --for HelmRelease/redis -n dev --context=infra-k8s

Troubleshooting

HelmRelease stuck in InstallFailed or UpgradeFailed

The most common cause is a values conflict or a CRD that isn't installed yet on the spoke. Get the full error:

bashcopyflux events --for HelmRelease/my-chart -n dev --context=infra-k8s

If it's a CRD issue, check that the Kustomization that installs the CRDs (dev-spoke) has dependsOn pointing to dev-apps, or vice versa depending on your ordering.

dev-kubeconfig secret not found

The SealedSecret controller must be running on hub and the SealedSecret must be in the correct namespace. Verify:

bashcopykubectl get sealedsecret dev-kubeconfig -n dev --context=infra-k8s
kubectl get secret dev-kubeconfig -n dev --context=infra-k8s

If the SealedSecret exists but the plain Secret doesn't, the sealed-secrets controller is either not running or the private key doesn't match the encryption certificate.

Kustomization shows ReconciliationFailed: context deadline exceeded

The spoke API server is unreachable from hub, or the token in the kubeconfig has expired. Check connectivity and regenerate the token if needed.

prune: true deleted something it shouldn't have

If a resource was pruned incorrectly, it means it was either not tracked by the Kustomization's inventory or it was removed from git unintentionally. Check the Kustomization events and restore from git.

What you actually get

• One git repo, one place to look for any change across all clusters.
• Spoke clusters need no Flux installation — zero agent footprint.
• A git push to main triggers reconcile automatically (GitLab CI calls flux reconcile source git flux-system).
• Manual flux suspend / flux resume for HelmRelease lets you freeze a specific chart on a specific cluster without touching others.
• Centralized audit trail: every change to every spoke is a git commit on hub.
• Upgrade path is simple: upgrade Flux once on hub, it manages its own controllers and simultaneously manages all spokes.