FluxCD hub-and-spoke: one cluster manages two others

Published: 2026-02-09

Our GitOps topology is three k3s clusters — infra, dev, and test — with FluxCD running only on infra. The infra cluster acts as the hub: it reads the Git repo and applies HelmReleases not just to itself but to dev and test via remote kubeconfig references. This is the hub-and-spoke pattern.

This post covers the full setup, the operational nuances we learned after six months of running this topology, and the troubleshooting commands that have saved us repeatedly.

Why hub-and-spoke

The alternative is running Flux on every cluster. That's fine when clusters are independent, but ours share chart versions, Vault paths, and monitoring configs. With hub-and-spoke there's one source of truth and one place to watch for drift.

Operationally: if dev gets inconsistent, you check the Kustomization status on the infra cluster — not on dev.

The more subtle advantage is lifecycle management. When you upgrade Flux itself, you do it once on infra. All spokes inherit the upgrade through the kubeconfig mechanism without needing direct access. Spoke clusters can be entirely air-gapped from the internet — only the hub needs Git access.

Tradeoffs

If clusters are in separate security domains (different teams, different compliance requirements), running Flux per cluster may be better. Hub-and-spoke shines when you're one team operating multiple clusters with strong config overlap.

Directory layout

copyfluxcd/
  kustomization.yaml          # root — lists infra + spoke kustomizations
  infra-kustomization.yaml    # HelmReleases targeting infra cluster (self)
  dev-kustomization.yaml      # HelmReleases targeting dev cluster
  test-kustomization.yaml     # HelmReleases targeting test cluster (commented until ready)
  base/
    monitoring/               # shared chart configs
    external-secrets/
    ...
  custom/
    monitoring/               # env-specific overrides
    ...

The base/ directory contains HelmRelease manifests without a spec.kubeConfig field. The per-cluster kustomization files add it via JSON patches. This keeps base portable — if you ever want to run Flux directly on dev, you just remove the kubeConfig injection.

The custom/ directory holds environment-specific value overrides. For example, dev might have replicas: 1 while test has replicas: 2. These are Kustomize strategic merge patches on top of base.

How spoke targeting works

Every HelmRelease that should land on dev or test gets a spec.kubeConfig injected by the Kustomization patch:

yamlcopy# dev-kustomization.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: dev
  namespace: flux-system
spec:
  interval: 10m
  path: ./fluxcd/custom
  sourceRef:
    kind: GitRepository
    name: flux-system
  kubeConfig:
    secretRef:
      name: dev-kubeconfig    # SealedSecret, decrypted by infra's Sealed Secrets
  patches:
    - patch: |
        - op: add
          path: /spec/kubeConfig
          value:
            secretRef:
              name: dev-kubeconfig
      target:
        kind: HelmRelease

Flux applies the HelmRelease manifest on infra, but the Helm controller calls the dev API server using the kubeconfig from dev-kubeconfig secret.

The kubeConfig field on the Kustomization itself handles non-HelmRelease resources (ConfigMaps, Secrets, etc.). The patch on HelmRelease targets ensures the Helm controller also uses the remote kubeconfig when performing Helm operations.

Why the patch is needed in addition to kubeConfig

Setting kubeConfig on the Kustomization tells the Kustomize controller to apply resources to the remote cluster. But HelmReleases are special — the Helm controller reconciles them separately. Without the patch adding spec.kubeConfig to each HelmRelease, the Helm controller would execute against the local (infra) cluster instead.

This is a common trap when first setting up hub-and-spoke: Kustomization shows Ready=True, but the charts land on the wrong cluster.

Kubeconfig secret

The kubeconfig is extracted from the dev node, the server address replaced with the node's external IP, and then sealed using the infra cluster's Sealed Secrets controller:

bashcopy# 1. Extract and fix address
kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml config view --raw \
  | sed 's/127.0.0.1/192.168.1.20/g' > /tmp/dev.yaml

# 2. Create the Secret manifest
kubectl create secret generic dev-kubeconfig \
  --namespace=flux-system \
  --from-file=value=/tmp/dev.yaml \
  --dry-run=client -o yaml > /tmp/dev-secret.yaml

# 3. Seal it with infra's public cert
kubeseal --cert .kubeconfigs/sealed-secrets-infra.pem \
  --format yaml \
  < /tmp/dev-secret.yaml \
  > fluxcd/dev-kubeconfig-sealed.yaml

# 4. Commit and push
git add fluxcd/dev-kubeconfig-sealed.yaml && git commit -m "add dev kubeconfig" && git push

Flux detects the new file, infra's Sealed Secrets controller decrypts it, and the hub immediately has access to the dev cluster.

Namespace matters

The kubeconfig Secret must be in the same namespace as the Kustomization that references it — flux-system in most setups. If the secret is in a different namespace, Flux will fail to find it and log secret "dev-kubeconfig" not found.

Certificate rotation

k3s rotates its cluster CA only when explicitly requested (e.g., k3s certificate rotate). However, if you rebuild the dev cluster, the CA changes and the stored kubeconfig becomes invalid. After a cluster rebuild:

bashcopy# Re-extract, re-seal, commit
kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml config view --raw \
  | sed 's/127.0.0.1/192.168.1.20/g' > /tmp/dev.yaml

kubectl create secret generic dev-kubeconfig \
  --namespace=flux-system \
  --from-file=value=/tmp/dev.yaml \
  --dry-run=client -o yaml \
  | kubeseal --cert .kubeconfigs/sealed-secrets-infra.pem --format yaml \
  > fluxcd/dev-kubeconfig-sealed.yaml

git add fluxcd/dev-kubeconfig-sealed.yaml && git commit -m "rotate dev kubeconfig" && git push

dependsOn ordering

Some HelmReleases depend on others being deployed first (e.g., monitoring CRDs before PrometheusRules). Use spec.dependsOn:

yamlcopyspec:
  dependsOn:
    - name: kube-prom-stack   # name of the Kustomization, not a directory

This is a common confusion: dependsOn refers to the Kustomization object name, not a path or file name.

Cross-spoke dependencies

You cannot have a dependsOn that crosses spokes. If dev's PrometheusRules depend on infra's kube-prom-stack CRDs, you need to install the CRDs separately on dev (e.g., via a dedicated CRD HelmRelease) and make the PrometheusRules Kustomization depend on that.

Attempting to reference a Kustomization from a different kubeConfig context in dependsOn silently fails — Flux will wait forever because it looks for the named Kustomization in the local flux-system namespace.

Checking spoke health

From the infra cluster:

bashcopy# Check all Kustomizations
flux get kustomizations -A

# Check HelmReleases on the dev spoke
flux get helmreleases -n dev --kubeconfig .kubeconfigs/infra.yaml

# Force reconcile
flux reconcile kustomization dev --kubeconfig .kubeconfigs/infra.yaml

# Watch reconciliation in real time
flux get kustomization dev -w

Drift on dev shows up as a failed HelmRelease in flux get hr output — no need to SSH into the dev node.

Status fields to watch

bashcopykubectl get kustomization dev -n flux-system -o yaml | grep -A5 "conditions:"

Key condition types:

• Ready: True — all resources applied successfully
• Reconciling: True — in progress
• Stalled: True — blocked, usually a dependency waiting

For HelmReleases, add --verbose to flux get to see the Helm status message:

bashcopyflux get helmrelease monitoring -n flux-system --verbose

Suspend and resume a spoke

During cluster maintenance (node OS updates, etc.), suspend the spoke to prevent Flux from fighting your changes:

bashcopyflux suspend kustomization dev
# ... do maintenance ...
flux resume kustomization dev

Flux will immediately reconcile after resume, bringing the cluster back to the desired state. Any manual changes made during maintenance will be reverted — this is intentional GitOps behavior.

To prevent specific resources from being reconciled without suspending the entire kustomization, annotate them:

bashcopykubectl annotate helmrelease myapp -n app kustomize.toolkit.fluxcd.io/reconcile=disabled

When to add a new spoke

If a new cluster is added (e.g., staging):

1. Run install-k3s.yaml + Cilium for the new node
2. Run get-kubeconfig.yaml to extract and seal the kubeconfig
3. Add staging-kustomization.yaml to the repo (copy from dev, change names)
4. Uncomment the entry in root kustomization.yaml
5. Push — Flux picks it up in under a minute

The time from "cluster is ready" to "all apps deployed by Flux" is typically 3–5 minutes. The bottleneck is usually chart downloads on the first pull, not Flux reconciliation.

Troubleshooting common issues

HelmRelease lands on infra instead of spoke

• Cause: spec.kubeConfig not in HelmRelease (patch not applied)
• Fix: check that the patch target.kind: HelmRelease is correct and the path ./fluxcd/custom contains the right HelmReleases

Kustomization is Ready but charts are missing from spoke

• Cause: Kustomization applied resources to spoke, but HelmRelease needs a separate spec.kubeConfig
• Fix: add the kubeConfig patch to the Kustomization

secret "dev-kubeconfig" not found

• Cause: Secret in wrong namespace, or Sealed Secrets controller failed to decrypt
• Fix: check kubectl get sealedsecret -n flux-system, check Sealed Secrets controller logs

Reconciliation stuck on Progressing

• Cause: dependent resource not ready (CRDs not installed yet, etc.)
• Fix: flux get hr -A to find the blocked release, check events with kubectl describe helmrelease