Bootstrapping k3s clusters with Ansible and Flux

Published: 2026-02-12

Spinning up a new Kubernetes cluster should be reproducible. Ansible handles the OS level; FluxCD takes over from there. Here's how the bootstrap pipeline works, step by step, including the nuances that matter when automating real hardware.

Tools

bashcopybrew install kubectl helm fluxcd/tap/flux kubeseal ansible
ansible-galaxy install -r ansible/requirements.yml

The Ansible collection used is ansible.posix plus the standard kubernetes.core. k3s is installed via the official install script, pinned to a version in group_vars/all.yaml. Pinning is important: k3s upgrades occasionally require node drain procedures that aren't safe to do unattended.

Cluster inventory layout

copyansible/
├── ansible.cfg
├── group_vars/
│   ├── all.yaml      ← k3s version, common config
│   ├── infra.yaml    ← infra-specific vars (SANS, token)
│   ├── dev.yaml
│   └── test.yaml
└── inventory/
    ├── infra/hosts.yaml
    ├── dev/hosts.yaml
    └── test/hosts.yaml

sre, loadgds, and demo are Yandex Cloud Managed Kubernetes — no Ansible inventory for them. Kubeconfig comes from yc managed-kubernetes cluster get-credentials.

group_vars/all.yaml sample

yamlcopyk3s_version: "v1.31.4+k3s1"
k3s_token: "{{ vault_k3s_token }}"   # from Ansible Vault
k3s_extra_args: "--disable traefik --flannel-backend=none"

The token is the shared secret all k3s nodes use to join the cluster. Rotate it after bootstrap — it's only needed for node join, not for ongoing operation.

Step 1: install k3s

bashcopyansible-playbook ansible/install-k3s.yaml -i ansible/inventory/dev/

The playbook:

1. Disables swap and removes it from /etc/fstab. k3s and kubelet have undefined behaviour with swap enabled; this is a hard requirement.
2. Sets kernel parameters: net.ipv4.ip_forward=1, net.bridge.bridge-nf-call-iptables=1, fs.inotify.max_user_watches=524288, fs.inotify.max_user_instances=512. The inotify limits prevent "too many open files" errors in clusters with many pods watching ConfigMaps.
3. Loads kernel modules: br_netfilter (bridge traffic rules), overlay (containerd overlay filesystem), nf_conntrack (connection tracking for NetworkPolicy). The overlay module guard checks if it's already loaded before trying to load it — modprobe returns an error on some kernels if the module is already built-in.
4. Downloads and runs the k3s install script with --disable traefik --flannel-backend=none — Cilium replaces the default CNI, APISIX replaces Traefik.
5. Writes a kubeconfig to the control-plane node.

k3s is installed without Traefik and without Flannel because APISIX handles ingress and Cilium handles networking. Starting k3s with a non-existent CNI means pods won't schedule until Cilium is installed in the next step — this is expected.

Why Cilium before Flux

The CNI must be functional before Flux can start, because Flux itself runs as pods. If you bootstrap Flux before Cilium, the Flux pods will be stuck in Pending state (no network). The bootstrap order is: k3s → Cilium → Sealed Secrets controller → Flux.

Sealed Secrets must come before Flux because Flux will immediately try to reconcile the repo, which includes SealedSecrets that need the controller to be present.

Step 2: collect kubeconfigs

bashcopyansible-playbook ansible/get-kubeconfig.yaml -i ansible/inventory/dev/

Copies the kubeconfig from the control-plane node to .kubeconfigs/dev.yaml locally with the context renamed to dev-k8s.

The kubeconfig has 127.0.0.1 as the server address (k3s binds to localhost by default). The playbook patches it to the node's external IP so it works from outside:

yamlcopy- name: Replace localhost with external IP
  ansible.builtin.replace:
    path: ".kubeconfigs/dev.yaml"
    regexp: "https://127.0.0.1"
    replace: "https://{{ ansible_host }}"

Step 3: install Cilium

bashcopyansible-playbook ansible/upgrade-cilium.yaml -i ansible/inventory/dev/

Despite the name, this playbook also handles initial Cilium installation. Cilium is deployed via Helm directly (not through Flux) because it's a prerequisite for pod networking — Flux itself needs a functional CNI to start.

For on-prem clusters the playbook enables:

• kubeProxyReplacement: true (eBPF-based kube-proxy — replaces kube-proxy completely)
• L2 announcements (l2announcements.enabled: true — makes LoadBalancer services work on bare metal)
• Hubble with UI (hubble.enabled: true, hubble.relay.enabled: true, hubble.ui.enabled: true)

Wait for Cilium to report all agents healthy before proceeding:

bashcopycilium status --wait

At this point, pods should start scheduling and the cluster becomes functional.

Step 4: bootstrap Flux on hub

bashcopyansible-playbook ansible/bootstrap-flux.yaml -i ansible/inventory/infra/

This runs flux bootstrap gitlab against the infra cluster, pointing to the infra git repo. It creates the flux-system namespace, installs the Flux CRDs and controllers, and creates a GitRepository object that watches the repo's main branch.

From this point Flux reconciles continuously. The gotk-components.yaml and gotk-sync.yaml files in fluxcd/flux-system/ pin the exact Flux version.

The bootstrap command is idempotent — running it again on an already-bootstrapped cluster updates Flux if the version pinned in the repo has changed.

GitLab token for Flux

Flux needs a GitLab personal access token (or deploy token) with read_repository scope to clone the repo. This is configured as a Kubernetes Secret in the flux-system namespace:

bashcopyflux bootstrap gitlab \
  --owner=my-group \
  --repository=infra \
  --branch=main \
  --path=fluxcd \
  --token-auth \
  --personal  # use personal access token

For production, use a deploy token scoped to the repository with read_repository only.

Step 5: seal the spoke kubeconfigs

Spoke clusters don't run Flux. Hub Flux needs their kubeconfigs to deploy to them. Those kubeconfigs are stored as SealedSecrets in git.

bashcopykubectl create secret generic dev-kubeconfig \
  --from-file=value=.kubeconfigs/dev.yaml \
  --namespace flux-system \
  --dry-run=client -o yaml | \
  kubeseal \
    --format yaml \
    --controller-name sealed-secrets-controller \
    --controller-namespace kube-system \
    --kubeconfig .kubeconfigs/infra.yaml \
  > fluxcd/dev-kubeconfig-sealed.yaml

The namespace is flux-system — this must match where the Kustomization that references the kubeconfig runs. If you use --namespace dev here but the Kustomization is in flux-system, Flux won't find the secret.

Step 6: commit and push

bashcopygit add fluxcd/
git commit -m "feat: bootstrap dev cluster"
git push

GitLab CI runs flux reconcile source git flux-system on push to main. Within seconds Flux creates the dev namespace on hub, decrypts the kubeconfig, and starts reconciling HelmReleases against the dev cluster.

Monitor reconciliation:

bashcopyflux get kustomizations -w
flux get helmreleases -A

The first reconciliation downloads all Helm charts — this takes a few minutes on a cold cluster. Subsequent reconciliations are fast (seconds).

Tuning node logging

bashcopyansible-playbook ansible/tune-node-logging.yaml -i ansible/inventory/dev/

Caps journald's disk usage (SystemMaxUse=500M) and sets RateLimitBurst=10000 to prevent log floods from filling the disk. Standard practice for long-running cluster nodes. Without this, a misbehaving pod that logs at high rate can fill /var/log and crash the node.

Upgrading k3s

k3s upgrades are done via the same install playbook with a new version in group_vars/all.yaml. For single-node clusters, the upgrade is in-place:

bashcopy# Update k3s_version in group_vars/all.yaml, then:
ansible-playbook ansible/install-k3s.yaml -i ansible/inventory/dev/

For multi-node clusters, drain each node before upgrading:

bashcopykubectl drain <node> --ignore-daemonsets --delete-emptydir-data
ansible-playbook ansible/install-k3s.yaml -i ansible/inventory/dev/ --limit <node>
kubectl uncordon <node>

k3s 1.28+ supports automated upgrades via the system-upgrade-controller, which handles this drain/uncordon sequence automatically if you prefer.

Upgrading Cilium

bashcopyansible-playbook ansible/upgrade-cilium.yaml -i ansible/inventory/dev/

The playbook performs a rolling Cilium upgrade via Helm with a drain/uncordon sequence if there are multiple nodes. Safe to run on a live cluster.

For major Cilium upgrades (e.g., 1.14 → 1.15), check the Cilium upgrade guide for required steps — some versions require disabling kube-proxy replacement temporarily during the upgrade to avoid networking gaps.