Ansible playbook for k3s: sysctl, inotify, swap, and kernel modules

Published: 2026-02-03

Installing k3s on a bare VM is five commands. Making that idempotent, reproducible, and safe for 200+ pods running .NET apps is an Ansible playbook. This post walks through the full pre-task checklist, the k3s install arguments, and the Cilium bootstrap sequence — with the reasoning behind every decision.

Why a playbook at all

When you have six clusters, re-running the same manual steps on each node is a liability. A missed sysctl setting means a cluster that looks healthy but drops traffic under load. A swap line left in /etc/fstab means a node that breaks after the first VM reboot. The playbook encodes tribal knowledge as code: it runs identically every time and leaves a diff in git when something changes.

The playbook is structured as:

Pre-tasks: OS-level preparation (swap, kernel modules, sysctl, limits)
k3s install
Cilium install (CNI must be present before node can go Ready)
Post-tasks: wait for node Ready, copy kubeconfig to control host

Pre-tasks before k3s install

Disable swap

Kubernetes refuses to start on a node with swap enabled. The kubelet checks for swap at startup and exits if it finds any — unless you explicitly pass --fail-swap-on=false, which we don't. Disabling it at runtime is not enough — you also need to remove it from /etc/fstab or it comes back after reboot:

yaml- name: Disable swap
  ansible.builtin.shell: swapoff -a
  changed_when: false

- name: Remove swap from /etc/fstab
  ansible.builtin.replace:
    path: /etc/fstab
    regexp: '^([^#].*\s+swap\s+.*)$'
    replace: '# \1'

The regexp comments out any non-comment line that contains swap as a filesystem type. It handles both swapfile and swap partition entries. The line is commented rather than deleted so it's visible in the file history if you need to diagnose why swap was disabled.

If your VMs are provisioned from a cloud image that puts swap on a separate device (common in some Yandex Cloud images), also verify swapon --show returns nothing after the task runs.

Kernel modules

Cilium and k3s networking need br_netfilter (bridge traffic sees iptables rules) and overlay (container filesystem layers):

yaml- name: Load kernel modules
  community.general.modprobe:
    name: "{{ item }}"
    state: present
  loop:
    - br_netfilter
    - overlay

- name: Persist kernel modules
  ansible.builtin.copy:
    dest: /etc/modules-load.d/k3s.conf
    content: |
      br_netfilter
      overlay
    mode: "0644"

The modprobe task loads the modules for the current session. The copy task persists them via modules-load.d so they survive reboots. Both tasks are necessary — modprobe alone doesn't survive reboots, and modules-load.d alone doesn't load the modules immediately.

On some minimal Linux images (especially stripped-down cloud VMs), overlay may not be compiled as a module — it might be built-in to the kernel. In that case modprobe will return an error. You can guard against this:

yaml- name: Load overlay module (may be built-in)
  community.general.modprobe:
    name: overlay
    state: present
  ignore_errors: true

Verify with lsmod | grep overlay after the task runs. If the module is built-in, lsmod won't show it, but it's there.

sysctl for networking and Cilium

yaml- name: Set sysctl params
  ansible.posix.sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    sysctl_set: true
    reload: true
  loop:
    - { key: net.bridge.bridge-nf-call-iptables,  value: "1" }
    - { key: net.bridge.bridge-nf-call-ip6tables, value: "1" }
    - { key: net.ipv4.ip_forward,                 value: "1" }
    # Cilium: without this, rp_filter drops legitimate pod traffic
    - { key: net.ipv4.conf.all.rp_filter,         value: "0" }
    - { key: net.ipv4.conf.default.rp_filter,     value: "0" }

Each setting explained:

net.bridge.bridge-nf-call-iptables: 1 — when a packet crosses a bridge, it should also pass through the iptables rules. Without this, network policies implemented via iptables are bypassed for bridged traffic. Required even when Cilium uses eBPF, because k3s creates bridge interfaces during startup.

net.ipv4.ip_forward: 1 — enables packet forwarding between interfaces. This is what allows pods on node A to reach pods on node B: the node acts as a router. Without this, inter-node pod traffic is silently dropped at the kernel level.

net.ipv4.conf.all.rp_filter: 0 and net.ipv4.conf.default.rp_filter: 0 — rp_filter=0 is the one that trips people up. Strict reverse path filtering drops packets where the source IP doesn't match the routing table from Cilium's perspective — which happens constantly with eBPF-based routing.

With Cilium in kubeProxyReplacement=true mode, the kernel's routing table doesn't know about virtual pod IPs — Cilium handles them in eBPF. If rp_filter is strict (1 or 2), it checks if the source IP of an incoming packet has a route back through the same interface, and drops it if not. This produces intermittent packet loss that's extremely hard to debug with tcpdump because the drop happens before the packet reaches userspace.

Setting it to 0 disables the check. Some security guidelines flag this as a risk; in a Kubernetes environment with Cilium, it's the correct setting.

inotify and file descriptor limits

.NET apps use inotify heavily for config file watching (ASP.NET configuration reload, IFileProvider-based hot reload) and socket management. Default kernel limits cause EMFILE (too many open files) or ENOSPC (inotify watch limit hit) errors under load:

yaml- name: Raise inotify and fd limits
  ansible.posix.sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    sysctl_set: true
    reload: true
  loop:
    - { key: fs.inotify.max_user_instances, value: "8192"    }
    - { key: fs.inotify.max_user_watches,   value: "1048576" }
    - { key: fs.inotify.max_queued_events,  value: "32768"   }
    - { key: fs.file-max,                   value: "1048576" }

fs.inotify.max_user_instances — maximum inotify instances per user. Default is 128. With 200+ pods per node, each potentially creating multiple inotify instances, you hit this fast. 8192 gives headroom.

fs.inotify.max_user_watches — maximum files watched per user. Default is 65536. ASP.NET apps watching their wwwroot and config directories can consume thousands of watches each. 1048576 (1M) is generous but not unreasonable.

fs.inotify.max_queued_events — if the kernel generates events faster than the application processes them, events are queued up to this limit. When the queue fills, an IN_Q_OVERFLOW event is generated and subsequent events are dropped. 32768 prevents this under normal load.

fs.file-max — system-wide file descriptor limit. This is separate from the per-process limit (ulimit -n). With many containers, each holding open sockets and files, the system limit can be hit. 1048576 is 1M file descriptors system-wide.

These live in group_vars/all.yaml so they're consistent across all clusters.

k3s install arguments

k3s is installed with Flannel disabled and kube-proxy disabled — Cilium replaces both:

yaml- name: Install k3s
  ansible.builtin.shell: |
    curl -sfL {{ k3s_install_url }} | \
      INSTALL_K3S_VERSION={{ k3s_version }} \
      K3S_TOKEN={{ k3s_token }} \
      sh -s - \
        --flannel-backend=none \
        --disable-kube-proxy \
        --disable traefik \
        --disable servicelb \
        --kubelet-arg=max-pods={{ k3s_max_pods }}
  args:
    creates: /usr/local/bin/k3s

creates: makes this idempotent — if the binary exists, the task is skipped.

Each flag explained:

--flannel-backend=none — disables k3s's built-in CNI (Flannel). Without this, k3s installs Flannel and creates its own CNI configuration. Cilium needs a clean slate: no existing CNI configuration in /etc/cni/net.d/.

--disable-kube-proxy — disables kube-proxy. Cilium in kubeProxyReplacement=true mode implements Service routing via eBPF, which is more efficient and has better observability. Running both would cause conflicts.

--disable traefik — removes the default ingress controller. We use APISIX or nginx-ingress deployed via Helm, which gives us more control over ingress configuration and upgrade timing.

--disable servicelb — removes Klipper ServiceLB, k3s's built-in LoadBalancer implementation. We use Cilium L2 announcements instead, which integrates with our existing network configuration.

--kubelet-arg=max-pods={{ k3s_max_pods }} — default is 110 pods per node. With the k3s_max_pods variable set to 250 in group_vars/, nodes handle 200+ pods without pressure. Increase this only if your node has the RAM for it — each pod has baseline overhead (~10-20MB depending on sidecars).

Variables

yaml# group_vars/k3s_nodes.yaml
k3s_version: v1.29.4+k3s1
k3s_install_url: https://get.k3s.io
k3s_token: "{{ vault_k3s_token }}"  # from Ansible Vault
k3s_max_pods: 250
cilium_version: 1.15.4
node_ip: "{{ ansible_default_ipv4.address }}"
k3s_node_name: "{{ inventory_hostname }}"

k3s_token comes from Ansible Vault. Never hardcode tokens in group_vars — even in internal repos.

Cilium install before waiting for node Ready

The node won't reach Ready state until a CNI is present. So Cilium must be installed immediately after k3s, before Flux bootstraps:

yaml- name: Install Cilium via Helm
  ansible.builtin.shell: |
    helm upgrade --install cilium cilium/cilium \
      --version {{ cilium_version }} \
      --namespace kube-system \
      --set ipam.mode=kubernetes \
      --set kubeProxyReplacement=true \
      --set k8sServiceHost={{ node_ip }} \
      --set k8sServicePort=6443 \
      --set operator.replicas=1 \
      --set l2announcements.enabled=true \
      --set hubble.enabled=true \
      --set hubble.relay.enabled=true \
      --set hubble.ui.enabled=true
  environment:
    KUBECONFIG: /etc/rancher/k3s/k3s.yaml

k8sServiceHost and k8sServicePort tell Cilium where the Kubernetes API is — necessary when kube-proxy is disabled, because Cilium can't rely on the kubernetes service ClusterIP being reachable before eBPF is set up.

operator.replicas=1 is appropriate for single-node clusters. For multi-node setups, set it to 2 for HA.

Then wait for the DaemonSet and the node:

yaml- name: Wait for Cilium DaemonSet to be ready
  ansible.builtin.shell: |
    kubectl -n kube-system rollout status daemonset/cilium --timeout=120s
  environment:
    KUBECONFIG: /etc/rancher/k3s/k3s.yaml
  retries: 5
  delay: 15
  register: cilium_ready
  until: cilium_ready.rc == 0
  changed_when: false

- name: Wait for node Ready
  ansible.builtin.shell: |
    k3s kubectl get node {{ k3s_node_name }} \
      -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
  register: node_ready
  retries: 30
  delay: 10
  until: node_ready.stdout == "True"
  changed_when: false

Wait for the DaemonSet rollout first, then for the node condition. If you skip the DaemonSet wait and go straight to node Ready, you can get a false positive: the node might show Ready for a brief window before Cilium's network policies are enforced.

Post-install: fetch kubeconfig

After the node is ready, copy the kubeconfig to the Ansible control host so you can use it immediately:

yaml- name: Fetch kubeconfig
  ansible.builtin.fetch:
    src: /etc/rancher/k3s/k3s.yaml
    dest: "{{ playbook_dir }}/kubeconfigs/{{ inventory_hostname }}.yaml"
    flat: true

- name: Update server address in kubeconfig
  ansible.builtin.replace:
    path: "{{ playbook_dir }}/kubeconfigs/{{ inventory_hostname }}.yaml"
    regexp: 'https://127\.0\.0\.1:6443'
    replace: "https://{{ node_ip }}:6443"
  delegate_to: localhost

k3s writes 127.0.0.1 as the server address. Replace it with the actual node IP so the kubeconfig works from outside the node.

Idempotency

Every task is idempotent:

swapoff -a with changed_when: false — always exits 0, no false changes reported
modprobe — if module is already present, no-op
sysctl — Ansible checks current value, only writes if different
k3s install uses creates: guard — skipped if binary exists
Helm upgrade --install — creates or upgrades, never fails on existing release

Run the playbook twice on the same node: second run changes nothing and takes ~30 seconds.

Full playbook skeleton

yaml---
- name: Prepare node and install k3s with Cilium
  hosts: k3s_nodes
  become: true
  vars_files:
    - vault.yaml

  pre_tasks:
    - name: Disable swap
      # ...
    - name: Remove swap from /etc/fstab
      # ...
    - name: Load kernel modules
      # ...
    - name: Persist kernel modules
      # ...
    - name: Set sysctl params
      # ...
    - name: Raise inotify and fd limits
      # ...

  tasks:
    - name: Add Helm repo for Cilium
      ansible.builtin.shell: helm repo add cilium https://helm.cilium.io && helm repo update
      changed_when: false

    - name: Install k3s
      # ...

    - name: Install Cilium via Helm
      # ...

    - name: Wait for Cilium DaemonSet to be ready
      # ...

    - name: Wait for node Ready
      # ...

  post_tasks:
    - name: Fetch kubeconfig
      # ...
    - name: Update server address in kubeconfig
      # ...

The tasks section is intentionally short. Pre-tasks handle OS-level prep; the CNI bootstrap is in tasks; post-tasks deal with outputting artifacts. This structure makes it easy to run only part of the playbook with --tags if needed.

weblog