Kubespray: a Dedicated Control Plane and Two Workers to Offload a Saturated Node

Published: 2026-06-23

A single-node test cluster is fine until one box is doing everything — the control plane, etcd, and a heavy database like ClickHouse — and the node runs out of CPU. The symptom looks like slow storage; the cause is usually contention. The fix isn't a faster disk, it's splitting roles across VMs: a dedicated control plane that nothing else competes with, plus workers sized for the workload. This walks through building that layout with Kubespray on Ubuntu 24.04 — one control-plane+etcd node and two workers, Cilium with kube-proxy replacement — and, just as importantly, making the load actually spread once the nodes exist.

Diagnose first: is it really the disk?

Before adding hardware, confirm what's saturated. On a node that "feels like slow disks", check whether tasks are stalling on I/O or on CPU — without installing anything:

bashcopynproc                                       # how many cores you actually have
cat /proc/loadavg                           # load vs cores
cat /proc/pressure/cpu /proc/pressure/io    # PSI: % of time stalled, last 10/60/300s
vmstat 1 3                                  # 'wa' = iowait%, 'b' = procs blocked on I/O

If wa is low (single digits), b is ~0, and PSI io is low while PSI cpu is high and load exceeds your core count — the disk is fine and the CPU is the bottleneck. What feels like disk lag is everything queuing behind busy cores: etcd's fsync, the kubelet, the API server, all waiting for CPU. Faster storage changes nothing here. (If instead wa is high and processes sit in D state, then it really is storage — a different problem.)

The cure for CPU saturation is more cores and fewer things fighting over them — which is exactly what a dedicated control plane plus workers buys you.

The layout: one control plane, two workers

copycp1       10.0.0.10   control-plane + etcd   (dedicated, tainted)
worker1   10.0.0.11   heavy workload (the DB)
worker2   10.0.0.12   everything else

Three roles that were colliding on one node now have room:

• cp1 runs only the control plane and the single etcd member. It stays tainted — etcd's fsync and the API server no longer compete with application CPU. This node can be small (a control plane for a modest cluster wants ~2–4 cores); give it a fast, ideally separate disk for /var/lib/etcd.
• worker1 / worker2 carry the workloads. Two of them means twice the schedulable CPU and room to keep the heavy hitter away from everything else.

This is deliberately not three control-plane nodes. Three would give etcd HA (a quorum of 3 tolerates one failure) but puts control-plane overhead back on every node — the opposite of isolating it. On a test stand the goal is offload, not HA. One etcd member is fine; growing into a real 3-node HA control plane later is an inventory edit (last section).

The trap: more nodes don't spread load by themselves

Adding workers does nothing on its own. The scheduler places pods by their resource requests; a pod with no requests can land anywhere, and several can pile onto the same worker. Worse, a single large pod does not split — one ClickHouse pod uses one node's cores no matter how many workers exist. You get offload only if you:

1. Give every heavy pod real requests/limits so the scheduler can place and bound them.
2. Keep the dominant workload on its own node, so it isn't sharing cores with the rest.

Both are shown after the install. Skip them and you've just moved the saturation onto one worker.

The control machine

Kubespray runs from a control machine (your laptop, or a small admin VM) and pushes over SSH. It does not need to be one of the nodes. The thing that bites people here is the Ansible version: Kubespray pins a specific ansible-core, and a newer one from your distro fails in confusing ways. Use the pinned stack in a venv.

bashcopy# Pin to a release tag — never build a stand off main
git clone -b v2.31.0 https://github.com/kubernetes-sigs/kubespray.git
cd kubespray

python3 -m venv .venv && source .venv/bin/activate
pip install -U pip
pip install -r requirements.txt   # pins ansible==11.13.0 → ansible-core 2.18.x

v2.31.0 (April 2026) ships Kubernetes up to 1.35.4 and Cilium 1.19.3, and supports Ubuntu 24.04 (cgroup v2, which is what current Kubespray expects). requirements.txt pins ansible==11.13.0, i.e. ansible-core 2.18.x; the role metadata declares requires_ansible: ">=2.18.0,<2.19.0". Do not apt install ansible — that gives you 2.19+ and the syntax check fails before anything deploys.

You also need SSH key access to all three nodes and a sudo-capable login. If the login is in the sudo group but without NOPASSWD, pass -K and type the sudo password once (below).

Inventory

Copy the sample and edit it — the sample carries hundreds of sensible defaults you want to keep.

bashcopycp -rfp inventory/sample inventory/mycluster

inventory/mycluster/inventory.ini for the one-control-plane, two-worker layout:

inicopy[kube_control_plane]
cp1 ansible_host=10.0.0.10 ip=10.0.0.10 ansible_user=ubuntu etcd_member_name=etcd1

[etcd:children]
kube_control_plane

[kube_node]
worker1 ansible_host=10.0.0.11 ip=10.0.0.11 ansible_user=ubuntu
worker2 ansible_host=10.0.0.12 ip=10.0.0.12 ansible_user=ubuntu

What the fields mean:

• ansible_host= — the address Ansible SSHes to.
• ip= — the address kubelet and etcd bind to. Set it explicitly; on a multi-NIC VM the autodetected IP is often the wrong one (the CNI or a bridge interface).
• ansible_user= — SSH login. Can also be set globally with --user.
• [etcd:children] kube_control_plane puts etcd on the control-plane node(s). Because [kube_control_plane] has exactly one host, etcd gets exactly one member — an odd count, as etcd requires.

The key difference from a "use every node" layout: cp1 is not listed under [kube_node]. That's what keeps it a dedicated control plane — it joins as a control-plane node (kubelet runs, but it carries the NoSchedule taint) and never becomes a scheduling target for your workloads.

Cluster configuration

Three files under inventory/mycluster/group_vars/ carry the decisions.

k8s_cluster/k8s-cluster.yml

yamlcopy# Pin the version explicitly. Kubespray computes a default dynamically,
# so leaving it unset means a minor bump can change under you on the next run.
kube_version: v1.35.4

# The sample default is calico — switch to cilium to match the rest of the fleet
kube_network_plugin: cilium

container_manager: containerd

# Pull the admin kubeconfig (and a matching kubectl) back to the control machine
kubeconfig_localhost: true
kubectl_localhost: true

# Add any address you'll reach the API through to the apiserver cert SANs
supplementary_addresses_in_ssl_keys:
  - 10.0.0.10

kube_proxy_mode lives in this file too, but it becomes irrelevant once Cilium replaces kube-proxy (next section) — Kubespray skips installing kube-proxy entirely.

k8s_cluster/addons.yml

yamlcopyhelm_enabled: true                   # default false; handy on a stand where you'll helm install things
local_path_provisioner_enabled: true # Kubespray ships NO StorageClass by default — see Storage below

k8s_cluster/k8s-net-cilium.yml

yamlcopycilium_version: "1.19.3"

# Cilium handles service load-balancing in eBPF; kube-proxy is not installed at all
cilium_kube_proxy_replacement: true

# On-prem L2 LoadBalancer (optional — leave false until you need LB IPs)
cilium_l2announcements: false
cilium_loadbalancer_ip_pools: []

How kube-proxy actually gets removed

When kube_network_plugin: cilium and cilium_kube_proxy_replacement is truthy, Kubespray adds addon/kube-proxy to kubeadm's skipped init phases — kube-proxy is never created. Both conditions are required: set only the CNI and you still get kube-proxy; set only the Cilium flag without selecting Cilium and it does nothing. In v2.31.0 the accepted value is the boolean true (legacy Cilium strings like "strict" are deprecated upstream in 1.19). You do not also need kube_proxy_remove: true — that's the generic fallback for other CNIs and is redundant here.

Running the install

bashcopyansible-playbook -i inventory/mycluster/inventory.ini \
  --become --become-user=root \
  --ask-become-pass \
  cluster.yml

--become escalates to root on the hosts; --ask-become-pass (-K) prompts once for the sudo password — needed when the login has sudo but not NOPASSWD. If SSH itself uses a password rather than a key, add -k too.

cluster.yml is the full build: it preps the OS (disables swap, loads br_netfilter/overlay, sets sysctls), installs containerd, runs kubeadm init on cp1, joins both workers, lays down etcd, and installs Cilium. On three VMs it takes roughly 20–30 minutes. The run is idempotent — re-running converges rather than rebuilding, so a failed run is usually safe to re-launch after you fix the cause.

Kubespray handles swap, sysctls, kernel modules, containerd, kubeadm, etcd, PKI, and Cilium. You are responsible for: python3 on each node, SSH key + sudo access, the firewall (Kubespray does not manage it — sudo ufw disable on a test stand, or open the k8s/etcd/Cilium ports), time sync (chrony — etcd is skew-sensitive), registry reachability, and unique hostnames.

Keep the control plane dedicated, and isolate the heavy workload

This is the step that turns three VMs into actual offload.

Do not untaint cp1. Kubespray taints control-plane nodes node-role.kubernetes.io/control-plane:NoSchedule by default — here that's exactly what you want. (On a 2-VM stand you'd remove it to reclaim the node; here the whole point is to keep workloads off the control plane.) Confirm it's still tainted:

bashcopykubectl get node cp1 -o jsonpath='{.spec.taints}'
# [{"effect":"NoSchedule","key":"node-role.kubernetes.io/control-plane"}]

Pin the heavy workload to one worker. Taint worker1 so only the database tolerates it, then target it:

bashcopykubectl taint nodes worker1 dedicated=db:NoSchedule

yamlcopy# in the DB's pod spec
spec:
  nodeSelector:
    kubernetes.io/hostname: worker1
  tolerations:
    - key: dedicated
      value: db
      effect: NoSchedule
  containers:
    - name: clickhouse
      resources:
        requests: { cpu: "6", memory: 24Gi }   # without requests the scheduler can't reason about capacity
        limits:   { cpu: "10", memory: 32Gi }   # without limits it's a noisy neighbour

Everything else now schedules onto worker2 (and never the control plane). For workloads that have several replicas, spread them with anti-affinity instead of pinning:

yamlcopyaffinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchLabels: { app: my-api }
          topologyKey: kubernetes.io/hostname

And give the rest of your noisy pods (log shippers, sidecars) explicit CPU requests — pods without them are the usual reason one node quietly fills up.

Storage on a multi-node cluster

Going multi-node changes how storage behaves. Kubespray installs no StorageClass by default (unlike k3s) — kubectl get sc is empty until you choose one. The options, and how each behaves when a pod moves between nodes:

On a CPU-constrained stand the ranking is easy: local-path (cheapest) or NFS (light, RWX) over Longhorn — Longhorn's synchronous replication and per-volume engines add exactly the CPU you're trying to free up. local_path_provisioner_enabled: true (set above) creates a default StorageClass named local-path; its PVs are node-pinned, so a pinned DB pod and its data stay together on worker1 anyway.

Getting the kubeconfig

With kubeconfig_localhost: true, the admin config lands on the control machine:

bashcopyexport KUBECONFIG=$PWD/inventory/mycluster/artifacts/admin.conf
kubectl get nodes -o wide

copy# NAME      STATUS   ROLES           AGE   VERSION    INTERNAL-IP
# cp1       Ready    control-plane   9m    v1.35.4    10.0.0.10
# worker1   Ready    <none>          7m    v1.35.4    10.0.0.11
# worker2   Ready    <none>          7m    v1.35.4    10.0.0.12

The server: line in admin.conf points at cp1's IP. If you reach the cluster through a different address — a NAT, a hostname, a jump host — edit server: to match, and make sure that address was in supplementary_addresses_in_ssl_keys, or the API server's certificate has no SAN for it and TLS refuses the connection.

Day-2: scale workers, grow into HA

The same inventory drives the lifecycle playbooks:

bashcopy# Add worker3: append it to [kube_node], then
ansible-playbook -i inventory/mycluster/inventory.ini --become -K scale.yml

# Cleanly drain and remove a node
ansible-playbook -i inventory/mycluster/inventory.ini --become -K \
  -e node=worker2 remove-node.yml

# Rolling, one-node-at-a-time version upgrade (bump kube_version first)
ansible-playbook -i inventory/mycluster/inventory.ini --become -K upgrade-cluster.yml

# Wipe k8s/etcd/CNI off every host, back to bare OS — destructive
ansible-playbook -i inventory/mycluster/inventory.ini --become -K reset.yml

When the stand outgrows a single control plane, grow into HA: add two more hosts to [kube_control_plane] so etcd has three members (the first count that tolerates a failure), then run upgrade-cluster.yml to expand etcd and the control plane. Adding worker capacity is just scale.yml after appending the host. The inventory is the cluster — there's no rebuild.

What can go wrong

Added a worker, but nothing moved onto it

Pods without resource requests don't get rebalanced, and existing pods don't migrate on their own. Set requests on the heavy pods, use anti-affinity or a node taint to place them, and delete/redeploy so the scheduler re-places them. A single large pod will still occupy one node — it can't be split.

Accidentally untainted the control plane

Old muscle memory from single/two-node clusters. If you ran kubectl taint nodes cp1 ...-, re-apply it so workloads leave the control plane:

bashcopykubectl taint nodes cp1 node-role.kubernetes.io/control-plane=:NoSchedule

"ansible-core version is not supported"

You ran Ansible outside the venv (or apt install ansible, which is 2.19+). Recreate the venv and pip install -r requirements.txt — the pinned ansible==11.13.0 is ansible-core 2.18.x, the only supported line for v2.31.0.

Gathering facts fails: /usr/bin/python3 not found

A minimal or cloud Ubuntu image without Python. Kubespray needs python3 on every node before it can do anything: ssh ubuntu@10.0.0.12 sudo apt-get install -y python3.

Missing sudo password

The login has sudo but not NOPASSWD. Add -K (--ask-become-pass). If the SSH transport is also password-based, add -k.

etcd won't go healthy / install hangs at the etcd join

Almost always a clock skew or an even etcd count. Keep exactly one host in [etcd] here, and make sure chrony is running on all nodes — etcd rejects peers whose clocks disagree.

Pods stuck Pending, or kube-proxy is still present

bashcopysudo ufw status        # should be inactive on a test stand — it blocks CNI/etcd otherwise
kubectl -n kube-system get pods | grep kube-proxy   # should be empty with Cilium replacement
grep -r cilium_kube_proxy_replacement inventory/mycluster/group_vars/   # must be true
grep -r kube_network_plugin            inventory/mycluster/group_vars/   # must be cilium
kubectl -n kube-system get pods -l k8s-app=cilium -o wide

Summary

• The symptom was "slow disks"; the cause was CPU contention. Check wa/PSI before buying storage — a saturated single node makes everything, including etcd, feel like disk lag.
• Split roles across VMs: a dedicated, tainted control plane + two workers. etcd and the API server stop competing with application CPU; two workers double the schedulable capacity.
• Nodes alone don't spread load. Set requests/limits, pin the dominant workload to its own worker, and use anti-affinity for replicated apps — a single large pod can't be split.
• Cilium with cilium_kube_proxy_replacement: true removes kube-proxy entirely — but only when kube_network_plugin: cilium is set too.
• Pick storage by CPU budget: local-path or NFS over Longhorn on a constrained stand. Kubespray ships no StorageClass by default.
• The inventory is the cluster. Add workers with scale.yml; grow to a 3-node HA control plane later by moving hosts into [kube_control_plane] — an edit, not a rebuild.