← weblog / 2026-07-13-bare-metal-k8s-cilium-l2-maintenance
2026-07-13·kubernetesciliumbare-metaldevops·6 min read

Bare-metal k8s: keep Cilium L2 LoadBalancer & node maintenance reliable

Published: 2026-07-13

On bare-metal Kubernetes, the first production problem is not scheduling pods — it is keeping the external IP reachable when the node behind it changes. This post shows how to make Cilium L2 LoadBalancer IPs stable, what node-level networking settings matter, and how to keep upgrades and reboots from breaking service reachability.

The problem

A bare-metal cluster can allocate a LoadBalancer service IP, but no cloud controller puts that IP on the wire. That leaves you with a service whose EXTERNAL-IP is assigned in Kubernetes, but the rest of the network still does not know where to send packets.

This becomes worse when nodes are rebooted or drained: the service may move, but clients still use stale ARP/NDP entries. On a physical LAN, a single wrong announcement or missing rp_filter setting can make the load balancer address disappear.

bash$ kubectl get svc apisix -n ingress-apisix
NAME    TYPE           CLUSTER-IP    EXTERNAL-IP      PORT(S)
apisix  LoadBalancer   10.96.45.23   172.16.57.193    80:32000/TCP

The IP is there, but the cluster still has to own it on the wire.

How it works

Cilium's L2 announcement feature solves the traffic path at the network layer.

  • CiliumLoadBalancerIPPool defines the pool of host-local IPs allowed for LoadBalancer services.
  • CiliumL2AnnouncementPolicy controls which nodes advertise those IPs and on which host interfaces.
  • When a service is created, Cilium allocates an IP from the pool and starts answering ARP/NDP for that address.
  • If the service moves to another node, Cilium retires the old announcement and creates a new one from the same IP.

The key point is that Kubernetes only knows the service IP. The real network reachability depends on Cilium managing ARP/NDP on the physical interface.

Why host networking tuning matters

On bare metal, Linux packet forwarding and reverse path filtering are the usual gotchas:

  • net.ipv4.ip_forward=1 is required for the node to forward traffic between interfaces.
  • net.ipv4.conf.all.rp_filter=0 and net.ipv4.conf.default.rp_filter=0 are required when the node receives traffic for a service IP and forwards it to a pod via Cilium.
  • net.ipv4.conf.<iface>.rp_filter=0 is useful when the announcement interface is not the same as the primary route.

If rp_filter is 1, the kernel may drop valid replies because the return path does not match the incoming interface.

When you need a fixed IP

Assigning a specific loadBalancerIP is the usual production pattern for landing DNS, firewall rules, or port-based access control.

yamlapiVersion: v1
kind: Service
metadata:
  name: apisix
  namespace: ingress-apisix
spec:
  type: LoadBalancer
  loadBalancerIP: 172.16.57.193
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: apisix

Cilium will allocate this exact IP from the pool, or fail if it is already taken.

Setup

Step 1: define the pool

Choose a block that is inside your LAN and outside DHCP.

yamlapiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
  name: baremetal-lb-pool
spec:
  blocks:
    - cidr: 172.16.57.192/28   # 14 usable IPs

Apply it with kubectl apply -f cilium-lb-pool.yaml.

Step 2: restrict announcements to the right nodes and interface

Pick the physical interface that reaches the upstream router or switch. If your node uses bonded interfaces, use the bond name.

yamlapiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
  name: baremetal-l2-policy
spec:
  serviceSelector:
    matchLabels:
      app.kubernetes.io/managed-by: Flux
  nodeSelector:
    matchLabels:
      kubernetes.io/os: linux
      node-role.kubernetes.io/worker: ""
  interfaces:
    - eth0
  externalIPs: true
  loadBalancerIPs: true

The interfaces field is the most important guard. If Cilium announces on the wrong NIC or on a bridge interface that does not carry the LAN, the address will never reach clients.

Step 3: tune node networking

On every bare-metal node, set these sysctls:

bashcat <<'EOF' | sudo tee /etc/sysctl.d/99-cilium-l2.conf
net.ipv4.ip_forward=1
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.eth0.rp_filter=0
EOF
sudo sysctl --system

If your nodes have multiple physical NICs or VLANs, repeat net.ipv4.conf.<iface>.rp_filter=0 for the chosen interface.

Step 4: verify Cilium sees the pool and policy

bashkubectl get ciliumloadbalancerippool -A
kubectl get ciliuml2announcementpolicy -A
kubectl describe ciliuml2announcementpolicy baremetal-l2-policy

The output should show the pool in Active state and the policy bound to worker nodes.

Step 5: provision a LoadBalancer service

Create a service using loadBalancerIP, then verify the external IP appears:

bashkubectl apply -f apisix-service.yaml
kubectl get svc apisix -n ingress-apisix -o wide

Then test reachability from outside the cluster:

bashcurl -I http://172.16.57.193

If packets do not arrive, check ARP on the upstream router or client:

baship neigh show 172.16.57.193

A healthy result looks like:

text172.16.57.193 dev eth0 lladdr 52:54:00:12:34:56 REACHABLE

Step 6: make maintenance safe

For bare-metal maintenance, use the same drain/reboot workflow as for any Kubernetes node, but keep the load balancer IP behavior in mind.

bashkubectl drain node-01 --ignore-daemonsets --delete-emptydir-data
sudo reboot

On a reboot, Cilium stops announcing the address. The service will become unreachable until another node takes ownership. To keep failover predictable:

  • use a single pool shared across the cluster
  • pin loadBalancerIP for production services
  • mark only worker nodes as announcers
  • keep ARP cache timeouts short on upstream devices if you control them

If the node enters the cluster again, Cilium should restore the announcement automatically.

What can go wrong

The IP stays pending. That usually means the service is not matching the policy, or the pool does not have free addresses.

bashkubectl get svc apisix -n ingress-apisix -o jsonpath='{.status.loadBalancer.ingress}'
  • Check CiliumLoadBalancerIPPool for free blocks.
  • Check CiliumL2AnnouncementPolicy selectors and interfaces.
  • Check that the service label selector matches the policy.

Clients see the wrong MAC. If two nodes announce the same IP, upstream ARP caches can alternate between MACs.

  • Use ip neigh show <ip> on a client.
  • Use sudo tcpdump -nei eth0 arp or icmp on the candidate nodes.
  • Verify policies are not overlapping and the IP pool is unique.

After reboot, the service is still unreachable. The cluster may have recovered, but the router or client still has stale ARP.

  • Clear the upstream ARP entry if possible.
  • Reduce the ARP cache timeout on your switch/router, or use a shorter ip neigh entry on Linux clients.
  • On the Kubernetes node, kubectl get pods -n kube-system -l k8s-app=cilium and kubectl logs can confirm Cilium restarted cleanly.

The announcement happens on the wrong interface. This is the most common bare-metal bug.

bashkubectl describe ciliuml2announcementpolicy baremetal-l2-policy
  • Confirm interfaces: points to the LAN-facing device.
  • Confirm the node's firmware or cloud-init did not rename the interface.
  • Confirm ip route and ip addr show the chosen interface as the one used for external traffic.

Summary

  • Bare-metal Kubernetes needs a network-level owner for LoadBalancer IPs; Cilium L2 announcements provide it.
  • Define a pool, restrict announcements to the LAN-facing interface, and pin service IPs for production.
  • Tune ip_forward and rp_filter on bare-metal nodes so reply packets are not dropped.
  • Drain and reboot with the same LoadBalancer IP behavior in mind: the address may disappear until another node reacquires it.
  • Diagnosing failures is usually one of: policy mismatch, wrong interface, overlapping IP ranges, or stale ARP on the upstream network.