Traefik → APISIX on a single node: what broke during the ingress migration
Published: 2026-07-15
Traefik had been running on my k0s cluster for 224 days without a single complaint — holding 80/443 via hostNetwork, shipping JSON access logs, never once crashing. The reason for switching to APISIX wasn't "Traefik is bad," it was "I want a younger, lighter stack while I'm already cleaning up monitoring and logging." On paper, swapping ingress controllers sounds like a rolling update. On a single-node bare-metal cluster it turned into a straight cutover with downtime and a dozen findings, some of which would have silently broken production if I hadn't verified each one live.
The problem
The cluster is one physical node, k0s, no external LoadBalancer. Traefik held 80/443 via hostNetwork: true — the only option that works when a LoadBalancer Service sits in <pending> forever. 19 domains (personal site, VPN dashboard, Docker registry, Grafana, VictoriaMetrics, webmail) went through Traefik IngressRoute — its own CRD, not the standard networking.k8s.io/v1 Ingress.
The plan looked clean: install APISIX next to Traefik, run tests, flip traffic without losing a second. The plan died on the first attempt — the Kubernetes scheduler refused to place the APISIX pod:
text0/1 nodes are available: 1 node(s) didn't have free ports for the requested pod ports
Two pods with hostNetwork: true, both wanting 80 and 443, one node. Blue-green is physically impossible here without a second node. So: Traefik goes first, APISIX comes in right after, and whatever breaks along the way gets fixed live.
How it works
apisix-ingress-controller paired with the apisix/apisix chart is two components in one Helm release: the gateway itself (data plane on OpenResty) and a controller that watches Ingress objects and pushes config into APISIX via the Admin API. By default APISIX drags along etcd — not for application data, but as intermediate storage for routes between the controller and the gateway.
bashhelm repo add apisix https://apache.github.io/apisix-helm-chart
helm repo update
Then the install with the full set of flags that actually turned out to be needed (each one explained below):
bashhelm upgrade --install apisix apisix/apisix \
--namespace apisix \
-f apisix-logs-values.yaml \
--set apisix.ssl.enabled=true \
--set apisix.ssl.containerPort=443 \
--set ingress-controller.enabled=true \
--set ingress-controller.gatewayProxy.createDefault=true \
--set ingress-controller.gatewayProxy.provider.controlPlane.service.name=apisix-admin \
--set ingress-controller.gatewayProxy.provider.controlPlane.auth.adminKey.value=edd1c9f034335f136f87ad84b625c8f1 \
--set hostNetwork=true \
--set service.http.containerPort=80 \
--set service.type=ClusterIP \
--set 'securityContext.capabilities.add[0]=NET_BIND_SERVICE' \
--set securityContext.runAsUser=0 \
--set updateStrategy.type=Recreate \
--set etcd.replicaCount=1 \
--set etcd.persistence.enabled=false \
--wait --timeout 180s
Every line here is the cost of one broken attempt.
Step by step
Step 1: HTTPS is off by default
The first install went through cleanly, pods came up, but HTTPS didn't answer at all. The cause: apisix.ssl.enabled in the chart's values defaults to false. Without it the gateway simply doesn't open a listener on 443, no matter what the Service or hostNetwork say.
bashhelm show values apisix/apisix | grep -A3 "^ ssl:"
yamlssl:
enabled: false # there it is
containerPort: 9443
I spent about twenty minutes staring at ports and Services before it occurred to me to check the feature flag itself instead of its plumbing.
Step 2: the right key for the TLS port
By analogy with service.http.containerPort you'd expect service.tls.containerPort to exist. It doesn't — I checked with helm show values rather than guessing from memory:
yamlservice:
tls:
servicePort: 443
# no containerPort here at all
The HTTPS listener port lives somewhere else — apisix.ssl.containerPort, in a separate SSL section, not the Service section. Same concept, two different spots in the values tree — a classic trap in Helm charts whose structure evolved faster than its docs.
Step 3: commas in --set break the JSON string
JSON access logs were part of the reason for the whole switch. I set the format:
bash--set apisix.nginx.logs.accessLogFormat='{"time":"$time_iso8601","host":"$http_host","status":$status}'
The install went through with no errors. The logs inside the pod looked like this:
text["time":"2026-07-15T07:26:53+00:00" "host":"antonnovikov.com" "status":200]
Square brackets instead of curly, spaces instead of commas — valid JSON turned into garbage without a single error from Helm. The cause: --set parses its value on unescaped commas, treating them as separators between different key=value pairs. The JSON-format string is nothing but commas — Helm silently turned one string into a YAML list of separate "key":"value" pairs, and the chart template rendered the list however it pleased.
The right approach is to never touch --set for values with commas, and pass them through a file instead:
yaml# apisix-logs-values.yaml
apisix:
nginx:
logs:
accessLogFormat: '{"time":"$time_iso8601","remote_addr":"$remote_addr","host":"$http_host","request_uri":"$request_uri","status":$status}'
accessLogFormatEscape: json
bashhelm upgrade apisix apisix/apisix -n apisix -f apisix-logs-values.yaml --reuse-values
Step 4: the pod can't bind to port 80
text2026/07/15 07:26:53 [emerg] 1#1: bind() to 0.0.0.0:80 failed (13: Permission denied)
Ports below 1024 need either root or the CAP_NET_BIND_SERVICE capability. Traefik already had this solved in its own chart values — I just forgot to carry the same setting over for APISIX:
yamlsecurityContext:
capabilities:
add: [NET_BIND_SERVICE]
runAsNonRoot: false
runAsUser: 0
Step 5: etcd wants a PersistentVolumeClaim that doesn't exist
textFailedBinding: no persistent volumes available for this claim and no storage class is set
On bare-metal k0s with no dynamic provisioning, the chart's default etcd tries to request a PVC and hangs in Pending forever. etcd.persistence.enabled=false is fine here — the only data living in it is runtime route config, not anything that needs to survive a pod restart: the controller rebuilds it from Ingress objects in seconds.
Step 6: GatewayProxy doesn't create itself
ingress-controller.enabled=true turns the controller on, but without a GatewayProxy it has nowhere to push config:
textINFO provider.client syncing all resources
INFO provider.client no GatewayProxy configs provided
GatewayProxy is a separate CRD that links an IngressClass to a specific gateway's Admin API (address, port, auth key). The gatewayProxy.createDefault=true flag creates it automatically, but you still need to explicitly point it at the Admin API Service name and key:
bash--set ingress-controller.gatewayProxy.createDefault=true \
--set ingress-controller.gatewayProxy.provider.controlPlane.service.name=apisix-admin \
--set ingress-controller.gatewayProxy.provider.controlPlane.auth.adminKey.value=edd1c9f034335f136f87ad84b625c8f1
The key edd1c9f034335f136f87ad84b625c8f1 is APISIX's own default demo key; it matches between gateway and controller out of the box, but it's better to set it explicitly rather than rely on the defaults lining up.
Step 7: hostNetwork + rolling update — a dead end on one node
Every subsequent helm upgrade hung for minutes:
textWaiting for daemon set... 0 out of 1 new pods have been updated
The default deploy strategy is RollingUpdate: bring up the new pod first, then kill the old one. With hostNetwork: true and one node that's impossible — the new pod wants the same 80/443 the old one is still holding. Same problem as Step 0, except now it hits every update, not just the initial install alongside Traefik.
bash--set updateStrategy.type=Recreate
Recreate kills the old pod first, then creates the new one — a second of downtime on every update, but no infinite hang.
Step 8: 404s on CSS and JS after switching routes
The site came up, domains returned 200, but the browser was full of:
textRefused to apply style from '.../assets/css/styles.css' because its MIME type
('text/html') is not a supported stylesheet MIME type
The cause was in my own migration logic. In Traefik, seven subdomains (cv., weblog., status., and others) share one site pod via an addPrefix middleware: cv.antonnovikov.com/ gets turned into /cv/ before proxying. Translating the routes to standard Ingress, I applied the rewrite annotation to every path without exception — including /assets/*, which physically lives in the shared folder with no prefix. That turned into /cv/assets/... instead of /assets/... — a 404, with the backend helpfully serving an HTML error page instead of CSS, hence the text/html MIME type.
Traefik handled this with a separate route rule for /assets with no rewrite — I simply missed it in the first pass of the migration. The fix is two Ingress objects per domain instead of one: one with no rewrite for static assets and its own prefix, and one with rewrite for everything else.
yaml# passthrough — no rewrite, for /assets and static files
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: cv-subdomain-passthrough
annotations:
k8s.apisix.apache.org/http-to-https: "true"
spec:
ingressClassName: apisix
rules:
- host: cv.antonnovikov.com
http:
paths:
- path: /assets
pathType: Prefix
backend: {service: {name: antonnovikov-site, port: {number: 80}}}
- path: /cv
pathType: Prefix
backend: {service: {name: antonnovikov-site, port: {number: 80}}}
---
# catchall — with rewrite, for root and everything else
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: cv-subdomain
annotations:
k8s.apisix.apache.org/http-to-https: "true"
k8s.apisix.apache.org/rewrite-target-regex: "^/(.*)"
k8s.apisix.apache.org/rewrite-target-regex-template: "/cv/$1"
spec:
ingressClassName: apisix
rules:
- host: cv.antonnovikov.com
http:
paths:
- path: /
pathType: Prefix
backend: {service: {name: antonnovikov-site, port: {number: 80}}}
The standard Ingress API resolves priority on its own — a more specific path (/assets, /cv) wins over the general /, nothing extra needed configuring.
Step 9: two domains turned out to be redirects, not the site
tg.antonnovikov.com and sofyanovikova.ru looked like ordinary routes to the site backend in the Traefik dump. Digging into the middleware revealed they were actually permanent redirects to Telegram channels (redirectRegex with a .* pattern → a static URL), not proxying. The first pass of the migration honestly copied them as regular domains — and they would have stopped redirecting and started serving the site instead. APISIX has a ready-made annotation for this, no separate CRD needed:
yamlmetadata:
annotations:
k8s.apisix.apache.org/http-redirect: "https://t.me/example"
k8s.apisix.apache.org/http-redirect-code: "301"
Step 10: metrics exist, just not the right ones
apisix_nginx_http_current_connections showed up in /apisix/prometheus/metrics right away. apisix_http_status, apisix_http_latency, apisix_bandwidth — no, even though traffic was already flowing. Base nginx-level metrics are always written, but per-request metrics need the prometheus plugin enabled globally:
yamlapiVersion: apisix.apache.org/v2
kind: ApisixGlobalRule
metadata:
name: prometheus-metrics
namespace: apisix
spec:
ingressClassName: apisix
plugins:
- name: prometheus
enable: true
Plus the exporter itself listens on 127.0.0.1:9091 inside the pod by default — unreachable from outside via a Kubernetes Service, only through port-forward, which masks the problem while debugging. Had to explicitly rebind it:
yamlapisix:
pluginAttrs:
prometheus:
export_addr:
ip: "0.0.0.0"
port: 9091
And create a Service the chart doesn't set up on its own:
yamlapiVersion: v1
kind: Service
metadata:
name: apisix-metrics
namespace: apisix
spec:
selector:
app.kubernetes.io/name: apisix
app.kubernetes.io/instance: apisix
ports:
- name: metrics
port: 9091
targetPort: 9091
Step 11: --reuse-values after a failed update silently rolled everything back
After one of the JSON-format fixes I ran helm upgrade --reuse-values --set updateStrategy.type=Recreate — and the log format reverted to the broken shape from Step 3. --reuse-values pulls values not from the last upgrade attempt, but from the last successfully deployed release. One of the intermediate attempts had been marked failed (the --wait timeout expired even though the resources had applied) — and Helm rolled back to the release before it, silently burying the fix. Lesson: after --reuse-values, always cross-check with helm get values -n <ns> -a rather than trust that the upgrade applied what you just fixed.
What can go wrong
A domain returns 200 but the content is wrong. Check not just the status code but the actual <title> or a specific file — a 404 from the backend can masquerade as 200 if the frontend catches the error with its own SPA page.
The ingress controller keeps logging no GatewayProxy configs provided forever. GatewayProxy either wasn't created or points at the wrong Service. Check: kubectl get gatewayproxy -n apisix and kubectl logs -n apisix -l app.kubernetes.io/name=ingress-controller.
After helm upgrade a pod is stuck Pending. On a single node with hostNetwork: true this is almost always a port conflict with an old pod of the same role. Check: kubectl get events -n <ns> | grep FailedScheduling. Fixed by updateStrategy.type=Recreate, not by raising the timeout.
A metric exists in PromQL but has no data. Verify the plugin is actually attached (ApisixGlobalRule) and the exporter port is listening on 0.0.0.0, not 127.0.0.1 — kubectl exec into the pod and curling locally won't reveal this, you need to hit it through the Service.
Summary
- On a single bare-metal node with
hostNetwork, blue-green between two ingress controllers is impossible — the scheduler blocks conflicting ports. It's a direct cutover with deliberate downtime, not a rolling update. helm show valuesbefore install isn't a formality. Three real bugs (ssl.enabled, the wrong port key, a broken--setwith commas) were found only through it, not through documentation.- Never pass values containing commas through
--set— JSON strings, lists, anything with a,in it. Use-f values.yamlonly. - Migrating routes isn't a mechanical one-to-one CRD swap. Middleware that looked decorative (
addPrefixon only some paths,redirectRegexinstead of proxying) changes behavior if it isn't reproduced exactly. hostNetwork+ one node meansupdateStrategy: Recreateforever, not just for the initial install.- Ingress controller metrics and logs are separate work after the cutover itself, not an automatic consequence of installing the chart.