From APISIX to kgateway on Single-Node k0s: Gateway API Instead of Annotations
Published: 2026-11-13
Four months ago I moved my k0s cluster from Traefik to APISIX and wrote down everything that broke. Today APISIX is gone too: both clusters (the main node in Prague and a small node in Russia) now run kgateway, the CNCF Gateway API implementation built on Envoy. Two things drove the move. APISIX is three components (gateway, etcd, ingress controller) plus an Admin API that has to be guarded, and that Admin API already caused an incident here. And all routing lived in annotations only one controller understands. Below: how I checked that kgateway fits a single-node bare-metal cluster, what had to be rewritten by hand, and two mistakes I caught in production.
Why bother
The stack before the move:
apache/apisix:3.17withhostNetwork: trueon ports 80/443;- etcd as an intermediate route store;
apisix-ingress-controller2.1 with anadc-serversidecar that turnsIngressobjects andk8s.apisix.apache.org/*annotations into Admin API calls.
Together that was about 386 MiB of memory on the main node and 241 MiB on the RU node. The Admin API listens on 9180 on the host, and in September someone found it from outside together with the demo key. I closed it with a firewall rule and a proper Secret, but the class of problem stays as long as an Admin API exists.
kgateway is simpler. The controller reads Gateway API objects from Kubernetes and pushes Envoy config over xDS. No datastore of its own, no Admin API facing the world, and routes are plain HTTPRoute objects any other Gateway API controller understands.
First, check that it fits
Before installing anything I compared versions. kgateway 2.5.1 supports Kubernetes 1.33-1.37 and needs Gateway API CRDs 1.5-1.7. My clusters run k0s 1.34.2 and 1.36.4, so that part was fine. The CRDs were more interesting:
bashkubectl --context k0s get crd gateways.gateway.networking.k8s.io \
-o jsonpath='{.metadata.annotations.gateway\.networking\.k8s\.io/bundle-version}'
# v1.4.0
kubectl --context k0s get crd tcproutes.gateway.networking.k8s.io \
-o jsonpath='{.status.storedVersions}'
# ["v1alpha2"]
The main cluster had the v1.4.0 standard bundle plus experimental TCPRoute/TLSRoute/UDPRoute v1.3.0, which the APISIX chart brought in at some point. If storedVersions lists a version the new CRD no longer defines, the API server rejects the update. There were zero Gateway API objects in the cluster, but I didn't guess and ran a server-side dry run instead:
bashkubectl apply --server-side --force-conflicts --dry-run=server \
-f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.6.0/standard-install.yaml
It came back clean, so the main node got v1.6.0, the same as the RU node, rather than the v1.7.0 the quickstart suggests. One version on both clusters is easier to maintain.
The second key fact: kgateway does not read networking.k8s.io/v1 Ingress at all. The chart's RBAC has no rule for ingresses and the code has no such controller. ingress2gateway understands exactly one APISIX annotation, http-to-https. So all 20 annotated routes had to be rewritten by hand.
Installing
Two OCI charts, no external repos:
bashhelm upgrade -i kgateway-crds oci://cr.kgateway.dev/kgateway-dev/charts/kgateway-crds \
--version v2.5.1 -n kgateway-system --create-namespace
helm upgrade -i kgateway oci://cr.kgateway.dev/kgateway-dev/charts/kgateway \
--version v2.5.1 -n kgateway-system \
--set controller.resources.requests.memory=64Mi \
--set controller.resources.limits.memory=384Mi --wait
The controller ships with no requests or limits, so I set a memory limit right away. The chart creates GatewayClass kgateway by itself. The Envoy pod is created by the controller, one per Gateway.
Putting Envoy on 80/443 without a LoadBalancer
One node, no load balancer: APISIX held the ports with hostNetwork. kgateway's GatewayParameters has no hostNetwork or hostPort field. It does have deploymentOverlay, a strategic merge patch applied on top of the generated Deployment:
yamlapiVersion: gateway.kgateway.dev/v1alpha1
kind: GatewayParameters
metadata:
name: edge
namespace: kgateway-system
spec:
kube:
service:
type: ClusterIP
envoyContainer:
resources:
requests: {cpu: 50m, memory: 64Mi}
limits: {memory: 384Mi}
deploymentOverlay:
spec:
strategy: # one node + hostPort: free the ports first
type: RollingUpdate
rollingUpdate: {maxSurge: 0, maxUnavailable: 1}
template:
spec:
containers:
- name: kgateway-proxy
ports:
- {name: http, containerPort: 80, hostPort: 80, protocol: TCP}
- {name: https, containerPort: 443, hostPort: 443, protocol: TCP}
I chose hostPort over hostNetwork for two reasons. First, when a listener port is below 1024 the controller adds the pod sysctl net.ipv4.ip_unprivileged_port_start=0. With hostNetwork the kubelet rejects that sysctl and the pod never starts. Second, with hostPort the portmap CNI plugin only does DNAT, so Envoy sees the real client IP. I checked that portmap is in the flannel chain on both nodes:
bashssh root@node 'cat /etc/cni/net.d/*.conflist' | grep -A2 portmap
The Gateway points at these parameters through infrastructure.parametersRef.
Two mistakes I caught in production
I started with the RU node: one route, no auth, no rewrites. I applied the Gateway without deploymentOverlay first, tested it through port-forward, and only then stopped APISIX and added the overlay. That's where things went wrong.
Mistake 1: Recreate in the overlay doesn't work
The first overlay set strategy.type: Recreate with rollingUpdate: null to drop the default rollingUpdate. APISIX was already stopped and the new pod never showed up. The controller logs said:
textfailed to apply object apps/v1, Kind=Deployment kgateway-system/edge:
Deployment.apps "edge" is invalid: spec.strategy.rollingUpdate: Forbidden:
may not be specified when strategy `type` is 'Recreate'
The CRD schema prunes the null before the strategic merge ever sees it, and the generated Deployment always has rollingUpdate. The fix is to keep RollingUpdate with maxSurge: 0, maxUnavailable: 1. On a single node that behaves like Recreate: the old pod frees the ports, the new one takes them. The RU site was down for about three minutes. Lesson: run the overlay through --dry-run=server before you stop the old gateway, not after.
Mistake 2: 404 from HTTP/2 connection coalescing
Right after the RU cutover I got a message: https://vpn.ru.antonnovikov.com/ 404. My curl said 200. The listeners were the cause. I had one HTTPS listener per hostname:
yamllisteners:
- name: https-ru
hostname: ru.antonnovikov.com
...
- name: https-vpn
hostname: vpn.ru.antonnovikov.com
...
Both names are on one certificate. A browser that already has an HTTP/2 connection to ru. is allowed to reuse it for vpn.ru.: same IP, the certificate covers the name. But in Envoy the connection is bound to a filter chain by SNI, i.e. to listener https-ru, which has no virtual hosts for vpn.ru.. Hence the 404. curl opens a fresh connection each time, so it never sees this. You can reproduce it by sending a different Host than the SNI:
bashcurl -s -o /dev/null -w '%{http_code}\n' --http2 \
--resolve ru.antonnovikov.com:443:<RU_IP> \
-H 'Host: vpn.ru.antonnovikov.com' https://ru.antonnovikov.com/
The fix is one HTTPS listener without a hostname, carrying every certificate. kgateway puts all certificateRefs into a single TLS context and Envoy picks the certificate by SNI. The first one in the list is served for unknown SNI. On the main cluster this matters even more: one SAN certificate covers the apex, www, trips and 65 more subdomains (68 names), and koalka.ru and www.koalka.ru share a certificate too.
yaml- name: https
port: 443
protocol: HTTPS
allowedRoutes: {namespaces: {from: All}}
tls:
certificateRefs:
- {name: antonnovikov-com-san-tls, namespace: default} # default for unknown SNI
- {name: koalka-tls, namespace: default}
- {name: grafana-tls, namespace: monitoring}
# ... 15 Secrets across 5 namespaces in total
Secrets from other namespaces need a ReferenceGrant in each of them.
How the annotations translated
| APISIX | Gateway API / kgateway |
|---|---|
http-to-https: "true" |
one catch-all HTTPRoute on listener http with RequestRedirect scheme: https |
http-redirect + code 301 |
RequestRedirect to an external hostname, route attached to both listeners |
rewrite-target-regex |
TrafficPolicy.urlRewrite.pathRegex |
upstream-read-timeout |
rules[].timeouts.request or TrafficPolicy.timeouts |
response-rewrite-add-header |
ResponseHeaderModifier filter |
ApisixPluginConfig with proxy-rewrite |
RequestHeaderModifier filter |
auth-type: basicAuth + ApisixConsumer |
TrafficPolicy.basicAuth with htpasswd in a Secret |
ApisixGlobalRule prometheus |
Envoy metrics on :9091/metrics |
Most of it maps one to one. Five places were not obvious.
Redirect before auth. In APISIX I raised the redirect plugin priority to 3000 so that http:// went to https:// before basic-auth ran. Otherwise the browser sent the password in clear text. In Gateway API this is solved by structure: basicAuth is attached only to routes on the HTTPS listener, and the http listener only has the catch-all redirect. No priorities needed.
Basic-auth accepts only {SHA}. kgateway 2.5 understands only SHA-1 entries in htpasswd and skips bcrypt and apr1 with an error. Passwords didn't have to change: I built the new Secret from the old ones without printing values to the terminal:
bashfor s in monitoring-ui-basic-auth monitoring-federate-basic-auth; do
u=$(kubectl -n monitoring get secret $s -o jsonpath='{.data.username}' | base64 -d)
p=$(kubectl -n monitoring get secret $s -o jsonpath='{.data.password}' | base64 -d)
printf '%s:{SHA}%s\n' "$u" "$(printf '%s' "$p" | openssl dgst -sha1 -binary | base64)"
done > .htpasswd
kubectl -n monitoring create secret generic monitoring-htpasswd --from-file=.htpasswd
X-Real-IP. APISIX set it from $remote_addr, and the app builds its view counter and rate limits on it. Envoy doesn't set this header. Without it the counter would quietly drop to zero and rate limits would key on a spoofable X-Forwarded-For. I set it with a Gateway-wide policy:
yamlapiVersion: gateway.kgateway.dev/v1alpha1
kind: TrafficPolicy
metadata: {name: edge-defaults, namespace: kgateway-system}
spec:
targetRefs: [{group: gateway.networking.k8s.io, kind: Gateway, name: edge}]
headerModifiers:
request:
set: [{name: X-Real-IP, value: "%DOWNSTREAM_REMOTE_ADDRESS_WITHOUT_PORT%"}]
httpUpgrade: [{type: websocket}]
timeouts:
request: 60s
Envoy expands the variable, and set overwrites whatever the client sent. Check: curl -H 'X-Real-IP: 6.6.6.6' .../myip returned my real address.
Timeouts and websockets. An Envoy route times out after 15 seconds by default; APISIX used 60. So the Gateway gets 60 seconds, photo uploads get 180, and the Docker registry has the timeout disabled (0s), otherwise large layers never finish. Websockets are off by default in kgateway; without httpUpgrade Grafana Live would not work.
A hostname with an underscore. The tg. redirect pointed at https://antonnovikov_opslaika.t.me/. Gateway API validates the hostname in RequestRedirect and rejects _. I rewrote it to the equivalent https://t.me/<channel> with ReplaceFullPath.
cert-manager
Both ClusterIssuers solved HTTP-01 through ingressClassName: apisix. Without APISIX, renewals would have quietly failed a month later. Two things are needed: the --enable-gateway-api flag on cert-manager, and a solver that attaches a temporary HTTPRoute to the http listener:
yamlsolvers:
- http01:
gatewayHTTPRoute:
parentRefs:
- {name: edge, namespace: kgateway-system, kind: Gateway, sectionName: http}
The solver route is more specific than the catch-all redirect (hostname plus exact path /.well-known/acme-challenge/<token>), so it wins. I verified it with a staging certificate on both clusters before removing APISIX.
A separate trap: certificates once created by cert-manager's ingress-shim from the cert-manager.io/cluster-issuer annotation on an Ingress. Their ownerReferences point at the Ingress, so deleting the Ingress makes the garbage collector delete the Certificate too. I removed the reference before deleting the owner:
bashkubectl -n default patch certificate koalka-tls --type=json \
-p '[{"op":"remove","path":"/metadata/ownerReferences"}]'
How I checked that nothing changed
The Gateway with all routes ran next to APISIX, just without hostPort. I wrote a script with 39 checks: hosts, paths, redirects with query strings, basic-auth with right and wrong passwords, the registry's /v2/. One run went against live APISIX, the other against kgateway through port-forward:
basht() { curl -s -o /dev/null -m 20 --resolve "$2:$PORT:$IP" \
-w '%{http_code} %{redirect_url}' "$1://$2:$PORT$3"; }
./matrix.sh 443 80 <NODE_IP> > old.txt # APISIX
./matrix.sh 28443 28080 127.0.0.1 > new.txt # kgateway via port-forward
diff old.txt new.txt
The only difference was tg, and that one was intended. After the cutover I ran the same script against production, and again after APISIX was deleted. Same result. I also checked that Envoy picks the right certificate by SNI for every domain, and that the config dump shows the 180-second timeout and the regex rewrite for trips.
The cutover on the main node itself was kubectl scale deploy apisix --replicas=0, applying the overlay, and waiting for the rollout. Downtime was under a minute. I kept APISIX stopped for a few hours so that a rollback was a single scale, and deleted it only after deploying the app version that reads HTTPRoute.
Monitoring and logs
Envoy metrics have no host label. There is envoy_http_conn_manager_prefix (the listener) and envoy_cluster_name in the form kube_<ns>_<svc>_<port>. So error alerts now work per backend, not per domain. Another quirk: kgateway's :9091/metrics exports only counters that have changed at least once. The "connect failures" panel stays empty until there's a failure, and that's fine.
A JSON access log is enabled with a ListenerPolicy. I kept the APISIX field names (remote_addr, host, status, request_uri) so promtail and the dashboards needed almost no changes. Timings are now in milliseconds, and the fields are called request_time_ms so units don't mix with older records.
What can go wrong
The Envoy pod is Pending or never updates. Look at the controller logs first, not the pod events: an overlay apply error only shows up there. kubectl -n kgateway-system logs deploy/kgateway | grep error. If the pod waits for ports, the old gateway or the previous Envoy pod still holds them - check maxSurge: 0.
One domain returns 404 although the route exists. If listeners are split by hostname and the certificate is shared, it's almost certainly HTTP/2 coalescing. Test with curl using one SNI and a different Host. A single HTTPS listener fixes it.
Basic-auth returns 401 for the right password. The Secret isn't {SHA} or uses the wrong key (the default is .htpasswd). The TrafficPolicy status will say no valid users with {SHA} hash format found.
Certificates stop renewing a month later. cert-manager runs without --enable-gateway-api, or a ClusterIssuer still points at the old ingress class. On my main cluster cert-manager is installed by a k0s Helm extension, and the flag is currently a live patch on the Deployment. It still has to move into k0s.yaml, or it disappears on the next k0s restart.
Summary
- kgateway runs on both single-node k0s clusters without a LoadBalancer:
hostPortthroughdeploymentOverlay, withportmapkeeping the client IP. - The gateway, etcd and ingress controller (386 MiB) became Envoy plus a controller: 29 and 59 MiB on the main node at my traffic level, 145 MiB versus 241 MiB on the RU node. No Admin API and no etcd anymore.
- kgateway doesn't read Ingress. I rewrote 20 annotated routes as
HTTPRouteandTrafficPolicyby hand and confirmed identical behaviour with a before/after diff of 39 checks. - Use one HTTPS listener for all certificates, or HTTP/2 coalescing will produce 404s in browsers that curl never shows.
- Check X-Real-IP, timeouts (Envoy defaults to 15 seconds), websockets and the htpasswd format. These are APISIX defaults that Envoy will not reproduce on its own.