← weblog / 2026-02-22-apisix-kubernetes-ingress
2026-02-22·kubernetesapisixnetworkingobservability·5 min read

APISIX as a Kubernetes ingress: global rules, TLS, and OpenTelemetry

Published: 2026-02-22

APISIX replaced Traefik across all six clusters. It handles ingress routing, TLS termination, access logging to Elasticsearch, distributed tracing via OpenTelemetry, and gzip compression — all through declarative CRDs. This covers the production setup: dual ingress classes, global plugins, TLS management, otel-collector integration, and per-environment patching.

Why APISIX over Traefik

APISIX offers features that Traefik requires enterprise for:

  • Plugin ecosystem: 80+ built-in plugins, including OpenTelemetry, Kafka logger, IP restriction, OIDC
  • ApisixGlobalRule: apply plugins cluster-wide without touching individual routes
  • gRPC proxying out of the box
  • Admin API for dynamic config without restarts

The trade-off: steeper learning curve, heavier resource footprint (requires etcd or uses APISIX's own etcd).

Two ingress classes

apisix-int runs on every environment. It's the internal ingress with ingressClass: apisix. Internal services (Grafana, Kibana, app dashboards) are exposed through it on private IPs.

apisix-ext exists only on the demo cluster. It has a dedicated LoadBalancer IP and serves external domains (*.test.antonnovikov.com, *.demo.test.antonnovikov.com). External traffic from the internet goes here.

Both are deployed via HelmRelease through FluxCD. The base HelmRelease lives in fluxcd/base/apisix-int.yaml; per-environment patches override only the LoadBalancer IP and the otel-collector address.

HelmRelease base

yamlapiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: apisix-int
  namespace: ingress-apisix
spec:
  chart:
    spec:
      chart: apisix
      version: "2.8.*"
      sourceRef:
        kind: HelmRepository
        name: apisix
        namespace: flux-system
  retries: -1
  values:
    apisix:
      enabled: true
      kind: DaemonSet
      replicaCount: 1
    gateway:
      type: LoadBalancer
    ingress-controller:
      enabled: true
      apisix:
        serviceNamespace: ingress-apisix
    etcd:
      enabled: true
      replicaCount: 1

kind: DaemonSet ensures one APISIX pod per node, distributing load without needing HPA.

ApisixGlobalRule: cluster-wide plugins

Instead of attaching plugins to every route individually, ApisixGlobalRule applies them to all traffic.

Elasticsearch access logging

yamlapiVersion: apisix.apache.org/v2
kind: ApisixGlobalRule
metadata:
  name: elasticsearch-logger
  namespace: ingress-apisix
spec:
  plugins:
    - name: elasticsearch-logger
      enable: true
      config:
        endpoint_addr: "http://infra-master.elasticsearch.svc.cluster.local:9200"
        field:
          index: "apisix"
        pipeline: "apisix"
        batch_max_size: 1000
        buffer_duration: 60

All requests from all routes land in the apisix Elasticsearch index. pipeline: "apisix" runs an ES ingest pipeline that parses the raw log JSON into structured fields. Without the ingest pipeline, logs land as unstructured JSON blobs that can't be filtered in Kibana.

OpenTelemetry tracing

yamlapiVersion: apisix.apache.org/v2
kind: ApisixGlobalRule
metadata:
  name: opentelemetry
  namespace: ingress-apisix
spec:
  plugins:
    - name: opentelemetry
      enable: true
      config:
        sampler:
          name: always_on
        collector:
          address: "otel-collector.elasticsearch.svc.cluster.local:4318"
        additional_attributes:
          - apisix_service_id
          - apisix_balancer_ip
          - apisix_consumer
          - remote_addr
          - upstream_addr
          - upstream_status
          - upstream_response_time

Sampling is always_on. APISIX sends spans via OTLP/HTTP to otel-collector at port 4318. The collector bridges to APM Server (which only accepts gRPC).

additional_attributes adds APISIX-specific context to each span: which upstream received the request (upstream_addr), the HTTP status code (upstream_status), and the response time from the upstream. These show up as span attributes in Kibana APM and are invaluable for debugging slow routes.

Use the full FQDN for the otel-collector address. Short DNS names (otel-collector.elasticsearch.svc) sometimes fail to resolve within APISIX's internal Lua runtime.

Gzip, real-ip, and request-id

Three remaining global rules:

yaml# Real-IP: read X-Forwarded-For from load balancer
- name: real-ip
  enable: true
  config:
    source: http_x_forwarded_for
    recursive: true

# Gzip: compress responses > 1kb
- name: gzip
  enable: true
  config:
    min_length: 1024
    http_version: 1.1
    buffers:
      number: 32
      size: 4096
    types:
      - text/html
      - application/json
      - text/plain
      - application/javascript
      - text/css

# Request-ID: inject X-Request-ID header
- name: request-id
  enable: true
  config:
    header_name: X-Request-ID
    include_in_response: true

real-ip with recursive: true handles double-NAT situations where there are multiple load balancers in front of APISIX.

TLS with ApisixTls

TLS certificates are managed per-environment through ApisixTls CRDs:

yamlapiVersion: apisix.apache.org/v2
kind: ApisixTls
metadata:
  name: dev-antonnovikov-com
  namespace: ingress-apisix
spec:
  hosts:
    - "*.dev.test.antonnovikov.com"
  secret:
    name: dev-antonnovikov-com-tls
    namespace: ingress-apisix

The TLS secret contains tls.crt and tls.key, sealed with SealedSecrets. ApisixTls tells APISIX which cert to serve for which domain pattern. For wildcard certs, one ApisixTls object covers all subdomains.

For cert rotation: update the SealedSecret with the new cert, commit, push. APISIX picks up the change immediately without restart when the underlying Secret changes.

ApisixRoute example

yamlapiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: grafana
  namespace: ingress-apisix
spec:
  http:
    - name: grafana-rule
      match:
        hosts:
          - grafana.dev.test.antonnovikov.com
        paths:
          - "/*"
      backends:
        - serviceName: kube-prom-stack-grafana
          servicePort: 80
          namespace: observability

Routes can reference services in other namespaces. This avoids duplicating the route in each namespace.

Cilium L2 for LoadBalancer IPs (on-prem)

On dev and test (bare-metal k3s), LoadBalancer services get IPs from a Cilium L2 pool. The APISIX service gets a specific IP:

yamlvalues:
  gateway:
    type: LoadBalancer
    annotations:
      io.cilium/lb-ipam-ips: "172.16.57.200"

On Yandex Cloud managed clusters, the cloud load balancer assigns the IP automatically.

Patching APISIX per environment

The hub patch overrides only what changes between environments:

yaml# projects/dev/kustomization/hub/patches/apisix-int.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: apisix-int
  namespace: dev
spec:
  values:
    apisix:
      externalIPs:
        - 172.16.57.200
    plugins:
      - name: opentelemetry
        conf:
          collector:
            address: "otel-collector.elasticsearch.svc.cluster.local:4318"

Suspend/resume for manual intervention

bash# Freeze APISIX on dev (e.g., for manual config testing)
flux suspend helmrelease apisix-int -n dev --context=infra-k8s

# Restore
flux resume helmrelease apisix-int -n dev --context=infra-k8s

APISIX on Yandex Cloud clusters occasionally gets stuck in an upgrade→rollback loop due to DaemonSet update ordering. Suspend + fix + resume is the reliable recovery path.

Checking APISIX plugin status

bash# List loaded plugins
kubectl exec -n ingress-apisix deploy/apisix -- \
  curl -s http://localhost:9080/apisix/admin/plugins/list

# Check global rules
kubectl exec -n ingress-apisix deploy/apisix -- \
  curl -s http://localhost:9080/apisix/admin/global_rules

APISIX health check

bashkubectl port-forward -n ingress-apisix svc/apisix-admin 9080:9080
curl -s http://localhost:9080/apisix/admin/upstreams | jq '.total'