OpenTelemetry bridge: APISIX → otel-collector → Elasticsearch APM

Published: 2026-04-13

APISIX sends traces via OTLP/HTTP. The Elasticsearch APM Server accepts traces via OTLP/gRPC. They speak incompatible transports. An OpenTelemetry Collector in the middle bridges them.

The problem

APISIX's opentelemetry plugin sends spans to an OTLP/HTTP endpoint (port 4318). Elastic APM Server only accepts OTLP/gRPC (port 8200 or 4317). Without an adapter, traces from APISIX never reach Kibana APM.

Architecture

APISIX plugin
  │ OTLP/HTTP :4318
  ▼
otel-collector (Kubernetes Deployment)
  │ OTLP/gRPC :8200
  ▼
APM Server → Elasticsearch → Kibana APM UI

otel-collector HelmRelease

yamlapiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: otel-collector
  namespace: elasticsearch
spec:
  chart:
    spec:
      chart: opentelemetry-collector
      version: "0.97.0"
      sourceRef:
        kind: HelmRepository
        name: open-telemetry
        namespace: flux-system
  values:
    fullnameOverride: otel-collector    # critical — see below
    mode: deployment
    replicaCount: 1
    config:
      receivers:
        otlp:
          protocols:
            http:
              endpoint: 0.0.0.0:4318
            grpc:
              endpoint: 0.0.0.0:4317
      exporters:
        otlp/elastic:
          endpoint: "apm-server.elasticsearch.svc.cluster.local:8200"
          tls:
            insecure: true
      service:
        pipelines:
          traces:
            receivers: [otlp]
            exporters: [otlp/elastic]
          metrics:
            receivers: [otlp]
            exporters: [otlp/elastic]
          logs:
            receivers: [otlp]
            exporters: [otlp/elastic]

Why `fullnameOverride` is critical

APISIX's opentelemetry plugin resolves the collector address via DNS. If the Helm chart generates a name like otel-collector-opentelemetry-collector (the default), the configured address otel-collector.elasticsearch.svc.cluster.local:4318 fails to resolve.

fullnameOverride: otel-collector forces the Service name to exactly otel-collector. Without this, APISIX drops spans silently with no error in logs.

APISIX plugin configuration

The ApisixGlobalRule enables the plugin on all routes:

yamlapiVersion: apisix.apache.org/v2
kind: ApisixGlobalRule
metadata:
  name: opentelemetry
  namespace: ingress-apisix
spec:
  plugins:
    - name: opentelemetry
      enable: true
      config:
        sampler:
          name: always_on
        collector:
          address: "otel-collector.elasticsearch.svc.cluster.local:4318"
          request_timeout: 3
        additional_attributes:
          - apisix_service_id
          - apisix_balancer_ip
          - remote_addr
          - upstream_addr
          - upstream_status
          - upstream_response_time

Use a full FQDN for the collector address. Short DNS names (otel-collector.elasticsearch.svc) occasionally fail inside APISIX's Lua DNS resolver.

additional_attributes adds APISIX metadata to each span — routing decisions, balancer IP, upstream response time. These appear as custom attributes in Kibana APM and are useful for debugging load balancing issues.

APM Server

APM Server runs in the elasticsearch namespace:

yamlvalues:
  apmConfig:
    apm-server.yml: |
      apm-server:
        host: "0.0.0.0:8200"
      output.elasticsearch:
        hosts: ["http://dev-master.elasticsearch:9200"]
        username: "elastic"
        password: "${ELASTIC_PASSWORD}"
      apm-server.auth.anonymous:
        enabled: true
        allow_service: []

It accepts both OTLP/gRPC and the native APM protocol on port 8200.

Viewing traces in Kibana

After the pipeline is running:

Service map — which services call which endpoints, with latency heatmap
Transaction traces — full spans from APISIX to upstream, with timing breakdown
Error rate and latency percentiles per service and endpoint

The apisix_service_id attribute links each trace to the specific APISIX route, making it easy to correlate high-latency alerts with specific API endpoints.

Validating the pipeline

bash# Check otel-collector is receiving spans
kubectl logs -n elasticsearch deployment/otel-collector \
  --context=dev-k8s | grep -i "traces\|spans"

# Check APM server is receiving from collector
kubectl logs -n elasticsearch deployment/apm-server \
  --context=dev-k8s | grep "Publish"

# Generate a test trace (any request through APISIX)
curl -H "Host: myapp.dev.example.com" http://172.16.57.193/ -v

Check otel-collector metrics endpoint:

bashkubectl port-forward -n elasticsearch svc/otel-collector 8888:8888
curl http://localhost:8888/metrics | grep otelcol_receiver_accepted

Common issues

Spans not appearing in Kibana — check in order: APISIX → otel-collector → APM Server. Use kubectl logs at each step.

DNS resolution failure in APISIX — use full FQDN service.namespace.svc.cluster.local.

fullnameOverride forgotten — Helm generates a long name; APISIX can't resolve it; spans drop silently.

APM Server TLS errors — set tls.insecure: true in the exporter if APM Server doesn't have TLS configured.

Summary

OpenTelemetry Collector bridges APISIX (OTLP/HTTP on port 4318) to Elasticsearch APM Server (OTLP/gRPC on port 8200) — they don't speak the same transport natively
APISIX's opentelemetry plugin must point to http://otel-collector.namespace.svc.cluster.local:4318/v1/traces
Always use fullnameOverride in the otel-collector Helm chart — the auto-generated name is unpredictably long and breaks DNS references in APISIX config
If spans don't appear in Kibana, trace the pipeline: APISIX logs → otel-collector logs → APM Server logs
Set tls.insecure: true in the otel-collector exporter if APM Server doesn't have TLS configured

weblog