Elastic APM Server in Kubernetes

Published: 2026-02-14

APM Server is the ingestion endpoint for Elastic APM traces, metrics, and errors. It receives data from application agents (Go, Java, Node.js, Python) and forwards to Elasticsearch. In the OTLP world it also accepts spans over gRPC — which is how the OpenTelemetry Collector bridge connects.

This post covers the HelmRelease configuration, Elasticsearch credential management via ESO, OTLP ingestion setup, index template configuration, and the debugging workflow when spans stop arriving.

Architecture

The flow for traces in our stack:

copyApp (OTEL SDK) → otel-collector → APM Server → Elasticsearch → Kibana APM

APM Server serves two purposes: it accepts both Elastic APM agents (native format on port 8200) and OTLP spans (gRPC on port 8200). The otel-collector in the cluster aggregates traces from multiple apps and forwards them to APM Server, which indexes them into Elasticsearch apm-* indices.

This architecture separates concerns: apps don't need to know about Elasticsearch directly, and switching from Elastic APM to a different backend only requires changing the otel-collector pipeline, not the application code.

HelmRelease

The apm-server chart is part of the elastic Helm repository. It does not use an operator — just a Deployment and a Service:

yamlcopyapiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: apm-server
  namespace: observability
spec:
  chart:
    spec:
      chart: apm-server
      version: "7.17.*"
      sourceRef:
        kind: HelmRepository
        name: elastic
        namespace: flux-system
  retries: -1
  values:
    replicaCount: 1
    config:
      apm-server:
        host: "0.0.0.0:8200"
        rum:
          enabled: true
          allow_origins: ["*"]
        kibana:
          enabled: true
          host: "http://kibana.observability.svc.cluster.local:5601"
        output.elasticsearch:
          hosts:
            - "http://elasticsearch-master.observability.svc.cluster.local:9200"
          username: "${APM_ES_USER}"
          password: "${APM_ES_PASS}"
    secretMounts:
      - name: apm-credentials
        secretName: apm-es-credentials
        path: /usr/share/apm-server/config/credentials
    resources:
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 512Mi

retries: -1 — like all Elasticsearch-adjacent services, APM Server needs the ES cluster to be ready first, so unlimited retries for the HelmRelease. Without this, if APM Server deploys before Elasticsearch is ready, the HelmRelease fails and doesn't automatically retry.

rum.enabled: true — enables Real User Monitoring, which allows browser JavaScript to send traces directly to APM Server. allow_origins: ["*"] is convenient for development but should be restricted to your actual domain in production.

${APM_ES_USER} and ${APM_ES_PASS} — environment variables injected from the apm-es-credentials Secret via secretMounts. The credentials file is mounted at the path specified, and APM Server reads environment variables from it using Filebeat's keystore mechanism.

Resource sizing

The defaults (100m CPU, 256Mi RAM) are fine for low-traffic clusters (< 1000 requests/second). For higher traffic:

• 500m CPU is sufficient for most production workloads
• RAM scales with the number of concurrent connections and the batch size; 512Mi is safe for most setups
• For large trace volumes, consider a dedicated APM Server deployment per cluster

Elasticsearch credentials via ESO

The APM Server needs read/write access to create APM indices in Elasticsearch. These credentials live in Vault:

yamlcopyapiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: apm-es-credentials
  namespace: observability
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: apm-es-credentials
  data:
    - secretKey: APM_ES_USER
      remoteRef:
        key: secret/dev/elasticsearch
        property: apm_user
    - secretKey: APM_ES_PASS
      remoteRef:
        key: secret/dev/elasticsearch
        property: apm_password

Store the credentials in Vault:

bashcopyvault kv put secret/dev/elasticsearch \
  apm_user=apm_writer \
  apm_password=your-secure-password \
  kibana_user=kibana_system \
  kibana_password=another-secure-password

Elasticsearch user for APM

Create a dedicated Elasticsearch user with only the required permissions:

bashcopy# In Kibana Dev Tools or via Elasticsearch API
PUT /_security/role/apm_writer
{
  "indices": [
    {
      "names": ["apm-*"],
      "privileges": ["write", "create_index", "manage"]
    }
  ]
}

PUT /_security/user/apm_writer
{
  "password": "your-secure-password",
  "roles": ["apm_writer"]
}

Avoid using the elastic superuser for APM Server — if APM Server is compromised, the blast radius should be limited to APM indices only.

OTLP ingestion

APM Server 7.17 supports OTLP over gRPC on port 8200:

yamlcopyapm-server:
  otlp:
    grpc:
      enabled: true
      host: "0.0.0.0:8200"

The otel-collector in the cluster sends spans here via:

yamlcopyexporters:
  otlp/apm:
    endpoint: apm-server.observability.svc.cluster.local:8200
    tls:
      insecure: true

otel-collector pipeline configuration

A minimal otel-collector config that receives from apps and forwards to APM Server:

yamlcopyreceivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000
  memory_limiter:
    limit_mib: 256

exporters:
  otlp/apm:
    endpoint: apm-server.observability.svc.cluster.local:8200
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/apm]

The batch processor is important: without it, each span is sent as a separate gRPC call, which is extremely inefficient at high trace volumes.

Ingest pipeline for APM indices

APM creates its own index templates (apm-*). For single-node clusters with no replicas:

bashcopycurl -X PUT "http://elasticsearch:9200/_template/apm-server" \
  -H "Content-Type: application/json" \
  -d '{
    "index_patterns": ["apm-*"],
    "settings": {
      "number_of_replicas": 0
    }
  }'

Without this, Elasticsearch will try to create apm-* indices with 1 replica, which will remain in YELLOW state on a single-node cluster (no replica nodes to place the replica shards).

ILM policy for APM indices

APM indices grow fast. Set up ILM to roll over and delete old data:

bashcopyPUT /_ilm/policy/apm-rollover
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "5gb",
            "max_age": "7d"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Apply to the APM index template:

bashcopyPUT /_template/apm-server
{
  "index_patterns": ["apm-*"],
  "settings": {
    "number_of_replicas": 0,
    "index.lifecycle.name": "apm-rollover"
  }
}

Checking that spans arrive

bashcopy# port-forward to APM server
kubectl port-forward -n observability svc/apm-server 8200:8200

# health check — should return {"ok":true}
curl http://localhost:8200/

# look for span indices in ES
curl http://localhost:9200/_cat/indices/apm-*?v

Traces appear in Kibana → APM section. Each service shows a service map, latency histogram, and error rate — all populated automatically from the OTLP spans sent by APISIX through otel-collector.

APM Server not receiving spans

Common causes:

1. Connection refused — APM Server pod is not running, check kubectl get pods -n observability
2. Auth error — wrong ES credentials, check kubectl logs -n observability deploy/apm-server | grep "authentication"
3. OTLP port mismatch — otel-collector sending to wrong endpoint, check the collector's exporter config
4. Index template missing — ES rejecting writes because replica count > available nodes

bashcopy# Check APM Server logs
kubectl logs -n observability deploy/apm-server --tail=50

# Check ES connectivity from APM pod
kubectl exec -n observability deploy/apm-server -- \
  curl -s http://elasticsearch-master.observability.svc.cluster.local:9200/_cluster/health