ELK in Kubernetes: Elasticsearch, APM Server, and Kibana

Published: 2026-03-17

We use the ELK stack for two things: log search and application performance monitoring (APM). Logs come from Vector. APM traces come from the Elastic APM agent in .NET applications. Kibana ties them together.

Deploy via FluxCD

Three HelmReleases, all from the Elastic Helm repository:

yaml# elasticsearch
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: elasticsearch
  namespace: elk
spec:
  chart:
    spec:
      chart: elasticsearch
      version: "8.x"
      sourceRef:
        kind: HelmRepository
        name: elastic
  values:
    replicas: 1
    minimumMasterNodes: 1
    esJavaOpts: "-Xmx2g -Xms2g"
    resources:
      requests:
        cpu: 500m
        memory: 3Gi
      limits:
        cpu: 2000m
        memory: 4Gi
    volumeClaimTemplate:
      resources:
        requests:
          storage: 200Gi
      storageClassName: local-path
    persistence:
      enabled: true

esJavaOpts sets the JVM heap. The rule: heap = half the container memory limit. With memory: 4Gi limit, -Xmx2g leaves 2 GB for OS page cache (used by Lucene).

yaml# kibana
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kibana
  namespace: elk
spec:
  chart:
    spec:
      chart: kibana
      version: "8.x"
      sourceRef:
        kind: HelmRepository
        name: elastic
  values:
    elasticsearchHosts: "http://elasticsearch-master.elk.svc.cluster.local:9200"
    resources:
      requests:
        cpu: 200m
        memory: 512Mi
      limits:
        cpu: 1000m
        memory: 1Gi

yaml# apm-server
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: apm-server
  namespace: elk
spec:
  chart:
    spec:
      chart: apm-server
      version: "8.x"
      sourceRef:
        kind: HelmRepository
        name: elastic
  values:
    apmConfig:
      apm-server.yml: |
        apm-server:
          host: "0.0.0.0:8200"
        output.elasticsearch:
          hosts: ["http://elasticsearch-master.elk.svc.cluster.local:9200"]
    resources:
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 512Mi

APM agent in .NET

xml<PackageReference Include="Elastic.Apm.NetCoreAll" Version="1.*" />

One line in Program.cs:

csharpbuilder.Services.AddAllElasticApm();

Configure via environment variables:

yamlenv:
  - name: ELASTIC_APM_SERVER_URL
    value: "http://apm-server.elk.svc.cluster.local:8200"
  - name: ELASTIC_APM_SERVICE_NAME
    value: "my-service"
  - name: ELASTIC_APM_ENVIRONMENT
    value: "prod"
  - name: ELASTIC_APM_LOG_LEVEL
    value: "Warning"
  - name: ELASTIC_APM_TRANSACTION_SAMPLE_RATE
    value: "0.1"   # sample 10% — reduce APM storage

The agent auto-instruments ASP.NET Core requests, Entity Framework queries, and HttpClient calls. No manual span creation required for most use cases.

Custom APM spans

csharpvar transaction = Agent.Tracer.CurrentTransaction;
using var span = transaction.StartSpan("ProcessPayment", "payment");
span.SetLabel("order_id", orderId.ToString());
try {
    await _paymentService.ChargeAsync(orderId);
    span.Outcome = Outcome.Success;
} catch (Exception ex) {
    span.CaptureException(ex);
    throw;
}

Index lifecycle management

Without ILM, Elasticsearch grows unbounded:

jsonPUT _ilm/policy/k8s-logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "20gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "forcemerge": { "max_num_segments": 1 },
          "shrink": { "number_of_shards": 1 }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

Apply via Kibana Dev Tools or a CI job. Data older than 30 days is deleted automatically. Warm phase force-merges segments to reduce disk usage.

Useful Kibana queries

# All errors in the last hour
kubernetes.namespace: "app" AND log.level: ERROR

# Slow HTTP requests (APM)
transaction.duration.us > 5000000

# Specific pod
kubernetes.pod.name: "my-service-abc123"

# Exception trace
error.exception.message: *NullReference*

KQL (Kibana Query Language) is more readable than Lucene and supports autocomplete in Discover.

APM → Kibana service map

With APM instrumented across multiple services, Kibana's Service Map shows:

Which services call which
Average latency per edge
Error rate per service

This is the distributed tracing map that APM builds automatically from spans. No manual topology config.

Troubleshooting

APM data not showing in Kibana: Check APM Server logs and verify ELASTIC_APM_SERVER_URL is reachable from the app pod.

High memory / OOM: Reduce esJavaOpts heap if other workloads are competing for memory. Don't set heap > 50% of container limit.

Index not created: Vector writes vector-YYYY-MM-DD indices. If no logs appear, check Vector sink errors (see the Vector article).

weblog