Kafka alerting in Kubernetes: consumer lag, partitions, and disk

Published: 2026-02-18

Kafka in production needs four categories of alerting: broker availability, replication health, consumer lag, and resource saturation. Each category has different urgency and different metrics sources. Here's the PrometheusRule set we use in Kubernetes with kafka-exporter.

kafka-exporter deployment

kafka-exporter scrapes Kafka's metrics via the Kafka API (AdminClient) and exposes them in Prometheus format. It does not use JMX — which is why it's lighter than other Kafka exporters. We deploy it via the prometheus-kafka-exporter Helm chart:

yamlcopyapiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kafka-exporter
spec:
  chart:
    spec:
      chart: prometheus-kafka-exporter
      version: "3.0.1"
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
        namespace: flux-system
  values:
    kafkaServer:
      - kafka.kafka.svc.cluster.local:9092
    prometheus:
      serviceMonitor:
        enabled: true
        additionalLabels:
          release: kube-prom-stack
        interval: 30s
        relabelings:
          - targetLabel: job
            replacement: kafka
    resources:
      requests:
        cpu: 25m
        memory: 64Mi
      limits:
        cpu: 200m
        memory: 128Mi

The relabelings block overrides the default job label to kafka. All our alert expressions use job="kafka" to target these metrics. Without this, the job label defaults to the HelmRelease name or the ServiceMonitor name, which varies across environments.

What kafka-exporter exposes

Key metrics:

• kafka_brokers — number of brokers in the cluster
• kafka_topic_partitions — partition count per topic
• kafka_topic_partition_leader — which broker is leader for each partition
• kafka_topic_partition_under_replicated_partition — 1 if a partition is under-replicated
• kafka_consumergroup_lag — lag per partition per consumer group
• kafka_consumergroup_lag_sum — total lag per consumer group

Broker availability

yamlcopy- alert: KafkaDown
  expr: up{job="kafka"} < 1
  for: 5m
  labels:
    severity: disaster
    service: kafka
    groupp: admin
    url: 'https://grafana.example.com/d/kafka-overview/kafka-overview'
    event: 'Kafka Down |{{$labels.instance}}'
  annotations:
    description: >-
      Kafka broker {{$labels.instance}} has been down for more than 5 min.
      Timestamp: {{ with query "time()+10800" }}{{ . | first | value | humanizeTimestamp }}{{ end }}

yamlcopy- alert: KafkaExporterMissing
  expr: absent(kafka_brokers{job="kafka"})
  for: 5m
  labels:
    severity: high
    service: kafka
    event: 'Kafka Exporter Missing'
  annotations:
    description: No kafka_brokers series present — exporter may be down or can't reach Kafka.

The absent() alert catches the case where the exporter itself fails to scrape Kafka. Without it, KafkaDown would never fire because there'd be no up series at all — Prometheus would simply have no data rather than a 0 value.

Broker count monitoring

For multi-broker clusters:

yamlcopy- alert: KafkaBrokerCountChanged
  expr: kafka_brokers{job="kafka"} < 3
  for: 5m
  labels:
    severity: high
    event: 'Kafka Broker Count Low |{{$value}} brokers'
  annotations:
    description: Expected 3 Kafka brokers, only {{$value}} are visible. A broker may be down.

This catches the case where a broker fails to register with ZooKeeper/KRaft but the up metric for the exporter is still healthy (the exporter is running but can't see all brokers).

Under-replicated partitions

yamlcopy- alert: KafkaUnderReplicatedPartitions
  expr: sum(kafka_topic_partition_under_replicated_partition{job="kafka"}) > 0
  for: 10m
  labels:
    severity: high
    service: kafka
    event: 'Kafka Under-Replicated Partitions'
  annotations:
    description: >-
      Kafka has {{$value}} under-replicated partitions for more than 10 min.
      A replica is lagging behind or a broker is slow. This is a leading indicator of data loss risk.

Under-replicated partitions are a leading indicator: they appear before a broker actually goes down. If a broker fails while partitions are under-replicated, some messages have only one copy — no redundancy.

Why 10m and not 5m? Brief under-replication happens during rolling updates and leader elections. A 10-minute for filters out normal operational noise.

Offline partitions

More urgent than under-replicated:

yamlcopy- alert: KafkaOfflinePartitions
  expr: sum(kafka_topic_partition_leader{job="kafka"} < 0) > 0
  for: 1m
  labels:
    severity: disaster
    event: 'Kafka Offline Partitions |{{$value}} partitions'
  annotations:
    description: >-
      {{$value}} Kafka partitions have no leader. Producers and consumers for these
      partitions will fail immediately.

A leader < 0 value indicates no leader is elected for that partition. Producers and consumers for those partitions fail with LEADER_NOT_AVAILABLE.

Consumer lag

yamlcopy- alert: KafkaConsumerLagCritical
  expr: >
    sum(kafka_consumergroup_lag_sum{job="kafka"}) by (consumergroup) > 100000
  for: 5m
  labels:
    severity: average
    service: kafka
    event: 'Kafka Consumer Lag |{{$labels.consumergroup}}'
  annotations:
    description: >-
      Consumer group {{$labels.consumergroup}} lag is {{$value}} messages
      (above threshold of 100 000) for more than 5 min.

The threshold 100000 is workload-specific. Set it based on your consumer throughput:

• Consumer processes 10k messages/sec → lag of 100k = 10 seconds behind (acceptable)
• Consumer processes 100/sec → same lag = 1000 seconds (critical)

Tune thresholds per consumer group with separate rules:

yamlcopy- alert: KafkaHighPriorityConsumerLag
  expr: >
    kafka_consumergroup_lag_sum{job="kafka",
      consumergroup="payment-processor"} > 500
  for: 2m
  labels:
    severity: high
    event: 'Payment Consumer Lag |{{$labels.consumergroup}}'

Consumer group not consuming

An alert for consumer groups that have stopped consuming entirely:

yamlcopy- alert: KafkaConsumerGroupStopped
  expr: >
    delta(kafka_consumergroup_lag_sum{job="kafka"}[5m]) > 1000
    and
    kafka_consumergroup_lag_sum{job="kafka"} > 1000
  for: 5m
  labels:
    severity: high
    event: 'Kafka Consumer Stopped |{{$labels.consumergroup}}'
  annotations:
    description: >-
      Consumer group {{$labels.consumergroup}} lag has grown by more than 1000
      messages in 5 minutes and is not catching up.

This catches lag that's growing (not just high) — meaning the consumer is alive but can't keep up with production rate.

Disk usage (Kafka PVCs)

kafka-exporter doesn't expose disk metrics — those come from kubelet:

yamlcopy- alert: KafkaHighDiskUsage
  expr: >
    sum(kubelet_volume_stats_used_bytes{namespace="kafka"})
      by (namespace, persistentvolumeclaim)
    /
    sum(kubelet_volume_stats_capacity_bytes{namespace="kafka"})
      by (namespace, persistentvolumeclaim)
    > 0.9
  for: 15m
  labels:
    severity: warning
    service: kafka
    event: 'Kafka High Disk |{{$labels.persistentvolumeclaim}}'
  annotations:
    description: >-
      Kafka PVC {{$labels.persistentvolumeclaim}} is above 90% full.
      Consider reducing retention or adding storage.

Kafka has configurable retention (log.retention.hours, log.retention.bytes). If disk is filling up, the quickest fix is to reduce log.retention.hours on the affected topics:

bashcopykafka-configs.sh --bootstrap-server kafka:9092 \
  --entity-type topics \
  --entity-name my-topic \
  --alter --add-config retention.ms=3600000  # 1 hour

Memory usage

yamlcopy- alert: KafkaHighMemoryUsage
  expr: process_resident_memory_bytes{job="kafka"} / 1024 / 1024 / 1024 > 4
  for: 15m
  labels:
    severity: warning
    event: 'Kafka High Memory |{{$labels.instance}}'
  annotations:
    description: Kafka broker {{$labels.instance}} is using more than 4 GB RSS for 15 min.

Kafka brokers are JVM processes. RSS growing above the heap limit usually means page cache usage — not necessarily a problem. But crossing the container memory limit causes OOM kills. This alert gives time to adjust KAFKA_HEAP_OPTS or resources.limits.memory.

For Kubernetes Kafka (Strimzi or Bitnami chart), set JVM heap explicitly:

yamlcopyvalues:
  extraEnvVars:
    - name: KAFKA_HEAP_OPTS
      value: "-Xmx2g -Xms2g"
  resources:
    limits:
      memory: 4Gi  # heap (2g) + off-heap + overhead

Complete PrometheusRule manifest

yamlcopyapiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kafka-alerts
  namespace: observability
  labels:
    release: kube-prom-stack
spec:
  groups:
    - name: kafka
      interval: 30s
      rules:
        # Broker availability
        - alert: KafkaDown
          expr: up{job="kafka"} < 1
          for: 5m
          labels: { severity: disaster, service: kafka }
        - alert: KafkaExporterMissing
          expr: absent(kafka_brokers{job="kafka"})
          for: 5m
          labels: { severity: high, service: kafka }
        # Replication health
        - alert: KafkaUnderReplicatedPartitions
          expr: sum(kafka_topic_partition_under_replicated_partition{job="kafka"}) > 0
          for: 10m
          labels: { severity: high, service: kafka }
        # Consumer lag
        - alert: KafkaConsumerLagCritical
          expr: sum(kafka_consumergroup_lag_sum{job="kafka"}) by (consumergroup) > 100000
          for: 5m
          labels: { severity: average, service: kafka }
        # Disk
        - alert: KafkaHighDiskUsage
          expr: >
            sum(kubelet_volume_stats_used_bytes{namespace="kafka"}) by (persistentvolumeclaim)
            / sum(kubelet_volume_stats_capacity_bytes{namespace="kafka"}) by (persistentvolumeclaim)
            > 0.9
          for: 15m
          labels: { severity: warning, service: kafka }