Alertmanager routing: Telegram notifications with inhibition rules

Published: 2026-03-20

Alertmanager receives alerts from Prometheus and routes them to receivers. Without a thoughtful route tree, you get paged for 200 alerts when a single node goes down. With inhibition rules and grouped routing, one alert fires and the rest are silenced automatically.

The configuration structure

yamlalertmanager:
  config:
    global:
      resolve_timeout: 5m

    route:
      receiver: "null"
      group_by: [alertname, cluster, namespace]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      routes:
        - matchers: [severity=critical]
          receiver: telegram-critical
          repeat_interval: 4h
        - matchers: [severity=warning]
          receiver: telegram-warnings
          repeat_interval: 24h
        - matchers: [alertname=Watchdog]
          receiver: "null"

    inhibit_rules:
      - source_matchers: [severity=critical]
        target_matchers: [severity=warning]
        equal: [alertname, namespace]
      - source_matchers: [alertname=KubeNodeNotReady]
        target_matchers: [alertname=KubePodNotRunning]
        equal: [node]

    receivers:
      - name: "null"
      - name: telegram-critical
        telegram_configs:
          - bot_token: "${TELEGRAM_BOT_TOKEN}"
            chat_id: -100XXXXXXXXX
            message: |
              🔴 *{{ .GroupLabels.alertname }}*
              {{ range .Alerts }}
              *Namespace:* {{ .Labels.namespace }}
              *Summary:* {{ .Annotations.summary }}
              {{ end }}
      - name: telegram-warnings
        telegram_configs:
          - bot_token: "${TELEGRAM_BOT_TOKEN}"
            chat_id: -100XXXXXXXXX
            message: |
              ⚠️ *{{ .GroupLabels.alertname }}*
              {{ range .Alerts }}
              *Namespace:* {{ .Labels.namespace }}
              {{ .Annotations.description }}
              {{ end }}

Telegram bot setup

1. Create a bot via @BotFather, save the token
2. Add the bot to a group channel
3. Get the chat_id: curl https://api.telegram.org/bot<TOKEN>/getUpdates
4. chat_id for groups is negative (-100XXXXXXXXX)

Store the token in a SealedSecret, reference it in Alertmanager config:

yamlalertmanager:
  alertmanagerSpec:
    configSecret: alertmanager-secret

Inhibition rules explained

Inhibition suppresses target alerts when a matching source alert is firing.

yaml- source_matchers: [alertname=KubeNodeNotReady]
  target_matchers: [alertname=KubePodNotRunning]
  equal: [node]

If KubeNodeNotReady fires for node=worker-1, all KubePodNotRunning alerts with node=worker-1 are suppressed. You already know the node is down — you don't need 20 pod alerts confirming it.

yaml- source_matchers: [severity=critical]
  target_matchers: [severity=warning]
  equal: [alertname, namespace]

If a critical fires, suppress warnings with the same name in the same namespace.

Grouping explained

group_wait: 30s — wait 30s after first alert before sending, collecting more alerts
group_interval: 5m — minimum time between sending for the same group when new alerts arrive
repeat_interval: 12h — re-notify every 12h if still firing. Set 4h for critical, 24h for warnings

Silence during maintenance

bashamtool --alertmanager.url=http://alertmanager.monitoring.svc:9093 \
  silence add \
  alertname=KubeNodeNotReady \
  --duration=2h \
  --comment="Scheduled maintenance on worker-1"

Or via the Alertmanager UI.

Testing the routing

bash# Test which receiver handles an alert
amtool --alertmanager.url=http://alertmanager.monitoring.svc:9093 \
  config routes test \
  --verify.receivers=telegram-critical \
  alertname="KubeNodeNotReady" severity="critical"

# List all firing alerts and their state
amtool --alertmanager.url=http://alertmanager.monitoring.svc:9093 \
  alert query

# Check inhibitions
amtool --alertmanager.url=http://alertmanager.monitoring.svc:9093 \
  alert query --inhibited

Common pitfalls

All alerts going to "null": Missing a route or wrong matchers syntax. Use amtool config routes test to debug.

Telegram bot not delivering: The bot must be an admin in the group to send messages. Check the chat_id sign (group IDs are negative).

Alerts not resolving: resolve_timeout: 5m in global config. If Prometheus stops sending an alert (scrape fails), Alertmanager waits 5 minutes before marking it resolved.

Too many notifications for the same alert: Increase repeat_interval. During incident response you don't want to be re-notified every 4 hours.

weblog