Alertmanager routing to Telegram and Mattermost

Published: 2026-03-04

Alertmanager routes cluster alerts to two places: Telegram for mobile visibility, Mattermost for team threads. Both use the same custom Go template embedded directly in the kube-prom-stack HelmRelease values. The key architecture decision: Alertmanager doesn't talk to Telegram or Mattermost directly — it sends all alerts to abot, a small fan-out proxy.

The Telegram template

The template lives in alertmanager.templateFiles inside the HelmRelease values:

copy{{- define "telegram.default.message" -}}
{{- $alert := index .Alerts 0 -}}
{{- $desc := reReplaceAll ",?\\s*timestamp:.*$" "" $alert.Annotations.description -}}
{{- if eq .Status "resolved" }}✅
{{- else }}
  {{- with .CommonLabels.severity }}
    {{- if eq . "critical" }}🔴
    {{- else if eq . "warning" }}🟡
    {{- else if eq . "info" }}🔵
    {{- else }}🟠{{ end }}
  {{- else }}🔴{{ end }}
{{- end }} · {{ .CommonLabels.severity | toUpper }} · <a href="{{ .CommonLabels.grafana_url }}">dashboard</a>

🖥 {{ .CommonLabels.cluster }}
📦 {{ .CommonLabels.namespace }}{{ if .CommonLabels.pod }} · {{ .CommonLabels.pod }}{{ end }}
💬 {{ $desc }}
⏰ {{ $alert.StartsAt.Local.Format "2006-01-02 15:04:05" }}
{{- end -}}

Key details:

• $desc strips timestamp: ... suffixes from descriptions — Alertmanager sometimes appends them from label values.
• Severity maps to a color emoji: 🔴 critical, 🟡 warning, 🔵 info, 🟠 anything else.
• The cluster label is injected by externalLabels in the Prometheus spec — every alert carries which cluster it came from.
• Resolved alerts always get ✅ regardless of severity.

The Mattermost template

Identical logic, Markdown instead of HTML:

copy{{- define "mattermost.default.message" -}}
{{- $alert := index .Alerts 0 -}}
{{- $desc := reReplaceAll ",?\\s*timestamp:.*$" "" $alert.Annotations.description -}}
{{- if eq .Status "resolved" }}✅{{ else }}
  {{- with .CommonLabels.severity }}
    {{- if eq . "critical" }}🔴{{ else if eq . "warning" }}🟡{{ else }}🔵{{ end }}
  {{- end }}
{{- end }} **{{ .CommonLabels.severity | toUpper }}** · [dashboard]({{ .CommonLabels.grafana_url }})
🖥 `{{ .CommonLabels.cluster }}`
📦 `{{ .CommonLabels.namespace }}`
💬 {{ $desc }}
⏰ {{ $alert.StartsAt.Local.Format "2006-01-02 15:04:05" }}
{{- end -}}

Mattermost renders Markdown in webhook payloads. Telegram uses HTML with parse_mode=HTML.

Receiver configuration

yamlcopyalertmanager:
  enabled: true
  config:
    global:
      resolve_timeout: 5m

    route:
      group_by: [cluster, namespace, alertname]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: telegram-mattermost
      routes:
        - matchers:
            - severity = "critical"
          repeat_interval: 1h
          receiver: telegram-mattermost
        - matchers:
            - alertname = "Watchdog"
          receiver: "null"   # suppress the heartbeat alert

    receivers:
      - name: "null"
      - name: telegram-mattermost
        webhook_configs:
          - url: "http://abot.observability.svc.cluster.local:9444/alert"
            send_resolved: true

Why abot instead of direct webhook

Alertmanager's native Telegram receiver has limitations and Mattermost support requires a specific webhook format. abot provides:

1. A single Alertmanager webhook endpoint — no N receivers for N backends.
2. Fan-out to Telegram, Mattermost, and anything else.
3. Template rendering in Go with full control over the message format.
4. Built-in retry with exponential backoff.
5. HTTP proxy support — on-prem clusters can't reach api.telegram.org directly.

Telegram proxy

On-prem clusters route Telegram requests through an internal HTTP proxy:

yamlcopyconfig:
  telegram:
    botToken: "..."
    chatId: "-100..."
    proxy: "http://10.16.91.109:3128"
  mattermost:
    webhookUrl: "https://mattermost.internal/hooks/..."

The proxy is a tinyproxy instance on the internal network that forwards to the public internet.

group_by and repeat_interval

group_by: [cluster, namespace, alertname] ensures alerts from different clusters fire as separate groups. Without cluster in group_by, a flapping alert on dev could suppress the same alert on prod.

Routing to different channels by severity

Route critical alerts to a separate on-call channel:

yamlcopyroutes:
  - matchers:
      - severity = "critical"
    receiver: telegram-oncall
  - matchers:
      - severity = "warning"
    receiver: telegram-mattermost

Add a second receiver with a different chat ID for critical-only alerts.

Testing alert routing

bashcopy# Send a test alert to Alertmanager
curl -X POST http://alertmanager.observability.svc.cluster.local:9093/api/v2/alerts \
  -H 'Content-Type: application/json' \
  -d '[{
    "labels": {
      "alertname": "TestAlert",
      "severity": "warning",
      "cluster": "dev",
      "namespace": "app"
    },
    "annotations": {
      "description": "This is a test"
    }
  }]'

# Check abot logs
kubectl logs -n observability deployment/abot --context=dev-k8s -f

# Check active alerts
kubectl port-forward -n observability svc/alertmanager 9093:9093
curl -s http://localhost:9093/api/v2/alerts | jq '[.[] | {name: .labels.alertname, status: .status.state}]'

Common issues

Alerts not firing:

• Check group_wait — first alert in a group waits 30s before being sent.
• Check repeat_interval — if alert fired recently, it's silenced until the interval passes.
• Check inhibition rules — a critical alert might be inhibiting warning alerts.

Resolved alerts not sent:

• send_resolved: true must be set in the webhook config.

Template errors:

• alertmanager.yaml has an unusual template syntax — use alertmanager --check-config to validate.