abot: Alertmanager proxy for Telegram and Mattermost notifications

Published: 2026-05-12

abot is a small Go service that sits between Alertmanager and the notification channels. Alertmanager sends one webhook; abot fans it out to Telegram and Mattermost with proper formatting, retry logic, and proxy support.

Why not use Alertmanager receivers directly

Alertmanager has native Telegram and Slack receivers, but:

1. The native Telegram receiver doesn't support HTML template with all the formatting we need.
2. Mattermost uses a different webhook format than Slack — there's no native Mattermost receiver.
3. On-prem clusters can't reach api.telegram.org without going through an HTTP proxy. Configuring that proxy in Alertmanager affects all receivers, not just Telegram.
4. abot provides a single place to manage notification logic — adding a new channel (PagerDuty, email) means changing abot, not Alertmanager config.

Architecture

copyAlertmanager
    │  POST /alert (Alertmanager webhook format)
    ▼
abot (port 9444)
    ├── Telegram: api.telegram.org (via HTTP proxy if configured)
    └── Mattermost: chat.example.com/hooks/...

abot runs as a Deployment in observability namespace on every cluster.

Helm chart

The custom chart lives at charts/abot/. Key values:

yamlcopyreplicaCount: 1
image:
  repository: registry.example.com/docker/iac/prometheus/abot2
  tag: v0.78
service:
  port: 9444
  targetPort: 9100
serviceMonitor:
  enabled: true
  namespace: observability
  interval: 30s
  metricRelabelings:
    - sourceLabels: [__name__]
      regex: 'abot_.+'
      action: keep   # only keep abot's own metrics
resources:
  limits:
    cpu: "800m"
    memory: "256Mi"
  requests:
    cpu: "200m"
    memory: "128Mi"

Alertmanager configuration pointing to abot

yamlcopyalertmanager:
  enabled: true
  config:
    receivers:
      - name: abot
        webhook_configs:
          - url: "http://abot.observability.svc.cluster.local:9444/alert"
            send_resolved: true
    route:
      receiver: abot
      group_by: ["alertname", "cluster", "namespace"]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h

send_resolved: true ensures abot receives a notification when an alert clears. group_by prevents a single noisy incident from generating hundreds of individual messages.

Telegram proxy configuration

yamlcopyconfig:
  telegram:
    bot_token: "${TG_BOT_TOKEN}"
    chat_id: "-1003960777636"
    proxy: "http://10.16.91.109:3128"
    parse_mode: "HTML"

The proxy is an internal tinyproxy instance on the on-prem network. For YC clusters (sre, loadgds, demo) the proxy is not needed — they have direct internet access. The patch for YC environments removes the proxy field.

Message format

abot renders the Alertmanager payload using Go templates. The Telegram message uses HTML:

copy🔴 · CRITICAL · <a href="https://grafana.infra.test.antonnovikov.com">dashboard</a>

🖥 dev
📦 elasticsearch · elasticsearch-master-0
💬 JVM heap usage above 85% for 5 minutes
⏰ 2026-08-25 14:32:01

The Mattermost message uses Markdown with an attachment for coloring:

jsoncopy{
  "attachments": [{
    "color": "#cc0000",
    "text": "🔴 **CRITICAL** · [dashboard](...)\n🖥 `dev`\n📦 `elasticsearch`\n💬 JVM heap..."
  }]
}

Colors by severity: #cc0000 (critical), #ff8c00 (warning), #00cc44 (resolved).

Per-environment abot config via patches

The bot token and chat ID differ between environments (different Telegram groups for dev vs production alerts):

yamlcopy# projects/dev/kustomization/hub/patches/abot.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: abot
  namespace: dev
spec:
  values:
    config:
      telegram:
        chat_id: "-1009999999999"   # dev team chat
      mattermost:
        webhook_url: "https://chat.example.com/hooks/dev-channel-hook"

Critical production alerts go to a separate higher-priority chat.

abot's own metrics

abot exposes metrics about its notification delivery:

These feed into Prometheus via ServiceMonitor. A PrometheusRule can alert on delivery failures:

yamlcopy- alert: AbotNotificationFailed
  expr: increase(abot_notifications_total{status="failed"}[15m]) > 3
  for: 0m
  labels:
    severity: warning
  annotations:
    summary: "abot failed to deliver {{ $value }} notifications in 15m"

Debugging delivery issues

bashcopy# Check abot logs
kubectl logs -n observability deploy/abot

# Test webhook manually
kubectl port-forward -n observability svc/abot 9444:9444
curl -X POST http://localhost:9444/alert \
  -H "Content-Type: application/json" \
  -d '{"alerts":[{"status":"firing","labels":{"alertname":"TestAlert","severity":"warning"},"annotations":{"summary":"Test"}}]}'

If Telegram is unreachable from the pod, the logs will show connection errors to api.telegram.org. Solution: ensure the proxy is set correctly or check tinyproxy is running on the configured address.

Summary

• abot sits between Alertmanager and notification channels, handling Telegram HTML formatting, Mattermost webhooks, and proxy support in one place
• Single Alertmanager webhook receiver (url: http://abot.observability.svc.cluster.local:9444/alert) covers all channels
• Per-environment bot tokens and chat IDs are injected via Kustomize patches on the HelmRelease values
• abot exposes its own delivery metrics via ServiceMonitor — alert on abot_notifications_total{status="failed"} to catch silent delivery failures
• On-prem clusters route Telegram traffic through tinyproxy; YC clusters with direct internet access skip the proxy via a patch