Writing good PrometheusRules: labels, Grafana links, and humanizeTimestamp

Published: 2026-02-16

A PrometheusRule that fires at the right time and gives the on-call person enough context to act is worth ten dashboards. Bad alerts are either noisy (fire on every transient blip) or mute (misconfigured label selectors, never reach Alertmanager). Here's the pattern we settled on after iterating through dozens of production alerts.

The anatomy of an alert rule

yamlcopyapiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: my-service-alerts
  namespace: observability
  labels:
    release: kube-prom-stack    # must match Prometheus operator selector
spec:
  groups:
    - name: my-service
      interval: 30s
      rules:
        - alert: MyServiceDown
          expr: up{job="my-service"} < 1
          for: 5m
          labels:
            service: my-service
            severity: disaster
            groupp: admin
            url: 'https://grafana.example.com/d/my-service/overview'
            event: 'Service Down |{{$labels.job}}|{{$labels.instance}}'
          annotations:
            description: >-
              Service {{$labels.instance}} has been down for more than 5 min,
              timestamp: {{ with query "time()+10800" }}
              {{ . | first | value | humanizeTimestamp }}{{ end }}

The release: kube-prom-stack label is how the kube-prometheus-stack operator discovers PrometheusRule resources. Without it, the rule is silently ignored — no error, no warning. Always verify with:

bashcopykubectl get prometheusrule -n observability my-service-alerts -o yaml | grep -A5 "labels:"

And check that Prometheus loaded it:

bashcopykubectl port-forward -n observability svc/kube-prom-stack-prometheus 9090:9090
# then open localhost:9090/rules in browser

Why interval: 30s on the group

The default evaluation interval for Prometheus is 1m. For production services where you want fast alerting, set the group interval to 30s. This overrides the global interval for this specific group without touching everything else.

Setting the global interval to 30s affects every rule including recording rules that may be expensive to evaluate. Per-group interval is the surgical approach.

For non-critical alerts (disk usage, queue depth trends), 2m is sufficient and reduces load.

Labels for routing

Our Alertmanager routes on three label dimensions:

Note the double-p in groupp — this is intentional to avoid collision with the reserved Prometheus label group.

Alertmanager routing config

yamlcopyroute:
  group_by: ['alertname', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: disaster
      receiver: pagerduty-critical
      continue: false
    - match:
        groupp: admin
      receiver: telegram-admin
    - match:
        groupp: dev
      receiver: telegram-dev

The group_by: ['alertname', 'service'] ensures that if three instances of my-service go down simultaneously, they're bundled into one notification rather than three separate pages.

The url label: Grafana dashboard link

Put the Grafana dashboard URL in a label so Alertmanager can include it in the notification template:

yamlcopylabels:
  url: 'https://grafana.example.com/d/my-dashboard/my-service'

In the Alertmanager receiver template:

copy{{ range .Alerts }}
*{{ .Labels.event }}*
{{ .Annotations.description }}
🔗 {{ .Labels.url }}
{{ end }}

The on-call person gets a direct link to the relevant dashboard in the notification. No searching through Grafana's search interface while half-asleep at 3am.

For time-scoped links that pre-set the dashboard time range to "around when the alert fired":

yamlcopyurl: 'https://grafana.example.com/d/my-dashboard/my-service?from=now-1h&to=now'

The event label: human-readable title

The event label is a short description of what fired, including key label values:

yamlcopyevent: 'Kafka Down |{{$labels.job}}|{{$labels.instance}}'

This becomes the notification title. The pipe-separated context makes it easy to see at a glance which instance is affected. Without {{$labels.instance}}, you'd get "Kafka Down" with no indication of which broker.

More examples:

yamlcopyevent: 'High Error Rate |{{$labels.service}}| {{$value | humanize}}%'
event: 'Disk Full |{{$labels.node}}| {{$value | humanize}}% used'
event: 'PVC Unbound |{{$labels.persistentvolumeclaim}}|{{$labels.namespace}}'

humanizeTimestamp for timestamps

Prometheus alert annotations are evaluated at alert-time and can include PromQL expressions. Always add a human-readable timestamp so the on-call person knows exactly when it fired without having to correlate with Alertmanager's own timestamp:

yamlcopyannotations:
  description: >-
    {{ .Labels.instance }} has been down for 5 min,
    timestamp: {{ with query "time()+10800" }}
    {{ . | first | value | humanizeTimestamp }}{{ end }}

time() returns Unix epoch. +10800 adjusts for UTC+3 (adjust for your timezone: UTC+0 → +0, UTC+2 → +7200). humanizeTimestamp renders it as 2026-07-30 14:23:00 +0000 UTC.

Other useful humanize functions:

• humanize — converts 1048576 to 1.05M
• humanize1024 — converts 1048576 to 1.00Mi
• humanizeDuration — converts 3665 to 1h 1m 5s

for: duration — don't fire on flaps

Always set for:. Without it, a single scrape failure fires the alert immediately. Reasonable values:

Setting for: 0s (or omitting for:) is valid for alerts that should fire on the first evaluation — but be aware this causes instant firing on any scrape failure, which is noisy.

Group interval vs for

These are different concepts:

• interval: 30s (on the group) — how often Prometheus evaluates the rule expression
• for: 5m — how long the expression must return a positive result before the alert transitions from pending to firing

A 5-minute for with 30-second evaluation means Prometheus checks every 30 seconds and fires after 10 consecutive positive evaluations. If the condition clears at evaluation 7, the alert goes back to pending and the counter resets.

Check pending alerts via the Prometheus UI at /alerts or via API:

bashcopycurl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.state=="pending")'

Recording rules for expensive expressions

If an alert expression is evaluated 50+ times per minute across multiple alert groups, create a recording rule to precompute it:

yamlcopygroups:
  - name: my-service-recordings
    interval: 30s
    rules:
      - record: job:request_error_rate:5m
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
          /
          sum(rate(http_requests_total[5m])) by (job)

  - name: my-service-alerts
    interval: 30s
    rules:
      - alert: HighErrorRate
        expr: job:request_error_rate:5m > 0.05
        for: 5m

Recording rules use the job:metric_name:duration naming convention. The alert expression then references the precomputed metric instead of repeating the expensive range query.

Testing alert rules locally

bashcopy# Install promtool (ships with Prometheus)
promtool check rules my-prometheusrule.yaml

# Unit test
cat > test.yaml << 'EOF'
rule_files:
  - my-prometheusrule.yaml
tests:
  - interval: 1m
    input_series:
      - series: 'up{job="my-service",instance="host:9090"}'
        values: '1 1 1 0 0 0 0 0 0'
    alert_rule_test:
      - eval_time: 8m
        alertname: MyServiceDown
        exp_alerts:
          - exp_labels:
              severity: disaster
              instance: host:9090
EOF
promtool test rules test.yaml

Unit tests prevent regressions when refactoring alert expressions. The values: '1 1 1 0 0 0 0 0 0' syntax represents metric values at each interval step.

Common mistakes

1. Missing release label — PrometheusRule silently not loaded
2. for: too short — floods on-call with transient alerts during rolling updates
3. No service label — Alertmanager bundles unrelated alerts together
4. Duplicate alert names — second rule overwrites the first silently
5. PromQL errors in annotations — alert fires but description shows <error>; test with promtool check rules