Prometheus recording rules: pre-aggregating expensive queries

Published: 2026-03-11

Prometheus range queries over high-cardinality labels are slow. A 7-day request rate across hundreds of pods computed on every Grafana panel load adds up. Recording rules solve this by pre-computing the result at scrape time.

How recording rules work

A recording rule is a PromQL expression that runs on an interval and writes the result as a new metric with a simpler name. Grafana queries the pre-computed metric instead of the expensive original expression.

yamlapiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: recording-rules
  namespace: observability
  labels:
    release: kube-prom-stack   # must match Prometheus selector
spec:
  groups:
    - name: recording
      interval: 30s
      rules:
        - record: job:http_requests_total:rate5m
          expr: |
            sum by (job, namespace) (
              rate(http_requests_total[5m])
            )

interval: 30s means the result is updated every 30 seconds. Grafana panels using job:http_requests_total:rate5m respond instantly — no heavy range query at load time.

Naming convention

The standard: level:metric:operations

level — the aggregation level (job, namespace, cluster, instance)
metric — the source metric name
operations — what was done (rate5m, sum, p99, ratio)

Examples:

namespace:container_cpu_usage_seconds:rate5m
job:http_request_duration_seconds:p99
cluster:node_memory_working_set_bytes:sum
job:http_errors:rate5m_ratio

Practical rules for a multi-cluster setup

yamlgroups:
  - name: cluster-resources
    interval: 60s
    rules:
      # CPU usage per namespace — for quota dashboards
      - record: namespace:container_cpu_usage_seconds_total:rate5m
        expr: |
          sum by (namespace, cluster) (
            rate(container_cpu_usage_seconds_total{container!=""}[5m])
          )

      # Memory working set per pod — for pod detail panels
      - record: pod:container_memory_working_set_bytes:sum
        expr: |
          sum by (pod, namespace, cluster) (
            container_memory_working_set_bytes{container!=""}
          )

      # HTTP error rate per job
      - record: job:http_errors:rate5m
        expr: |
          sum by (job, namespace) (
            rate(http_requests_total{status=~"5.."}[5m])
          )
          /
          sum by (job, namespace) (
            rate(http_requests_total[5m])
          )

      # P99 latency per job
      - record: job:http_request_duration_seconds:p99
        expr: |
          histogram_quantile(0.99,
            sum by (job, namespace, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

SLO burn rate

Alerting on SLO burn rate requires range queries over multiple windows. Pre-compute them:

yaml  - name: slo-burn
    interval: 60s
    rules:
      - record: job:slo_error_budget_burn:1h
        expr: |
          (
            sum by (job) (rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum by (job) (rate(http_requests_total[1h]))
          ) / 0.001   # error budget = 0.1% = 0.001

      - record: job:slo_error_budget_burn:6h
        expr: |
          (
            sum by (job) (rate(http_requests_total{status=~"5.."}[6h]))
            /
            sum by (job) (rate(http_requests_total[6h]))
          ) / 0.001

      - record: job:slo_error_budget_burn:24h
        expr: |
          (
            sum by (job) (rate(http_requests_total{status=~"5.."}[24h]))
            /
            sum by (job) (rate(http_requests_total[24h]))
          ) / 0.001

Multi-window alert using these recordings fires faster and with less false-positive noise:

yaml- alert: HighErrorBudgetBurn
  expr: |
    job:slo_error_budget_burn:1h > 14
    and
    job:slo_error_budget_burn:6h > 6
  for: 2m
  labels:
    severity: critical
  annotations:
    description: "{{ $labels.job }} is burning error budget at {{ $value | humanize }}x rate"

The 14x burn rate at 1h means you'll exhaust the monthly error budget in ~2 days at this rate.

Kubernetes resource rules

These are the most commonly needed:

yaml- name: k8s-resource-recording
  interval: 60s
  rules:
    # Node CPU utilization
    - record: node:cpu_utilization:ratio
      expr: |
        1 - avg by (node) (
          rate(node_cpu_seconds_total{mode="idle"}[5m])
        )

    # Node memory pressure
    - record: node:memory_utilization:ratio
      expr: |
        1 - (
          node_memory_MemAvailable_bytes /
          node_memory_MemTotal_bytes
        )

    # PVC usage
    - record: persistentvolumeclaim:disk_usage:ratio
      expr: |
        1 - (
          kubelet_volume_stats_available_bytes /
          kubelet_volume_stats_capacity_bytes
        )

Where to put PrometheusRule objects

Recording rules go in base/monitoring/rules/recording.yaml and are applied on every cluster by the base Kustomization. Per-environment rules (different SLO thresholds for dev vs prod) go in patches/monitoring/rules/.

One PrometheusRule per concern: recording, node alerts, SLO alerts. Keeps the diff small when adjusting thresholds.

Verifying rules are loaded

bashkubectl port-forward -n observability svc/kube-prom-stack-prometheus 9090:9090
# http://localhost:9090/rules — shows all loaded recording and alerting rules

# Check if a recording rule is producing data
curl -s 'http://localhost:9090/api/v1/query?query=job:http_requests_total:rate5m' | jq '.data.result | length'

If the rule isn't in the list, check the PrometheusRule label selector matches the kube-prom-stack configuration.

weblog