kube-prometheus-stack on multi-cluster: spoke Prometheus, central Grafana

Published: 2026-02-27

Every spoke cluster runs its own Prometheus. Grafana runs once, on infra, and federates from all of them. No Thanos, no remote-write, no extra moving parts.

Why one Grafana on infra

Cross-cluster Grafana with multiple Prometheus datasources is simpler than remote-write or Thanos when you have fewer than ten clusters. Each Prometheus retains 7 days of data locally. Grafana queries them directly using cluster-specific datasources. Long-term storage goes to VictoriaLogs (for logs) — metrics beyond 7 days aren't needed in this setup.

Comparison:

For ≤10 clusters with 7-day retention needs, multi-datasource Grafana is the pragmatic choice.

Spoke HelmRelease: Grafana disabled

The base/monitoring/kube-prom-stack/helmrelease.yaml applies to every spoke:

yamlcopyvalues:
  grafana:
    enabled: false        # central Grafana on infra only

  defaultRules:
    create: false         # custom rules managed separately

  kubeScheduler:
    enabled: false        # k3s doesn't expose scheduler metrics
  kubeProxy:
    enabled: false
  kubeControllerManager:
    enabled: false

  alertmanager:
    enabled: false        # Alertmanager disabled in base

kubeScheduler, kubeProxy, and kubeControllerManager are disabled because k3s doesn't expose these component metrics endpoints by default.

defaultRules: create: false because custom PrometheusRules are managed in a separate Kustomization — this avoids conflicts between chart-generated rules and custom ones.

Prometheus image

The standard quay.io/prometheus/prometheus image is replaced with prompp/prompp — a drop-in replacement with PromQL improvements:

yamlcopyprometheus:
  prometheusSpec:
    image:
      registry: registry.deckhouse.io
      repository: prompp/prompp
      tag: "0.7.6"
    version: v3.2.1
    securityContext:
      fsGroup: 64535
      runAsGroup: 64535
      runAsNonRoot: true
    retentionSize: "18GB"

retentionSize prevents the PVC from filling up when retention: 7d is set. Without it, a traffic spike can produce more than expected data volume.

Per-env patches

Each environment has a patch that sets environment-specific things: storageClass and retention:

yamlcopy# projects/dev/kustomization/hub/patches/kube-prom-stack.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kube-prom-stack
  namespace: dev
spec:
  values:
    prometheus:
      prometheusSpec:
        retention: 7d
        storageSpec:
          volumeClaimTemplate:
            spec:
              storageClassName: local-path   # on-prem
              resources:
                requests:
                  storage: 20Gi
        externalLabels:
          cluster: dev                       # label all metrics with cluster name

The externalLabels.cluster is crucial for multi-cluster Grafana — it lets you filter dashboards by cluster.

For Yandex Cloud environments:

yamlcopystorageClassName: yc-network-ssd
tolerations:
  - key: "node.kubernetes.io/not-ready"
    operator: "Exists"

Alertmanager on infra only

Alertmanager is disabled in the base HelmRelease. Only infra enables it via patch, with routing rules for Telegram and Mattermost. Spoke Prometheus instances send their alerts to the infra Alertmanager via the standard Prometheus alerting configuration.

yamlcopy# infra patch
prometheus:
  prometheusSpec:
    alerting:
      alertmanagers:
        - namespace: observability
          name: alertmanager-operated
          port: web

ServiceMonitor convention

All custom exporters use labels: { release: kube-prom-stack } on their ServiceMonitor objects. The Prometheus operator discovers ServiceMonitors with this label:

yamlcopyprometheus:
  prometheusSpec:
    serviceMonitorSelector:
      matchLabels:
        release: kube-prom-stack
    podMonitorSelector:
      matchLabels:
        release: kube-prom-stack

Application metrics sit on port 9595 in the mon namespace. One ServiceMonitor covers all app services by matching the app label selector.

Grafana on infra

The infra patch enables Grafana and adds all spoke datasources:

yamlcopygrafana:
  enabled: true
  ingress:
    enabled: true
    hosts:
      - grafana.infra.test.antonnovikov.com
  additionalDataSources:
    - name: prometheus-dev
      type: prometheus
      url: http://prometheus-operated.observability.dev.svc.cluster.local:9090
      jsonData:
        timeInterval: "30s"
    - name: prometheus-staging
      type: prometheus
      url: http://prometheus-operated.observability.staging.svc.cluster.local:9090

Each datasource corresponds to one spoke cluster. In Grafana dashboards, switch between clusters by changing the datasource variable.

Checking status

bashcopy# All Prometheus instances across all spokes
kubectl get prometheus -A --context=infra-k8s

# Targets on a specific spoke
kubectl port-forward \
  -n observability svc/prometheus-operated 9090:9090 \
  --context=dev-k8s
# then open localhost:9090/targets

# Check that externalLabels are applied
curl -s http://localhost:9090/api/v1/labels | jq '.data[] | select(. == "cluster")'

Troubleshooting

bashcopy# Prometheus not discovering ServiceMonitors
kubectl get servicemonitor -A --context=dev-k8s
kubectl describe prometheus -n observability --context=dev-k8s | grep -A5 "Status"

# Check TSDB head stats (storage pressure)
curl -s http://localhost:9090/api/v1/status/tsdb | jq .data.headStats

# Reload Prometheus config without restart
kubectl exec -n observability prometheus-kube-prom-stack-prometheus-0 -- \
  kill -HUP 1