kube-prometheus-stack: the full monitoring setup

Published: 2026-03-18

kube-prometheus-stack is a single Helm chart that deploys Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics, and all the default recording rules and dashboards. It's the standard way to get Kubernetes observability up in one shot.

HelmRelease

yamlapiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: kube-prom-stack
  namespace: monitoring
spec:
  chart:
    spec:
      chart: kube-prometheus-stack
      version: "65.x"
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
        namespace: flux-system
  values:
    fullnameOverride: kube-prom-stack

    prometheus:
      prometheusSpec:
        retention: 30d
        retentionSize: 40GB
        storageSpec:
          volumeClaimTemplate:
            spec:
              storageClassName: local-path
              resources:
                requests:
                  storage: 50Gi
        serviceMonitorSelectorNilUsesHelmValues: false
        podMonitorSelectorNilUsesHelmValues: false
        ruleSelectorNilUsesHelmValues: false
        resources:
          requests:
            cpu: 500m
            memory: 2Gi
          limits:
            cpu: 2000m
            memory: 4Gi

    alertmanager:
      alertmanagerSpec:
        storage:
          volumeClaimTemplate:
            spec:
              storageClassName: local-path
              resources:
                requests:
                  storage: 5Gi

    grafana:
      adminPassword: ""    # set via secret
      persistence:
        enabled: true
        storageClassName: local-path
        size: 10Gi
      grafana.ini:
        server:
          root_url: https://grafana.example.com
      sidecar:
        dashboards:
          enabled: true
          searchNamespace: ALL

    nodeExporter:
      enabled: true

    kubeStateMetrics:
      enabled: true

serviceMonitorSelectorNilUsesHelmValues: false is critical. Without it, Prometheus only watches ServiceMonitors in the release namespace. Setting it to false makes Prometheus watch all namespaces. Same for podMonitorSelectorNilUsesHelmValues and ruleSelectorNilUsesHelmValues.

Grafana admin password via secret

yamlgrafana:
  admin:
    existingSecret: grafana-admin-secret
    userKey: admin-user
    passwordKey: admin-password

bashkubectl create secret generic grafana-admin-secret \
  --from-literal=admin-user=admin \
  --from-literal=admin-password=REDACTED \
  --dry-run=client -o yaml \
  | kubeseal --cert infra.pem -o yaml \
  > fluxcd/custom/monitoring/grafana-admin-secret.yaml

ServiceMonitor for your app

yamlapiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-service
  namespace: app
  labels:
    release: kube-prom-stack    # must match Prometheus's serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: my-service
  namespaceSelector:
    matchNames:
      - app
  endpoints:
    - port: http
      path: /metrics
      interval: 30s

The release: kube-prom-stack label is required. Without it, Prometheus ignores the ServiceMonitor.

Retention and storage tuning

With retention: 30d and retentionSize: 40GB, Prometheus drops data when either limit is reached.

For a medium cluster (50 nodes, 200 pods):

~10 MB/s ingest rate
Prometheus uses snappy compression (~70% reduction)
Effective: ~7.5 GB/day
30 days ≈ 225 GB compressed

50 GB is tight. Tune retention down to 14d, increase the PVC, or use remote_write to VictoriaMetrics for long-term storage.

Verifying everything works

bash# Prometheus targets
kubectl port-forward -n monitoring svc/kube-prom-stack-prometheus 9090:9090
# http://localhost:9090/targets — all should be UP

# Check Alertmanager config
kubectl port-forward -n monitoring svc/kube-prom-stack-alertmanager 9093:9093
# http://localhost:9093/#/status

# Check a ServiceMonitor is picked up
kubectl get servicemonitor -A | grep my-service
# Then check http://localhost:9090/targets for your service

Default dashboards included

Out of the box:

Kubernetes / Nodes — CPU, memory, disk per node
Kubernetes / Pods — per-pod resource usage
Kubernetes / Namespaces — aggregate per namespace
Alertmanager — alert history and routing
Node Exporter — full hardware metrics

Adding custom dashboards

yamlapiVersion: v1
kind: ConfigMap
metadata:
  name: my-custom-dashboard
  namespace: monitoring
  labels:
    grafana_dashboard: "1"
data:
  my-dashboard.json: |
    { ... }

With sidecar.dashboards.searchNamespace: ALL, Grafana picks up dashboards from any namespace. Deploy the ConfigMap with your application.

Alertmanager configuration

yamlalertmanager:
  config:
    global:
      resolve_timeout: 5m
    route:
      group_by: ['alertname', 'job']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'telegram'
      routes:
        - matchers:
            - severity = "critical"
          receiver: 'telegram-critical'
    receivers:
      - name: 'telegram'
        telegram_configs:
          - bot_token: "${TELEGRAM_BOT_TOKEN}"
            chat_id: -123456789
            message: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"
      - name: 'telegram-critical'
        telegram_configs:
          - bot_token: "${TELEGRAM_BOT_TOKEN}"
            chat_id: -987654321

weblog