Grafana datasource and dashboard provisioning via ConfigMap

Published: 2026-03-16

Grafana supports declarative provisioning: datasources and dashboards defined as YAML/JSON files mounted into the pod. Combined with Flux, dashboards live in git and are applied automatically — no manual import, no "someone deleted the dashboard" incidents.

Datasource provisioning

kube-prom-stack exposes datasource provisioning through Helm values:

yamlgrafana:
  additionalDataSources:
    - name: Prometheus-dev
      type: prometheus
      url: http://prometheus-operated.observability.svc.cluster.local:9090
      isDefault: false
      jsonData:
        timeInterval: "30s"
        exemplarTraceIdDestinations:
          - name: trace_id
            datasourceUid: tempo

    - name: Prometheus-test
      type: prometheus
      url: http://prometheus.test.svc.cluster.local:9090
      isDefault: false

    - name: VictoriaMetrics
      type: prometheus
      url: http://victoria-metrics.observability.svc.cluster.local:8428
      isDefault: false

    - name: Elasticsearch
      type: elasticsearch
      url: http://elasticsearch-master.observability.svc.cluster.local:9200
      jsonData:
        index: "vector-*"
        timeField: "@timestamp"
        logMessageField: message

    - name: Loki-VictoriaLogs
      type: loki
      url: http://victoria-logs.observability.svc.cluster.local:9428

All these datasources appear in Grafana on startup without any manual configuration.

Dashboard provisioning via sidecar

yaml# grafana Helm values
grafana:
  sidecar:
    dashboards:
      enabled: true
      label: grafana_dashboard
      labelValue: "1"
      searchNamespace: ALL
      folderAnnotation: grafana_folder
      provider:
        foldersFromFilesStructure: true

The sidecar container watches all ConfigMaps with grafana_dashboard: "1" across all namespaces and hot-reloads them into Grafana without a pod restart.

Create ConfigMaps from dashboard JSON files

yaml# kustomization.yaml
configMapGenerator:
  - name: grafana-dashboards-infra
    namespace: observability
    options:
      disableNameSuffixHash: true
    files:
      - dashboards/cluster-overview.json
      - dashboards/flux-reconciliation.json
      - dashboards/elasticsearch-cluster.json

Add the label and folder annotation:

yaml# configmap patch
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboards-infra
  namespace: observability
  labels:
    grafana_dashboard: "1"
  annotations:
    grafana_folder: "Infrastructure"

Dashboard JSON in git

Grafana has an "Export" button that produces the JSON. Before committing:

Strip the id field (it's cluster-specific)
Replace hardcoded datasource UIDs with template variables:

json{
  "templating": {
    "list": [
      {
        "name": "datasource",
        "type": "datasource",
        "query": "prometheus"
      }
    ]
  },
  "panels": [
    {
      "datasource": {"type": "prometheus", "uid": "${datasource}"}
    }
  ]
}

Dashboard variables make the same JSON work across dev, sre, and infra datasources.

Folder structure

dashboards/
  infrastructure/
    cluster-overview.json
    node-exporter.json
  application/
    apisix.json
    elasticsearch.json
  flux/
    flux-control-plane.json
    flux-reconciliation.json

Each subdirectory maps to a Grafana folder via grafana_folder annotation.

Alerting contact points as code

Grafana alerting contact points can also be provisioned via ConfigMap:

yamlgrafana:
  grafana.ini:
    alerting:
      enabled: true
  notifiers: []  # legacy
  alerting:
    contactpoints.yaml:
      apiVersion: 1
      contactPoints:
        - orgId: 1
          name: Telegram
          receivers:
            - uid: telegram-receiver
              type: telegram
              settings:
                bottoken: "${TELEGRAM_BOT_TOKEN}"
                chatid: "${TELEGRAM_CHAT_ID}"

Or use the grafana-contact-point secret provisioning approach for sensitive values.

Troubleshooting

Dashboards not loading:

bashkubectl logs -n observability deploy/kube-prom-stack-grafana -c grafana-sc-dashboard

Check for "no dashboards found" or ConfigMap label mismatch.

Datasource connection errors: Verify the service DNS name — it must match the service name and namespace of the Prometheus/ES instance. Use kubectl exec to test connectivity from inside the Grafana pod.

Duplicate dashboards after upgrade: If disableNameSuffixHash: false, Kustomize appends a hash suffix to ConfigMap names on every change. The sidecar creates new dashboards but doesn't clean up old ones. Use disableNameSuffixHash: true for dashboard ConfigMaps.

weblog