Multi-cluster Prometheus: remote_write to central storage

Published: 2026-05-10

Each cluster runs its own Prometheus (via kube-prom-stack). For long-term trends, cross-cluster comparisons, and retention beyond what a spoke can store, spoke Prometheus instances remote_write to a central VictoriaMetrics on the infra cluster.

remote_write configuration

In kube-prom-stack values, applied per spoke via a Kustomize patch:

yaml# patches/kube-prom-stack.yaml (dev environment)
spec:
  values:
    prometheus:
      prometheusSpec:
        externalLabels:
          cluster: dev
          environment: dev
        remoteWrite:
          - url: "http://victoria-metrics.observability.infra.svc.cluster.local:8428/api/v1/write"
            queueConfig:
              capacity: 10000
              maxSamplesPerSend: 1000
              maxShards: 10
            writeRelabelConfigs:
              - sourceLabels: [__name__]
                regex: "(node_|container_|kube_|up|probe_).+"
                action: keep
        retention: 3d
        retentionSize: "10GB"

externalLabels.cluster: dev tags every series with the cluster name — essential for cross-cluster dashboards where you need to filter by cluster. retention: 3d keeps spoke storage small since historical data lives in VictoriaMetrics.

The remote_write URL across clusters

On YC clusters and on-prem k3s, VictoriaMetrics is not directly reachable by in-cluster DNS (it's a different cluster). Two options:

Option 1: Expose VictoriaMetrics via APISIX (simpler)

yaml# ApisixRoute on infra
- name: vm-remote-write
  match:
    hosts:
      - vm-write.infra.test.antonnovikov.com
    paths:
      - /api/v1/write
  backends:
    - serviceName: victoria-metrics-victoria-metrics
      serviceNamespace: observability
      servicePort: 8428

yaml# remote_write URL in spoke patches
remoteWrite:
  - url: "https://vm-write.infra.test.antonnovikov.com/api/v1/write"

Option 2: WireGuard VPN between clusters — spokes connect to infra over VPN, use the internal ClusterIP directly.

writeRelabelConfigs for traffic reduction

Sending every Prometheus series to the central store is wasteful. writeRelabelConfigs filters to the most useful metrics:

yamlwriteRelabelConfigs:
  # Keep: infrastructure metrics
  - sourceLabels: [__name__]
    regex: "(node_|container_|kube_state_|up|probe_|kubelet_).+"
    action: keep
  # Drop: high-cardinality noisy metrics
  - sourceLabels: [__name__]
    regex: "(go_gc_|go_memstats_|process_).+"
    action: drop
  # Drop: short-lived pod metrics (replace with recording rules)
  - sourceLabels: [__name__, pod]
    regex: "container_.+;.+-[a-z0-9]{10}-.+"
    action: drop

This reduces remote_write traffic by ~60-70% compared to sending everything.

Cross-cluster Grafana dashboard

With all clusters writing to VictoriaMetrics, a single Grafana dashboard can compare them:

sum by (cluster) (
  rate(container_cpu_usage_seconds_total{container!=""}[5m])
)

Add a template variable to filter:

json{
  "name": "cluster",
  "type": "query",
  "query": "label_values(up, cluster)",
  "datasource": "VictoriaMetrics"
}

The variable populates from the cluster label injected by externalLabels.

Checking remote_write health

bash# On spoke: check remote_write queue
kubectl port-forward -n observability svc/prometheus-operated 9090:9090
# Then visit http://localhost:9090/graph?g0.expr=prometheus_remote_storage_queue_length

# On infra: check VictoriaMetrics ingestion rate
kubectl port-forward -n observability svc/victoria-metrics-victoria-metrics 8428:8428
# Then: http://localhost:8428/api/v1/query?query=sum(rate(vm_rows_inserted_total[5m]))

Remote_write failures show up in prometheus_remote_storage_failed_samples_total. A sustained non-zero value means the central VictoriaMetrics is unreachable or rejecting writes.

VictoriaMetrics retention and storage sizing

VictoriaMetrics on the infra cluster holds metrics from all spokes. Sizing:

yaml# VictoriaMetrics HelmRelease values
server:
  retentionPeriod: 90d
  persistentVolume:
    size: 50Gi
  resources:
    requests:
      memory: 2Gi
      cpu: 500m
    limits:
      memory: 4Gi

With 3 spokes writing ~500k samples/min each, 90 days fits in ~30–40Gi. VictoriaMetrics compresses significantly better than Prometheus TSDB (~10x), so actual disk usage is much lower than the raw sample count would suggest.

Authentication for remote_write

If VictoriaMetrics is exposed over the internet (via APISIX), add basic auth:

yaml# VictoriaMetrics side: enable basic auth
server:
  extraArgs:
    httpAuth.username: "prom-writer"
    httpAuth.password: "strong-password"

yaml# Spoke Prometheus remote_write
remoteWrite:
  - url: "https://vm-write.infra.test.example.com/api/v1/write"
    basicAuth:
      username:
        name: vm-remote-write-secret
        key: username
      password:
        name: vm-remote-write-secret
        key: password

The secret is a SealedSecret committed to the spoke overlay.

Alerting on remote_write lag

Add a PrometheusRule to alert when the remote_write queue is growing:

yaml- alert: RemoteWriteQueueFull
  expr: >
    prometheus_remote_storage_queue_length /
    prometheus_remote_storage_queue_capacity > 0.8
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "remote_write queue > 80% full on {{ $labels.cluster }}"
    description: "VictoriaMetrics may be unreachable or slow"

Summary

Spoke Prometheus instances write to central VictoriaMetrics with externalLabels.cluster tagging every series — mandatory for cross-cluster Grafana dashboards
writeRelabelConfigs filter to infrastructure metrics before writing, reducing traffic by ~60-70%
Spoke retention is set to 3 days; historical data lives in VictoriaMetrics (90 days, 50Gi)
If VictoriaMetrics is exposed over the internet, enable basic auth on the write endpoint
Alert on prometheus_remote_storage_queue_length / capacity > 0.8 to detect when VictoriaMetrics is unreachable before queues overflow

weblog