VictoriaMetrics Operator: VMAgent, VMSingle, VMServiceScrape

Published: 2026-03-06

The VictoriaMetrics Operator is an alternative to the Prometheus Operator. It uses the same CRDs in spirit (VMServiceScrape instead of ServiceMonitor, VMRule instead of PrometheusRule) but the storage backend is VictoriaMetrics — which means better compression, higher cardinality tolerance, and a simpler single-binary deployment.

Why both kube-prom-stack and VM Operator

The infra runs kube-prom-stack on most clusters (mainly for Grafana and the Alertmanager ecosystem). VictoriaMetrics is used as the central long-term storage: spoke Prometheus instances remote_write to a VMSingle on the infra cluster, giving 6 months of retention without disk pressure on spoke clusters.

Architecture:

• Spoke clusters: kube-prom-stack → collects metrics, evaluates alerts → remote_write to central VMSingle
• Infra cluster: VMSingle stores long-term data, Grafana queries both local Prometheus and remote VMSingle

VMSingle

yamlcopyapiVersion: operator.victoriametrics.com/v1beta1
kind: VMSingle
metadata:
  name: victoria-metrics
  namespace: observability
spec:
  retentionPeriod: "6"  # months
  storage:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 50Gi
    storageClassName: yc-network-ssd
  resources:
    requests:
      cpu: 500m
      memory: 1Gi
    limits:
      cpu: 2
      memory: 3Gi
  extraArgs:
    selfScrapeInterval: "10s"
    # Enable deduplication for remote_write from multiple Prometheus replicas
    dedup.minScrapeInterval: "30s"

VMSingle is a single-process deployment — no Thanos, no compactor, no querier. For 6 clusters with ~100k active series total, a single node handles it comfortably.

The dedup.minScrapeInterval option deduplicates samples from multiple Prometheus replicas that write the same series. Set it to match your scrape interval.

VMAgent

VMAgent replaces prometheus as the scraper. It's lighter, can shard, and sends data via remote_write only — it has no query engine:

yamlcopyapiVersion: operator.victoriametrics.com/v1beta1
kind: VMAgent
metadata:
  name: vmagent
  namespace: observability
spec:
  selectAllByDefault: true
  remoteWrite:
    - url: "http://victoria-metrics-victoria-metrics.observability.svc.cluster.local:8428/api/v1/write"
      # Optional: queue config for high-volume writes
      queueConfig:
        capacity: 10000
        maxSamplesPerSend: 2000
  resources:
    requests:
      cpu: 200m
      memory: 256Mi
    limits:
      cpu: 500m
      memory: 512Mi

selectAllByDefault: true means VMAgent automatically discovers all VMServiceScrape and VMPodScrape objects in the cluster.

VMAgent has a built-in WAL (write-ahead log) — if the remote endpoint is down, it buffers data locally and replays when the connection comes back. Default buffer: 1h. Configure with remoteWrite.urlRelabelConfig to filter what gets forwarded.

VMServiceScrape

Drop-in replacement for ServiceMonitor:

yamlcopyapiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
metadata:
  name: apisix
  namespace: ingress-apisix
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: apisix
  endpoints:
    - port: prometheus
      interval: 30s
      path: /apm/prometheus
      # Relabel to add cluster label
      metricRelabelConfigs:
        - sourceLabels: [__name__]
          action: keep
          regex: "apisix_.*"

The VictoriaMetrics Operator converts this to a Prometheus scrape config internally.

VMPodScrape

For pods without a Service:

yamlcopyapiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
  name: my-app
  namespace: app
spec:
  selector:
    matchLabels:
      app: my-app
  podMetricsEndpoints:
    - port: metrics
      interval: 30s

VMRule for alerts

yamlcopyapiVersion: operator.victoriametrics.com/v1beta1
kind: VMRule
metadata:
  name: node-alerts
  namespace: observability
spec:
  groups:
    - name: node
      rules:
        - alert: NodeDiskFull
          expr: |
            (node_filesystem_avail_bytes{mountpoint="/"} /
             node_filesystem_size_bytes{mountpoint="/"}) < 0.1
          for: 10m
          labels:
            severity: warning
          annotations:
            description: "Node {{ $labels.instance }} disk < 10%"

        - record: node:disk_usage:ratio
          expr: |
            1 - (node_filesystem_avail_bytes{mountpoint="/"} /
                 node_filesystem_size_bytes{mountpoint="/"})

VMRule objects are picked up by VMAlert for evaluation and Alertmanager integration.

Grafana datasource

yamlcopydatasources:
  - name: VictoriaMetrics
    type: prometheus
    url: http://victoria-metrics-victoria-metrics.observability.svc.cluster.local:8428
    isDefault: false
    jsonData:
      timeInterval: "30s"

  - name: Local-Prometheus
    type: prometheus
    url: http://kube-prom-stack-prometheus.monitoring.svc.cluster.local:9090
    isDefault: true
    jsonData:
      timeInterval: "30s"

Long-term dashboards (30d, 90d range) query VictoriaMetrics. Real-time dashboards query the local spoke Prometheus. This split keeps spoke Prometheus disks small while providing historical data on demand.

Comparing VictoriaMetrics vs Prometheus

For most setups < 1M active series, VMSingle is simpler and more disk-efficient than a Prometheus + Thanos stack.

Debugging

bashcopy# Check VMAgent status
kubectl describe vmagent vmagent -n observability

# Check remote_write queue
kubectl port-forward -n observability svc/vmagent 8429:8429
curl -s http://localhost:8429/metrics | grep vm_remotewrite_queue

# Query VMSingle directly
kubectl port-forward -n observability svc/victoria-metrics-victoria-metrics 8428:8428
curl -s 'http://localhost:8428/api/v1/query?query=up' | jq '.data.result | length'