Promtail: shipping Traefik access logs to VictoriaLogs

Published: 2026-06-12

VictoriaLogs is running and waiting for input — now something has to read Traefik's access logs off the node and push them in. That something is Promtail: a DaemonSet, one ConfigMap, and a pipeline that turns Traefik's JSON access log into labeled, queryable streams. This post also covers the detour I took through Kubernetes service discovery before deleting it in favor of a one-line glob.

Step 0: make Traefik write access logs

By default Traefik logs almost nothing. Access logging is enabled in the Helm values (in k0s this lives in the chart extension), and the format matters:

yamlcopylogs:
  general:
    level: INFO
  access:
    enabled: true
    format: json

format: json is the whole trick. The default CLF format would need regex parsing; JSON gives named fields for free: ClientHost, RequestMethod, RequestPath, DownstreamStatus, Duration, RouterName, ServiceName. The log goes to stdout, the kubelet writes it under /var/log/pods/, and anything that can tail a file can pick it up.

The Promtail DaemonSet

Promtail mounts the node's log directory read-only and runs everywhere (one node here, but a DaemonSet costs nothing and survives the day this becomes two nodes):

yamlcopyapiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail
  namespace: logging
spec:
  selector:
    matchLabels:
      app: promtail
  template:
    metadata:
      labels:
        app: promtail
    spec:
      serviceAccountName: promtail
      tolerations:
        - operator: Exists
      containers:
        - name: promtail
          image: grafana/promtail:3.0.0
          args:
            - -config.file=/etc/promtail/promtail.yaml
          resources:
            requests:
              cpu: 10m
              memory: 32Mi
            limits:
              cpu: 100m
              memory: 64Mi
          volumeMounts:
            - name: config
              mountPath: /etc/promtail
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: config
          configMap:
            name: promtail-config
        - name: varlog
          hostPath:
            path: /var/log

64 MiB limit. Promtail tailing one log stream uses a fraction of that.

The config: client and positions

yamlcopyserver:
  http_listen_port: 9080
  grpc_listen_port: 0
  log_level: warn

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://victoria-logs.logging.svc.cluster.local:9428/insert/loki/api/v1/push

The client URL is the Loki-compatible endpoint VictoriaLogs exposes — Promtail has no idea it isn't talking to Loki.

positions.filename: /tmp/positions.yaml is a deliberate shortcut. Positions track how far each file has been read; in /tmp they die with the pod, so a Promtail restart re-reads whatever log files still exist and ships duplicates. For access logs that's acceptable — a duplicated line in a dashboard is not an incident. If it ever bothers me, the fix is a hostPath mount for the positions file, not a PVC.

Scrape config: the version that works

yamlcopyscrape_configs:
  - job_name: traefik-access
    static_configs:
      - targets: [localhost]
        labels:
          job: traefik-access
          namespace: traefik
          __path__: /var/log/pods/traefik_traefik-*/*/*.log
    pipeline_stages:
      - docker: {}
      - json:
          expressions:
            ClientHost: ClientHost
            RequestMethod: RequestMethod
            RequestPath: RequestPath
            DownstreamStatus: DownstreamStatus
            Duration: Duration
            RouterName: RouterName
            ServiceName: ServiceName
            level: level
      - labels:
          ClientHost:
          RequestMethod:
          DownstreamStatus:
      - drop:
          expression: '.*middleware.*does not exist.*'

The kubelet's directory layout is /var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container>/*.log, so the glob traefik_traefik-* matches the Traefik pod in the traefik namespace regardless of its current UID or restart count. One line of path config replaces an entire discovery mechanism.

The pipeline:

1. docker: {} unwraps the container runtime envelope so the inner line is the raw Traefik JSON.
2. json: extracts the interesting fields from it.
3. labels: promotes three of them to stream labels. DownstreamStatus and RequestMethod are what dashboards filter on. Promoting ClientHost (a per-visitor IP) would be a cardinality crime against Loki — VictoriaLogs flattens high-cardinality labels into regular fields instead of exploding, which is the only reason this is safe here. Notably not promoted: RequestPath. It stays in the log body and gets filtered with LogsQL when needed.
4. drop: discards a config-noise error Traefik emits for every request to a route whose middleware reference is stale — one misconfigured redirect produced thousands of identical lines a day. Dropping at the agent keeps them out of storage entirely.

The version that didn't survive: kubernetes_sd

The first iteration discovered the Traefik pod through the Kubernetes API, the way every Promtail tutorial does it:

yamlcopykubernetes_sd_configs:
  - role: pod
    namespaces:
      names: [traefik]
relabel_configs:
  - source_labels: [__meta_kubernetes_pod_container_name]
    regex: traefik
    action: keep
  - source_labels: [__meta_kubernetes_pod_uid, __meta_kubernetes_pod_container_name]
    separator: /
    replacement: /var/log/pods/*$1*/$2/*.log
    target_label: __path__

It worked, and it's also the reason the ServiceAccount has a ClusterRole over pods, nodes, services, and endpoints. But for exactly one pod with a predictable name, it's a watch connection to the API server, four relabel rules, and a discovery cache — all to compute a path I can write by hand. When I rewrote the job as a static glob, nothing got worse and forty lines of config disappeared. Service discovery earns its keep when targets come and go; a single ingress controller doesn't.

The RBAC stays, though — the next scrape job (app logs by namespace) will want real discovery.

A note on Promtail's lifespan

Grafana has deprecated Promtail in favor of Alloy; promtail 3.x is in maintenance and EOL is announced for early 2026 in the LTS sense. I picked it anyway: the config above is twenty lines and battle-understood, Alloy can consume the same Loki endpoint when the migration happens, and on the receiving side VictoriaLogs doesn't care who's pushing. Migrating this job to Alloy is an hour of work the day it becomes necessary.

What can go wrong

Empty result in VictoriaLogs, Promtail logs say "no such file". The glob doesn't match the actual pod directory. Check what's really there: ls /var/log/pods/ | grep traefik. A pod in a different namespace, or a Helm release named something other than traefik, changes the <namespace>_<pod-name> prefix.

Lines arrive but fields don't parse. The runtime envelope and the parser disagree. Docker writes JSON envelopes ({"log": "...", "stream": ...}), containerd/CRI writes a text prefix (2026-06-12T10:00:00.000Z stdout F {...}). Promtail's docker: stage handles the former, the cri: stage the latter — if json: extracts nothing, look at a raw line from the file and check which envelope you actually have.

Duplicate log lines after a Promtail restart. That's the /tmp positions file (see above) — expected here. If you see duplicates without restarts, two Promtail pods are tailing the same path; check for a stray second DaemonSet pod on the node.

The drop stage eats too much. drop.expression matches against the whole line at that point in the pipeline. A too-broad regex silently discards real traffic — after editing it, confirm volume with a count query in VMUI: {job="traefik-access"} _time:15m | stats count().

Summary

• Traefik writes JSON access logs to stdout; the kubelet's /var/log/pods/<ns>_<pod>-*/ layout makes them tailable with one glob
• Promtail pushes to VictoriaLogs through the Loki protocol — no plugins, no awareness it's not Loki
• kubernetes_sd_configs was the first version; for a single well-known pod, a static __path__ glob does the same with forty fewer lines
• Promote only low-cardinality fields to labels; RequestPath stays in the body, and ClientHost is only safe because VictoriaLogs tolerates high-cardinality labels
• Drop known-noise lines at the agent, not at query time
• Positions in /tmp trade duplicate-on-restart for zero storage config — fine for access logs