Hubble: network flow observability inside Kubernetes

Published: 2026-02-08

Cilium's Hubble is a network observability layer built directly into the eBPF datapath. Every packet flowing through the cluster — pod-to-pod, pod-to-service, pod-to-external — is recorded with source, destination, verdict, and reason. No sidecar proxies, no sampling, no extra hops.

This post covers the full setup, CLI usage for common debugging scenarios, Prometheus metrics, and the operational patterns we use when diagnosing network issues in a six-cluster environment.

Why Hubble instead of tcpdump or network policies logs

Traditional Kubernetes network debugging tools have serious limitations:

• tcpdump requires exec-ing into a pod or node, and you need to know which node the traffic is on before you start. With 6 clusters and 200+ pods per node, that's a needle-in-haystack problem.
• Network policy deny logs don't exist in vanilla Kubernetes — dropped packets are silently discarded with no audit trail.
• Service mesh observability (Istio, Linkerd) requires sidecar injection, which adds latency, memory, and complexity to every pod.

Hubble records all flow events at the kernel level via eBPF. The agent runs on every node as part of the Cilium DaemonSet — no additional pods, no sidecar injection, no performance overhead beyond what Cilium already adds. The data is available cluster-wide through hubble-relay.

Components

hubble-agent — embedded in each Cilium agent pod. Exports flow events to gRPC.

hubble-relay — aggregates events from all nodes, exposes a single gRPC endpoint for cluster-wide visibility.

hubble-ui — a web UI for browsing flows and drawing service dependency maps in real time.

Enable all three in the Cilium Helm values:

yamlcopyhubble:
  enabled: true
  metrics:
    enabled:
      - dns:query;ignoreAAAA
      - drop
      - tcp
      - flow
      - icmp
      - http
  serviceMonitor:
    enabled: true
    labels:
      release: kube-prom-stack
  relay:
    enabled: true
    prometheus:
      enabled: true
      serviceMonitor:
        enabled: true
        labels:
          release: kube-prom-stack
  ui:
    enabled: true

The labels: release: kube-prom-stack on serviceMonitor is how Prometheus discovers the metrics endpoint — your kube-prometheus-stack is deployed with release=kube-prom-stack label selector.

The dns:query;ignoreAAAA metric ignores AAAA (IPv6) queries in the DNS metrics, which reduces cardinality significantly if your cluster is IPv4-only. Otherwise every hostname lookup generates two metrics entries.

hubble CLI

Install hubble locally and port-forward relay:

bashcopykubectl port-forward -n kube-system svc/hubble-relay 4245:80 &

# Last 100 flows cluster-wide
hubble observe --last 100

# Dropped flows in a namespace
hubble observe --namespace app --verdict DROPPED

# Flows from a specific pod
hubble observe --pod app/backend --protocol TCP

# Follow mode (like tail -f)
hubble observe --follow --namespace default

Or run the Hubble CLI directly inside the Cilium pod on any node:

bashcopykubectl exec -n kube-system ds/cilium -- hubble observe --last 50

This avoids the port-forward setup and is useful during incident response when you need answers fast.

The --verdict DROPPED filter is the fastest way to diagnose why traffic isn't reaching a pod. Common causes:

• NetworkPolicy blocking the flow — verdict shows POLICY_DENIED
• No active endpoint — verdict shows UNKNOWN_CONNECTION
• Port mismatch — verdict shows PORT_UNROUTABLE

Output format

The default output is human-readable. For structured processing:

bashcopy# JSON output (for piping to jq)
hubble observe --last 100 --output json | jq '.flow | select(.verdict == "DROPPED")'

# Count drops by source namespace
hubble observe --last 1000 --output json \
  | jq -r 'select(.flow.verdict == "DROPPED") | .flow.source.namespace' \
  | sort | uniq -c | sort -rn

DNS visibility

Hubble captures DNS queries at the pod level:

bashcopyhubble observe --type l7 --protocol DNS
hubble observe --type l7 --protocol DNS --pod app/backend

Output shows which pod queried what hostname and what the response was. This is invaluable when debugging why an app can't reach an external service — you see whether DNS is returning the right IP, whether the query never happens, or whether the query returns NXDOMAIN.

Common DNS issues Hubble reveals:

• App queries the wrong hostname (hardcoded IP or wrong env var)
• DNS resolution succeeds but the IP is unreachable (NetworkPolicy issue, separate from DNS)
• DNS queries are timing out (overloaded CoreDNS)

To check CoreDNS performance:

bashcopy# DNS queries to kube-dns from a specific pod
hubble observe --to-pod kube-system/coredns --protocol DNS --follow

If you see a high rate of retries from the same pod, CoreDNS might be a bottleneck.

NetworkPolicy debugging

Suppose traffic from frontend to backend is dropping silently. Without Hubble you'd add debug logs to the app and redeploy. With Hubble:

bashcopyhubble observe \
  --namespace app \
  --from-pod app/frontend \
  --to-pod app/backend \
  --verdict DROPPED

If the verdict is POLICY_DENIED with reason: INGRESS_DENIED, you know exactly which direction and which policy is blocking it. Then:

bashcopykubectl get ciliumnetworkpolicy -n app
kubectl get networkpolicy -n app

Find the policy and adjust the ingress selector. With classic NetworkPolicy, the fix is usually adding a missing from selector. With CiliumNetworkPolicy, you can also restrict by L7 — HTTP method, path, headers.

Using AUDIT mode for policy rollout

AUDIT mode lets you test a restrictive policy before enforcing it:

yamlcopyapiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: restrict-backend
  namespace: app
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
  policyAuditMode: true   # won't drop, just logs

Set policyAuditMode: true, deploy, watch Hubble for AUDIT verdicts for 24 hours, then remove the flag to enforce. Zero-downtime policy hardening.

bashcopy# Watch audit verdicts for the backend
hubble observe --namespace app --to-pod app/backend --verdict AUDIT --follow

Hubble UI service map

The UI renders a live dependency graph. Each node is a namespace or workload. Edges are colored by drop rate and annotated with request rate.

In our setup we expose it via ApisixRoute behind a group RBAC check — only internal users can access it:

yamlcopyapiVersion: apisix.apache.org/v2
kind: ApisixRoute
metadata:
  name: hubble-ui
  namespace: ingress-apisix
spec:
  http:
    - name: hubble
      match:
        hosts:
          - hubble.infra.example.com
        paths:
          - "/*"
      backends:
        - serviceName: hubble-ui
          serviceNamespace: kube-system
          servicePort: 80

The service map is the first place we look after a new deployment when something is "slow" — often the map shows a new downstream dependency that wasn't rate-limited and is getting hammered. The visual makes it immediately obvious: thick red edges going to a single service mean that service is dropping traffic.

Flow metrics in Prometheus

With serviceMonitor.enabled: true, Hubble relay exposes Prometheus metrics that feed into your existing Grafana:

promqlcopy# Packet drop rate per namespace
rate(hubble_drop_total[5m])

# Drop rate by reason (POLICY_DENIED, NO_TUNNEL_KEY, etc.)
sum by (reason) (rate(hubble_drop_total[5m]))

# HTTP request rate by destination
rate(hubble_http_requests_total[5m])

# HTTP error rate (4xx, 5xx)
rate(hubble_http_requests_total{status=~"[45].."}[5m])

# DNS error rate
rate(hubble_dns_error_total[5m])

# DNS query rate by pod
topk(10, rate(hubble_dns_queries_total[5m]))

We add a dedicated "Network" row to our cluster Grafana dashboard: drop rate per namespace, HTTP error rate by service, DNS query rate. It sits between the "Node" row and the "App" row and often surfaces the root cause of application-level errors before the app team notices them.

Alert on drop spikes

A useful PrometheusRule:

yamlcopy- alert: NetworkDropSpike
  expr: sum(rate(hubble_drop_total[5m])) by (namespace) > 10
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "High drop rate in {{ $labels.namespace }}"
    description: "{{ $value | printf \"%.1f\" }} drops/sec in {{ $labels.namespace }}"

This fires if any namespace has more than 10 drops/second sustained for 2 minutes. Common causes: a misconfigured NetworkPolicy after a deployment, a new service that hasn't been allowed through policy yet.

Cilium Network Policy verdicts

Each Hubble flow has a verdict and up to three policy verdicts:

AUDIT mode is useful when rolling out restrictive policies — set them to audit first, observe for a day, then flip to enforcing. Zero downtime policy hardening.

Retention and performance

Hubble flows are stored in-memory in a ring buffer on each node. The default ring buffer size is 4096 events per node. For high-traffic clusters, increase it:

yamlcopyhubble:
  eventQueueSize: 65536
  eventBufferCapacity: 65535

Flows older than the ring buffer capacity are lost. If you need longer retention, forward flows to an external store via Hubble's gRPC export or use the Prometheus metrics (which have Prometheus retention).

For very high traffic (10k+ flows/second), the per-node CPU overhead of event processing can become visible. Profile with cilium monitor --type drop before enabling all metrics simultaneously — start with drop and dns, add http and tcp only if needed.