Blackbox exporter and Prometheus Probes for endpoint monitoring

Published: 2026-05-06

Prometheus scrapes metrics from running pods. But it doesn't know if external URLs are reachable, if TLS certificates are about to expire, or if an HTTP endpoint returns 200. That's the job of the blackbox exporter combined with Probe objects from the Prometheus Operator.

Components

• prometheus-blackbox-exporter — runs HTTP/HTTPS/TCP checks on demand and exposes the results as Prometheus metrics.
• Probe CRD — tells the Prometheus Operator to scrape a list of URLs via the blackbox exporter on a schedule.

Both are deployed per-cluster. The blackbox exporter lives in base/monitoring/exporters/blackbox-exporter/. Probe objects are in spoke/monitoring/.

Blackbox exporter HelmRelease

yamlcopyapiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: blackbox-exporter
  namespace: observability
spec:
  chart:
    spec:
      chart: prometheus-blackbox-exporter
      version: ">=8.0.0"
      sourceRef:
        kind: HelmRepository
        name: prometheus-community
        namespace: flux-system
  interval: 1h
  values:
    config:
      modules:
        http_2xx:
          prober: http
          timeout: 10s
          http:
            valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
            valid_status_codes: [200, 301, 302]
            follow_redirects: true
            tls_config:
              insecure_skip_verify: false
        http_2xx_insecure:
          prober: http
          timeout: 10s
          http:
            valid_status_codes: [200]
            tls_config:
              insecure_skip_verify: true
        tcp_connect:
          prober: tcp
          timeout: 5s
        icmp:
          prober: icmp
          timeout: 5s
    serviceMonitor:
      enabled: true
      labels:
        release: kube-prom-stack  # must match Prometheus selector

Two HTTP modules: http_2xx for public-cert endpoints and http_2xx_insecure for internal services with self-signed certs.

Probe objects

Probe objects are in the spoke/monitoring/ layer — a separate Kustomization (dev-monitoring-spoke) that runs after dev-spoke because Probe CRDs come from kube-prom-stack.

yamlcopyapiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
  name: external-endpoints
  namespace: observability
spec:
  interval: 60s
  module: http_2xx
  prober:
    url: blackbox-exporter.observability.svc.cluster.local:9115
  targets:
    staticConfig:
      static:
        - https://elasticsearch.dev.test.example.com
        - https://prometheus.dev.test.example.com
        - https://headlamp.dev.test.example.com
        - https://links.dev.test.example.com
      relabelingConfigs:
        - sourceLabels: [__address__]
          targetLabel: instance

yamlcopyapiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
  name: internal-endpoints
  namespace: observability
spec:
  interval: 30s
  module: tcp_connect
  prober:
    url: blackbox-exporter.observability.svc.cluster.local:9115
  targets:
    staticConfig:
      static:
        - elasticsearch-master.elasticsearch.svc.cluster.local:9200
        - prometheus-operated.observability.svc.cluster.local:9090

Key metrics

The blackbox exporter exposes:

Query examples:

promqlcopy# Endpoints that are currently down
probe_success == 0

# Days until TLS cert expires
(probe_ssl_earliest_cert_expiry - time()) / 86400

# Average probe duration per endpoint
avg_over_time(probe_duration_seconds[1h])

Alerting rules

yamlcopyapiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: blackbox-alerts
  namespace: observability
  labels:
    release: kube-prom-stack
spec:
  groups:
    - name: blackbox
      rules:
        - alert: EndpointDown
          expr: probe_success == 0
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "Endpoint {{ $labels.instance }} is down"
            description: "Probe has been failing for 2 minutes"

        - alert: TLSCertExpiringSoon
          expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14
          for: 1h
          labels:
            severity: warning
          annotations:
            summary: "TLS cert for {{ $labels.instance }} expires in < 14 days"
            description: "Days remaining: {{ $value | humanizeDuration }}"

        - alert: SlowEndpoint
          expr: probe_duration_seconds > 5
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Endpoint {{ $labels.instance }} responding slowly"

EndpointDown fires after 2 minutes of failed probes to avoid noise from transient failures. TLSCertExpiringSoon gives 14 days of warning before certificate expiry.

Why a separate Kustomization for monitoring

Probe CRDs are installed by kube-prom-stack. If dev-monitoring-spoke ran at the same time as dev-spoke, the Probe objects might apply before the CRD exists, causing a reconcile error.

dependsOn: [dev-spoke] ensures the monitoring layer waits for the spoke layer (including kube-prom-stack installation) before applying Probe and PrometheusRule objects.

yamlcopy# dev-monitoring-spoke Kustomization
spec:
  interval: 10m
  path: ./fluxcd/projects/dev/kustomization/spoke/monitoring/
  dependsOn:
    - name: dev-spoke
      namespace: dev

Debugging probe failures

Manually hit the blackbox exporter to see exactly what it's reporting:

bashcopy# Check if a URL passes the http_2xx module
kubectl exec -n observability deploy/blackbox-exporter -- \
  wget -qO- "http://localhost:9115/probe?module=http_2xx&target=https://example.com"

# Or from outside the cluster via port-forward
kubectl port-forward -n observability svc/blackbox-exporter 9115:9115
curl "http://localhost:9115/probe?module=http_2xx&target=https://example.com"

The output is raw Prometheus text format — probe_success 1 means the check passed, probe_success 0 means it failed, and the other metrics explain why.

Summary

• Blackbox exporter + Probe CRDs replace custom ping scripts with proper Prometheus-native endpoint monitoring
• Two modules: http_2xx for public TLS endpoints, http_2xx_insecure for internal services with self-signed certs
• probe_ssl_earliest_cert_expiry gives automatic TLS expiry alerting — no manual tracking needed
• Probe objects go in a separate spoke/monitoring/ Kustomization with dependsOn: [dev-spoke] to wait for kube-prom-stack CRDs
• Debug probe failures by hitting http://localhost:9115/probe?module=http_2xx&target=<url> directly on the blackbox exporter