node-exporter alerts: clock skew, CPU, memory, and disk thresholds
Published: 2026-02-20
node-exporter is the workhorse of Kubernetes infrastructure monitoring. Every node exposes hundreds of host metrics. The art is picking the right subset to alert on — enough signal, not so much noise that alerts get ignored. This is the set we run in production with rationale for each threshold choice.
Deploying node-exporter
node-exporter ships as part of kube-prometheus-stack. In the HelmRelease values:
yamlvalues:
nodeExporter:
enabled: true
serviceMonitor:
relabelings:
- sourceLabels: [__meta_kubernetes_pod_node_name]
targetLabel: instance
The relabeling sets the instance label to the node name instead of IP:9100. This makes alert descriptions readable: "Node k3s-worker-1 CPU is high" rather than "Node 192.168.1.20:9100 CPU is high".
node-exporter runs as a DaemonSet and needs privileged access to host filesystems. The kube-prometheus-stack chart handles the required tolerations and host mounts automatically.
Clock skew
Clock drift is dangerous: JWT tokens expire at wrong times, TLS certificate validation fails, distributed systems (etcd, Kafka) make wrong ordering decisions. Alert before it becomes a problem:
yaml- alert: NodeClockSkew
expr: >
(node_timex_offset_seconds > 1
and deriv(node_timex_offset_seconds[5m]) >= 0)
or
(node_timex_offset_seconds < -1
and deriv(node_timex_offset_seconds[5m]) <= 0)
for: 15m
labels:
severity: warning
service: ne
event: 'Clock Skew |{{$labels.instance}}'
annotations:
description: >-
Clock skew on {{$labels.instance}} is {{$value}}s and
has been growing for 15 min. Check NTP sync.
The deriv() condition ensures we only alert when the skew is growing in the wrong direction, not when NTP is actively correcting it (which also shows offset > 1 briefly during correction). Without deriv(), the alert would fire and immediately resolve every time NTP does a correction — constant flapping.
Checking NTP status
When this alert fires:
bash# On the affected node
timedatectl status
chronyc tracking
ntpq -p
Common causes:
- NTP service disabled (happens after VM snapshot restore)
- Firewall blocking UDP 123 outbound
- NTP server unreachable from the node's network
Memory usage
yaml- alert: NodeHighMemoryUsage
expr: >
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 > 95
for: 10m
labels:
severity: warning
service: ne
event: 'High Memory |{{$labels.instance}}'
annotations:
description: >-
Node {{$labels.instance}} memory usage is above 95% for 10 min.
Available: {{ with query "node_memory_MemAvailable_bytes{instance=\"{{$labels.instance}}\"}" }}
{{ . | first | value | humanize1024 }}B{{ end }}
The threshold is 95%, not 80%. Linux uses available RAM for page cache; seeing 70-80% used is normal and healthy. MemAvailable_bytes accounts for reclaimable cache. Only the last 5% — where the OOM killer starts making decisions — is worth alerting on.
OOM kill monitoring
A complementary alert for when OOM already happened:
yaml- alert: NodeOOMKill
expr: increase(node_vmstat_oom_kill[5m]) > 0
for: 0s
labels:
severity: high
event: 'OOM Kill |{{$labels.instance}}'
annotations:
description: OOM kill occurred on {{$labels.instance}} in the last 5 minutes.
for: 0s — fire immediately, no for delay needed. An OOM kill already happened; no point waiting to confirm it.
CPU usage
yaml- alert: NodeHighCpuUsage
expr: >
100 - (avg by (instance) (
rate(node_cpu_seconds_total{mode="idle"}[15m])
) * 100) > 95
for: 10m
labels:
severity: warning
service: ne
event: 'High CPU |{{$labels.instance}}'
annotations:
description: Node {{$labels.instance}} CPU usage is above 95% for 10 min.
rate(node_cpu_seconds_total{mode="idle"}[15m]) averaged across all CPUs gives the idle fraction. Subtracting from 100 gives used %. The 15-minute window smooths out short spikes (batch jobs, GC pauses).
CPU steal (for VMs)
On VMs, CPU steal > 10% indicates the hypervisor is throttling:
yaml- alert: NodeHighCpuSteal
expr: >
avg by (instance) (
rate(node_cpu_seconds_total{mode="steal"}[15m])
) * 100 > 10
for: 10m
labels:
severity: warning
event: 'CPU Steal |{{$labels.instance}}'
annotations:
description: >-
CPU steal on {{$labels.instance}} is above 10%.
The hypervisor is overcommitting CPU on this host.
CPU steal > 10% causes unpredictable latency. If you see this consistently, the VM needs to move to a less loaded physical host.
Load average
Load average > number of CPUs × 2:
yaml- alert: NodeHighLoadAverage
expr: >
node_load15 / count without(cpu, mode)(node_cpu_seconds_total{mode="idle"}) > 2
for: 15m
labels:
severity: warning
event: 'High Load |{{$labels.instance}}'
annotations:
description: Load average on {{$labels.instance}} is {{$value}}x the number of CPUs for 15 min.
Disk usage per filesystem
yaml- alert: NodeFilesystemUsageHigh
expr: >
max by (instance, device) (
(
node_filesystem_size_bytes{fstype=~"ext.?|xfs"}
- node_filesystem_free_bytes{fstype=~"ext.?|xfs"}
) * 100
/
(
node_filesystem_avail_bytes{fstype=~"ext.?|xfs"}
+ node_filesystem_size_bytes{fstype=~"ext.?|xfs"}
- node_filesystem_free_bytes{fstype=~"ext.?|xfs"}
)
) > 95
for: 10m
labels:
severity: warning
service: ne
event: 'High Disk |{{$labels.instance}}|{{$labels.device}}'
annotations:
description: Device {{$labels.device}} on {{$labels.instance}} is above 95% used.
fstype=~"ext.?|xfs" excludes tmpfs, overlay, and virtual filesystems that generate constant noise.
Why use avail_bytes in the denominator instead of size_bytes? On ext4, up to 5% is reserved for root. avail_bytes is what non-root processes can actually use. Using size_bytes would underreport usage.
Disk fill rate prediction
Alert when disk will fill in less than 4 hours at current write rate:
yaml- alert: NodeDiskFillingSoon
expr: >
predict_linear(
node_filesystem_avail_bytes{fstype=~"ext.?|xfs"}[1h],
4 * 3600
) < 0
for: 30m
labels:
severity: high
event: 'Disk Filling |{{$labels.instance}}|{{$labels.device}}'
annotations:
description: >-
Disk {{$labels.device}} on {{$labels.instance}} will fill in less than 4h
at the current write rate.
inotify watches
Hit the inotify watch limit on nodes running many .NET or Node.js apps:
yaml- alert: NodeInotifyWatchesHigh
expr: node_filesystem_files_free{mountpoint="/"} < 100000
for: 5m
labels:
severity: warning
event: 'Low Inotify |{{$labels.instance}}'
annotations:
description: Free inotify watches on {{$labels.instance}} below 100k. Increase fs.inotify.max_user_watches.
Fix:
bash# Temporary
sysctl fs.inotify.max_user_watches=1048576
# Permanent
echo "fs.inotify.max_user_watches=1048576" >> /etc/sysctl.d/99-inotify.conf
sysctl -p /etc/sysctl.d/99-inotify.conf
Network errors
yaml- alert: NodeNetworkErrors
expr: >
rate(node_network_receive_errs_total{device!~"lo|veth.*|cilium.*"}[5m]) > 10
or
rate(node_network_transmit_errs_total{device!~"lo|veth.*|cilium.*"}[5m]) > 10
for: 5m
labels:
severity: warning
event: 'Network Errors |{{$labels.instance}}|{{$labels.device}}'
annotations:
description: Network errors on {{$labels.instance}} device {{$labels.device}}.
device!~"lo|veth.*|cilium.*" excludes virtual interfaces that normally show spurious errors in some kernel versions. This filters noise from the CNI layer.
Complete PrometheusRule
yamlapiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: node-exporter-alerts
namespace: observability
labels:
release: kube-prom-stack
spec:
groups:
- name: node-exporter
interval: 60s # 1 min is sufficient for host metrics
rules:
- alert: NodeClockSkew
# ...
- alert: NodeHighMemoryUsage
# ...
- alert: NodeOOMKill
# ...
- alert: NodeHighCpuUsage
# ...
- alert: NodeFilesystemUsageHigh
# ...
- alert: NodeDiskFillingSoon
# ...
- alert: NodeNetworkErrors
# ...
interval: 60s on the node-exporter group — host metrics don't need 30-second evaluation. This cuts Prometheus evaluation load in half compared to a 30s interval across all these rules.