← weblog / 2026-08-13-mail-down-memory-full-wrong-arch
2026-08-13·kubernetesk0sincidentmemorydocker·7 min read

One Evening, Three Problems: Mail Down, Memory Full, Wrong Docker Arch

Published: 2026-08-13

I run everything on one small VM: 4GB RAM, k0s, one node. This evening mail.antonnovikov.com gave 503. I fixed that, then I found the real problem was memory on the whole node, not just mail. Then I broke a small site during redeploy because of wrong CPU architecture. This post is the story, in order, with the commands I really used.

Problem 1: mail.antonnovikov.com — 503

First check, always the same:

bashkubectl get pods -A | grep -v Running

Output:

webmail   snappymail-7fd545b5bf-tvhxx   0/1   CrashLoopBackOff   221 (13s ago)   251d

221 restarts. Not new, this pod is crashing for a long time, but today it finally stayed down long enough for APISIX to return 503 (no healthy backend).

bashkubectl describe pod -n webmail snappymail-7fd545b5bf-tvhxx

Last State:  Terminated
  Reason:    OOMKilled
  Exit Code: 137

OOMKilled. Memory limit was 256Mi, container growing slowly (normal PHP-FPM worker leak) and getting killed roughly once a day for 251 days. Nobody noticed because the pod comes back fast — until today the timing was bad and it stayed crashed a bit too long.

Fix: raise the limit.

yamlresources:
  limits:
    memory: "384Mi"  # was 256Mi
    cpu: "500m"

bashkubectl patch deployment snappymail -n webmail --type='json' \
  -p='[{"op":"replace","path":"/spec/template/spec/containers/0/resources/limits/memory","value":"384Mi"}]'

Mail came back. But this made me check the node, not just the pod.

Problem 2: the node has almost no free memory

bashssh root@$SERVER_IP free -h

              total   used   free   shared  buff/cache  available
Mem:          3.8Gi   3.4Gi  120Mi  27Mi     553Mi       404Mi
Swap:         2.0Gi   801Mi  1.2Gi

120Mi free, 800Mi already in swap. Not good for a 4GB box running full k0s control plane plus ~30 pods (VPN proxies, monitoring stack, mail, a few small sites).

bashkubectl describe node antonnovikov.com | grep -A5 "Allocated resources"

Resource   Requests          Limits
--------   --------          ------
cpu        1105m (18%)       7250m (120%)
memory     1909048192 (47%)  6240Mi (163%)

163% memory limits overcommit. On a single node that's not automatically a problem (nothing to fail over to anyway), but it tells me: if two or three pods grow at the same time, someone gets OOM-killed.

What I actually reduced

I did NOT just raise limits everywhere — on one node that changes nothing real, RAM used is RAM used regardless of the declared limit. I looked for real, safe savings:

VictoriaMetrics (vmsingle) — was using 396Mi.

bashhelm get values vmsingle -n monitoring -a > /tmp/vmsingle-live.yaml
# memory.allowedPercent: 30 -> 20  (cache ceiling, no data loss)
helm upgrade vmsingle vm/victoria-metrics-single -n monitoring -f /tmp/vmsingle-live.yaml

Later I also doubled the global scrape interval, 30s -> 60s. This cut ingestion rate in half — less CPU, less disk write, less memory pressure, and it also matters for problem 3 below.

Grafana dashboard sidecar — a k8s-sidecar container that watches ConfigMaps and reloads dashboards live. Cost: ~90-130Mi, just sitting there watching. I don't edit dashboards often, so:

yamlsidecar:
  dashboards:
    enabled: false  # was true

bashhelm upgrade grafana grafana/grafana -n monitoring -f platform/monitoring/values/grafana.yaml

Dashboards still work. If I change one now, I need kubectl rollout restart deployment/grafana -n monitoring to pick it up — small price for -100Mi.

tor-proxy — rotating Tor HTTP proxy, was running with TOR_INSTANCES=2 (2 tor daemons), 223Mi.

bashkubectl set env deployment/tor-proxy -n tor TOR_INSTANCES=1

Saved ~85Mi. Still works, just fewer parallel exit circuits.

What I did NOT touch

APISIX, even though it's the single biggest consumer (~400Mi). My own install script has a comment from a past incident:

"Under real load (blackbox probes + real traffic), CPU hit the 500m limit and got stuck — both http and https stopped responding."

Limits were raised from 256Mi/500m to 512Mi/1000m after that. I'm not going back there. If I ever want to reduce APISIX, it needs a second node, not a smaller limit on this one.

Alerting stack (alertmanager, vmalert, kube-state-metrics, blackbox-exporter) — I almost suggested cutting this because Slack/PagerDuty connectors weren't set up. Wrong assumption: alerts go to Telegram directly, so this stack is actually doing its job. Lesson: "connector X isn't configured" does not mean "feature isn't used." Check the real receiver config, not just which integrations you personally connected.

Problem 3: high disk read, ~40-70 MB/s, all evening

Grafana showed a flat, sustained disk read graph. My first guess was "something is scanning a lot of data." Wrong guess. I checked per-process I/O first and found nothing that explained 40-70 MB/s:

bash# snapshot /proc/<pid>/io read_bytes twice, 5s apart, compute delta

Nothing stood out. Then I checked swap activity directly:

bashvmstat 2 6

procs -----------memory---------- ---swap-- -----io----
 r  b   swpd   free  buff  cache    si   so    bi    bo
 5  0  758556 119432 9736 672540  7836   74  44346  252

si (swap in) and bi (block read) both high, moving together. That's the answer: this isn't one chatty process, it's swap thrashing. With ~120Mi free RAM, the kernel is constantly paging processes' memory in and out of swap, and every page-in is a real disk read. The graph is flat because the memory pressure is constant, not a one-time spike.

So problem 3 and problem 2 are the same problem. Freeing real RAM (vmsingle, grafana sidecar, tor-proxy) is the actual fix for the disk graph too, not some separate disk tuning.

Problem 4: redeployed a site, broke it with wrong CPU architecture

While doing all this I also redeployed a small static site (mne-pohui.rf) with a design tweak. Simple docker build && push && kubectl apply, done it a hundred times. This time it went wrong:

bashkubectl get pods -n default -l app=mne-pohui-site

mne-pohui-site-587fc87586-gmp62   0/1   CrashLoopBackOff

bashkubectl logs -n default -l app=mne-pohui-site

exec /docker-entrypoint.sh: exec format error

exec format error almost always means: wrong CPU architecture. My laptop is Apple Silicon (arm64), the k0s node is amd64. The deploy script normally uses docker buildx build --platform linux/amd64, but docker buildx was not installed on this machine at all — it silently fell back to plain docker build, which built for arm64 (my laptop's native arch), pushed that, and the amd64 node couldn't run it.

Also found on the way: no insecure-registries configured for the local Docker engine (I use colima, not Docker Desktop), so pushing to the internal 91.184.248.13:30500 registry failed with an HTTPS error before I even got to the arch problem.

Fixed both:

bash# 1. install the missing buildx plugin
brew install docker-buildx
ln -sf /opt/homebrew/opt/docker-buildx/bin/docker-buildx ~/.docker/cli-plugins/docker-buildx

yaml# 2. ~/.colima/default/colima.yaml
docker:
  insecure-registries:
    - 91.184.248.13:30500

bashcolima stop && colima start

Rebuilt for real with --platform linux/amd64, pushed, kubectl rollout restart. Old pod kept serving the whole time (Kubernetes doesn't remove the old replica until the new one is Ready), so this never caused real downtime for visitors — just a stuck rollout that I had to notice and fix.

What can go wrong (if you copy this)

  • Raising a container's memory limit doesn't fix a leak, it delays the crash. If snappymail OOMs again at 384Mi, the leak is still there — either restart PHP-FPM workers periodically or find the actual leak.
  • Don't cut resources on a component that has a documented incident about being cut before. Read your own past comments in scripts before "optimizing." I almost touched APISIX before I re-read my own install script.
  • A memory graph and a disk graph can be the same root cause. Check swap (vmstat, si/so) before chasing a specific process for disk I/O — on a memory-starved box it's almost always swap.
  • exec format error = architecture mismatch, not a broken image. Check docker buildx build --platform is actually being used, not silently skipped because buildx isn't installed.
  • Rolling update protects you more than you think. A broken new pod under a Deployment doesn't take the site down by itself — old replica stays until new one is Ready. Still fix it, but it's not an emergency by itself.

Summary

  • Mail 503 → snappymail OOMKilled, raised limit 256Mi → 384Mi
  • Real cause was node-wide memory pressure (163% limits overcommit, 120Mi free, 800Mi swap)
  • Freed real RAM: vmsingle cache ceiling 30%→20% + scrape interval 30s→60s, grafana dashboard-sidecar off (~100Mi), tor-proxy 2→1 instance (~85Mi)
  • Left APISIX alone — past incident already proved cutting its limits breaks the site under load
  • Alerting stack stays — it's wired to Telegram, "no Slack/PagerDuty connector" was a wrong signal that it's unused
  • High disk I/O was swap thrashing from memory pressure, not a specific process — same root cause as everything else
  • Separately: broke and fixed a redeploy because of arm64/amd64 mismatch — buildx wasn't installed, docker build silently built for the wrong architecture