weblog.antonnovikov.com
Notes from production
Technical writing about DevOps, Kubernetes, infrastructure and platform engineering: what broke, why, and the commands that fixed it.
Tags, archive, series1 active
Tags
Archive
More
Posts 106
-
Eight years in DevOps: an engineer's path from IT support to Kubernetes
A DevOps career path: 8 years from IT support to a 25+ cluster AWS/EKS reference architecture — Sirena GDS, a greenfield startup, FluxCD at MixVel, MTS fintech, Red Rose, now Finstar.
-
One Evening, Three Problems: Mail Down, Memory Full, Wrong Docker Arch
Incident log from a 4GB single-node k0s box: snappymail OOMKilled loop, 163% memory overcommit, swap thrashing disguised as disk I/O, and a docker buildx arch mismatch.
-
Traefik → APISIX on a single node: what broke during the ingress migration
A direct cutover from Traefik to APISIX on bare-metal k0s: hostNetwork port conflicts, a broken --set with commas, an addPrefix bug causing 404s on assets, and eight more findings.
-
Bare-metal k8s: keep Cilium L2 LoadBalancer & node maintenance reliable
Make Cilium L2 LoadBalancer IPs stable on bare-metal Kubernetes, tune node networking, and keep upgrades and reboots from breaking reachability.
-
Lost the only SSH key on a Yandex Cloud fleet: getting back in
Why detaching a boot disk fails, the snapshot-based rescue technique that works instead, and scaling it to 80 hosts with Ansible without losing quorum.
-
Kubespray: a Dedicated Control Plane and Two Workers to Offload a Saturated Node
Splitting an overloaded single Kubernetes node into a dedicated control plane plus two workers with Kubespray v2.31: inventory, Cilium, and workload isolation.
-
k3s on Two VMs: Choosing a Load Balancer and Storage for a Test Cluster
Installing k3s server+agent on two VMs, comparing MetalLB vs Cilium L2 for LB, and choosing a StorageClass: local-path, NFS Provisioner, or Longhorn — with pitfalls explained for each.
-
Wildcard TLS with ACME and cert-manager: Automating Let's Encrypt via Yandex Cloud DNS
How ACME and DNS-01 work, why wildcard certs require it, and how to automate Let's Encrypt in Kubernetes with cert-manager and the Yandex Cloud DNS webhook in a FluxCD GitOps setup.
-
CoreDNS as Authoritative DNS in Kubernetes: Corefile and Zones
How CoreDNS becomes authoritative for cluster.local, how to add custom zones via the file plugin or stub forwarding, and what to do when DNS breaks.
-
Full observability on one VPS: metrics, logs, alerts, dashboards — 452 MiB requested
How VictoriaMetrics, VictoriaLogs, Promtail, vmalert, and Grafana fit together on a 2-core VPS: architecture, the full resource bill, and repeating patterns.
-
Promtail: shipping Traefik access logs to VictoriaLogs
Traefik JSON access logs into VictoriaLogs with Promtail: a static path glob instead of kubernetes_sd, label cardinality choices, and pipeline drop stages.
-
OpenTelemetry as APM replacement: one SDK, any backend
How OpenTelemetry replaces vendor APM tools (Elastic, Datadog, Dynatrace) with one SDK, one protocol, and a Grafana stack for zero licensing cost.
-
VictoriaLogs on k0s: a log database that fits in 128 MiB
Single-pod VictoriaLogs: 90 lines of YAML, Recreate strategy for hostPath storage, Loki-compatible ingestion, LogsQL basics, and Grafana datasource pitfalls.
-
vmsingle: one VictoriaMetrics binary instead of the whole Prometheus stack
VictoriaMetrics single-node on k0s: embedded scraper with static targets, cAdvisor RBAC, hostPath PV, and a stable ClusterIP to survive CoreDNS restarts.
-
Technical SEO on a FastAPI blog: what this site actually does
A complete walkthrough of the SEO stack on weblog.antonnovikov.com: structured data, hreflang, IndexNow, sitemaps, and Atom feeds.
-
Shadowsocks proxy in Kubernetes
Running shadowsocks-libev in a Kubernetes pod as a NodePort service. Cipher selection, UDP relay, Secret management, and the setup script.
-
How this site is built
FastAPI, k0s, Traefik, WireGuard, and plain Markdown — a tour of the infrastructure behind this personal site.
-
Rotating Tor HTTP proxy in Kubernetes
Running zhaow-de/rotating-tor-http-proxy in k0s: N Privoxy+Tor circuits, HAProxy round-robin, stats UI on NodePort 30444.
-
NuGet library CI: versioning from branch name and dual-registry publish
dotnet pack with version from branch name, push to GitLab Package Registry via CI_JOB_TOKEN, selective publish to Artifactory for stable releases with manual gate and user restriction.
-
SonarQube in GitLab CI: dotnet-sonarscanner and coverage delta
GIT_DEPTH:0 for blame, custom SDK image per service via rules variables override, dotnet-coverage collection, coverage delta check against last successful pipeline via GitLab API.
-
VPN status monitoring: WireGuard + IKEv2 in real time
How a 30-second systemd timer, wg show dump, and a plain JSON file power the live VPN status page.
-
GitLab CI multi-environment pipeline: workflow rules and deploy gates
workflow.rules to suppress duplicate pipelines, stages for dev/test/demo/preprod/prod, per-environment deploy jobs extending a shared .deploy anchor, manual prod gate with user restrictions, Grafana deploy annotations.
-
Helm chart config pattern: Files.Get, tpl, and the config/ directory
Storing appsettings.json as a consul-template HCL source in config/, loading rendered output into ConfigMaps with Files.Get + tpl, subPath volumeMounts, and checksum annotations for forced rolling restarts.
-
Beszel: lightweight server monitoring with Telegram alerts
A Go binary that fits in 50 MiB and sends Telegram alerts when the server goes down. Why not Grafana.
-
consul-template in the GitLab CI deploy job
Using consul-template -once to render Vault secrets into Helm chart config/ before helm upgrade. HCL template patterns for per-service and shared secrets, CA cert rendering, and why this beats ESO for CI-time secret injection.
-
Vault JWT auth from GitLab CI: no static tokens
Configuring Vault JWT auth with GitLab JWKS, per-environment roles with bound_claims on ref and project_path, short-lived Vault tokens in CI jobs, and kubeconfig fetched from Vault at deploy time.
-
Security defaults in a FastAPI app: headers, auth, and input validation
CSP, HSTS, HTTP Basic Auth with timing-safe comparison, and path traversal prevention — what's in the app and what's missing.
-
Debugging Flux reconciliation: a field guide
Where to look when Flux doesn't reconcile: get all, describe, force reconcile, suspend/resume, common error patterns for Kustomization and HelmRelease.
-
Self-hosted email: Postfix, Dovecot, CoreDNS, DKIM, SPF, and DMARC
Running @antonnovikov.com email on the same VPS. Getting deliverability right with strict SPF, DKIM signing, and p=reject DMARC.
-
abot: Alertmanager proxy for Telegram and Mattermost notifications
A custom Go service between Alertmanager and notification channels: fan-out, retry, HTTP proxy support, per-cluster routing.
-
Multi-cluster Prometheus: remote_write to central VictoriaMetrics
Spoke Prometheus instances remote_write to a central VictoriaMetrics on the infra cluster. externalLabels.cluster, writeRelabelConfigs for traffic reduction, cross-cluster dashboards.
-
IKEv2 and WireGuard running in separate Kubernetes namespaces
Why VPN pods run inside k0s with hostNetwork and privileged mode, and the tradeoffs that come with it.
-
Blackbox exporter and Prometheus Probes for endpoint monitoring
Using prometheus-blackbox-exporter and Probe CRDs to monitor HTTP endpoints, TCP ports, and TLS certificate expiry across clusters.
-
Three proxies on one server: HTTP, SOCKS5, and Shadowsocks
tinyproxy, microsocks, and ss-server behind the same VPS. How the health endpoints work and why three protocols instead of one.
-
Ansible for multi-cluster k3s management
Inventory structure, node preparation role, k3s install, Cilium bootstrap before Flux, kubeconfig extraction with host substitution. Idempotent playbooks for OS-level changes.
-
Adding a new environment to a FluxCD hub-and-spoke cluster
Full checklist: provision cluster, seal kubeconfig, create Kustomize overlays, patches, routes, and register in the Flux root kustomization.
-
Deploying to k0s with a shell script instead of CI/CD
One shell script, a self-hosted Docker registry, and envsubst. No GitHub Actions, no ArgoCD — here's why that's enough.
-
Yandex Cloud Managed Kubernetes: specifics vs on-prem k3s
What YC manages for you, storage classes, node groups, CCM service accounts, no Cilium L2 needed, kubeconfig via YC CLI, and Flux patch differences.
-
Terraform Operator: running Terraform apply from inside Kubernetes
GalleyBytes terraform-operator runs Terraform as a CRD workload. State on a PVC, SSH key reused from Flux, outputs written to object status.
-
Self-hosted Docker registry inside the k0s cluster
registry:2 on NodePort 30500, no external dependencies for deploys, and automatic cleanup of old image tags.
-
LimitRange, ResourceQuota, and VPA: controlling resource usage
LimitRange injects default requests/limits. ResourceQuota caps namespace totals and blocks cloud LoadBalancers in dev. VPA in Off mode gives right-sizing recommendations.
-
Proxy mesh in Kubernetes: Tor exit nodes and geo-routing with hola-proxy
Running Tor rotating proxies and country-specific hola-proxy instances in a Kubernetes namespace, with TCP health probes and internal routing.
-
Kustomize configMapGenerator and secretGenerator
Generating ConfigMaps and Secrets from files with content-hash rolling updates, disableNameSuffixHash for Helm, env file sources, and HelmRelease valuesFrom.
-
Automatic TLS with cert-manager, Let's Encrypt, and Traefik IngressRoutes
ClusterIssuers, HTTP-01 challenges, multi-SAN certificates, and addPrefix middleware for subdomain routing.
-
OpenTelemetry bridge: APISIX → otel-collector → Elasticsearch APM
APISIX sends OTLP/HTTP traces; APM Server wants gRPC. An otel-collector deployment bridges the two. fullnameOverride is critical.
-
Flux Image Update Automation: auto-bump image tags in git
ImageRepository scans the registry, ImagePolicy selects the right tag (semver, regex), ImageUpdateAutomation commits the updated tag back to git via marker annotations.
-
Helm charts declared in k0s config: Flannel, Traefik, cert-manager, Prometheus
k0s has a built-in Helm controller. Declare charts in k0s.yaml and they install automatically on startup — no FluxCD required.
-
Kubernetes RBAC patterns: per-namespace roles in GitOps
Aggregated ClusterRoles for developer and devops, RoleBindings scoped per namespace managed by Flux, ClusterRoleBinding for monitoring read-only access.
-
Trivy + SonarQube: security scanning in a GitLab CI pipeline
Filesystem and live Kubernetes cluster scans with Trivy. SARIF for the repo scan, JSON-to-external-issues conversion for k8s findings, imported into SonarQube.
-
Setting up a Debian VPS for k0s: sysctl, kernel modules, and the install
What 01-prepare-vm.sh actually does: disabling swap, tuning inotify limits, loading kernel modules, and installing k0s single-node.
-
GitLab CI for an infra repo: lint, validate, and flux reconcile
Pipeline stages for a Kubernetes infra repository: yamllint, kubectl kustomize per environment, flux reconcile on merge, Telegram/Mattermost notifications.
-
cert-manager: wildcard certificates with DNS-01 challenge
ClusterIssuer with ACME DNS-01 via Yandex Cloud DNS webhook. Certificate CR for *.dev.test.antonnovikov.com, sealed distribution to spoke clusters.
-
Cilium L2 LoadBalancer on bare-metal k3s
No cloud controller, no MetalLB — Cilium's built-in L2 announcements assign LoadBalancer IPs on bare-metal k3s using ARP.
-
Cilium NetworkPolicy: default-deny and DNS-aware rules
Default-deny ingress per namespace, allow from APISIX, CiliumNetworkPolicy toFQDNs for external APIs, L7 HTTP path rules, cluster-wide Prometheus scrape policy.
-
Helm chart testing: lint, unit tests, and ct in CI
helm lint, helm-unittest for template assertions, chart-testing (ct) for changed-chart detection, CI/CD stub values files, and what to actually test in your Helm charts.
-
GitLab CI DinD: Docker builds with BuildKit caching
Docker-in-Docker setup with TLS, registry layer caching with mode=max, registry mirror to bypass DockerHub rate limits, multi-arch builds with QEMU, and Trivy scan in the same pipeline.
-
env-view: a custom Helm chart for per-environment service dashboards
A static nginx page rendered from Helm values — every cluster gets a links page listing all service URLs, managed by FluxCD.
-
CloudNativePG: production PostgreSQL in Kubernetes
CNPG operator: streaming replication with automatic failover, WAL archival to S3, scheduled base backups, point-in-time recovery, PgBouncer connection pooling, and monitoring with PrometheusRule.
-
Elasticsearch Index Lifecycle Management
ILM policy hot→delete for vector-* indices: rollover at 20GB or 7 days, delete after 30 days. Index template, write alias bootstrap, idempotent bootstrap script.
-
Loki and LogQL: logs without Elasticsearch overhead
Grafana Loki in SingleBinary mode, Grafana Alloy as promtail replacement, LogQL stream selectors, JSON parsing, rate queries, and trace ID correlation between logs and Jaeger.
-
Alertmanager routing: Telegram notifications with inhibition rules
Alertmanager route tree with severity-based routing, Telegram bot receiver, inhibition rules to suppress cascading alerts, grouping configuration, silence API, and amtool for testing routing decisions.
-
External Secrets Operator + HashiCorp Vault: Kubernetes auth flow
ESO authenticates to Vault using a Kubernetes ServiceAccount token. ClusterSecretStore, ExternalSecret, and the bootstrap steps for Vault Kubernetes auth.
-
kube-prometheus-stack: the full monitoring setup
kube-prometheus-stack HelmRelease with Prometheus persistence and retention tuning, Grafana admin secret, ServiceMonitor for custom apps, sidecar dashboard loading, and storage sizing estimates.
-
ELK in Kubernetes: Elasticsearch, APM Server, and Kibana
ELK stack via Helm in FluxCD, Elastic APM agent in .NET with zero-code auto-instrumentation, index lifecycle management, and correlating logs with APM service maps in Kibana.
-
Grafana datasource and dashboard provisioning via ConfigMap
additionalDataSources in kube-prom-stack values, sidecar dashboard discovery from ConfigMaps with grafana_dashboard label, dashboards in git, folder structure.
-
Vector: log pipeline from Kubernetes to Elasticsearch
Vector DaemonSet, VRL transforms for JSON parsing and health-check filtering, sensitive field redaction, ServiceMonitor for pipeline metrics, and dual sink to Elasticsearch plus S3 archive.
-
Elasticsearch in Kubernetes: HelmRelease, ingest pipelines, and bootstrap
Deploying Elasticsearch via Flux from a git-bundled chart, then bootstrapping index templates, ingest pipelines, and replica settings with a shell script.
-
OpenTelemetry in .NET: traces, metrics, and logs with minimal boilerplate
OTel SDK setup in Program.cs, Prometheus exporter for /metrics, custom ActivitySource spans, custom Meter counters and histograms, OTLP exporter, and OTel Collector HelmRelease.
-
Helm values-per-environment: the -f layering pattern
Base values.yaml with production defaults, minimal per-env override files, FluxCD HelmRelease inline values, dry-run template verification, and why to avoid --set in production pipelines.
-
Prometheus recording rules: pre-aggregating expensive queries
PrometheusRule with record: rules for CPU/memory/HTTP rate. Naming convention level:metric:operations. SLO burn rate multi-window pre-computation.
-
Self-hosted Docker registry in Kubernetes: setup, auth, and cleanup
registry:2 on a NodePort, PVC storage, insecure-registries config for Docker hosts, CI runner push workflow, imagePullSecret, and weekly garbage collection CronJob for deleted blobs.
-
VictoriaLogs + Vector: replacing EFK with a lightweight log stack
Vector DaemonSet routes app logs to Elasticsearch and error-level logs to VictoriaLogs. VRL transforms, deduplication, and the VictoriaLogs Elasticsearch-compatible sink.
-
Multi-stage Dockerfile for .NET: build once, run lean
Three-stage pattern with restore/build/runtime layers, cache optimization by separating project files from source code, non-root user, readOnlyRootFilesystem, BuildKit registry caching in GitLab CI.
-
GitLab CI Docker deploy pipeline: build, push, and rolling update
Full GitLab CI pipeline for containerized apps: Docker build with BuildKit layer caching, push to private registry, kubectl rollout with SHA-pinned image tags, multi-environment promotion, and registry cleanup job.
-
VictoriaMetrics Operator: VMAgent, VMSingle, VMServiceScrape
VMSingle for 6-month retention, VMAgent as scraper with remote_write only, VMServiceScrape as drop-in for ServiceMonitor, VMRule, Grafana datasource setup.
-
DefectDojo in Kubernetes: security findings as a first-class CI gate
DefectDojo HelmRelease, Trivy and SonarQube report upload via API, product and engagement model, SLA enforcement in CI that fails pipelines on unacknowledged high-severity findings.
-
Alertmanager routing to Telegram and Mattermost
Custom Go templates for alert messages, severity-mapped emoji, cluster labels in group_by, and repeat intervals that don't wake you up every 30 minutes.
-
APISIX as Kubernetes ingress: rate limiting, JWT routing, and Lua plugins
APISIX Ingress Controller with ApisixRoute, limit-count for rate limiting per consumer, JWT claim-based routing for A/B testing, serverless Lua plugin for request transformation, canary releases.
-
Traefik Middleware for security headers: HSTS, CSP, and frame protection
Traefik Middleware resources for HSTS, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, and Content-Security-Policy. IngressRoute wiring and testing headers with curl.
-
Flux Notification Controller: commit status, Telegram, Mattermost
Provider and Alert CRDs: GitLab commit status on every reconcile, generic webhook to abot for Telegram fan-out, per-namespace Alert objects for environment-scoped noise.
-
Cilium network policy: default-deny and workload isolation
Default-deny CiliumNetworkPolicy per namespace, selective allow rules for DNS, ingress controller, inter-service traffic, FQDN-based egress policies, and L7 HTTP policy for path-level control.
-
kube-prometheus-stack on multi-cluster: spoke Prometheus, central Grafana
Grafana on one cluster, Prometheus on all six. Spoke kube-prom-stack with Grafana disabled, externalLabels.cluster for cross-cluster dashboards.
-
Trivy Operator in Kubernetes: automated vulnerability scanning
Trivy Operator deployment via Helm, VulnerabilityReport and ConfigAuditReport CRDs, Prometheus metrics integration, alerting on HIGH/CRITICAL CVEs, and Trivy in GitLab CI with SARIF upload.
-
Terraform modules for PostgreSQL RBAC: roles, users, and default privileges
Reusable Terraform modules for PostgreSQL using the cyrilgdn/postgresql provider: db_role, db_user, default_privileges, S3 backend state locking, and GitLab CI pipeline for plan/apply with env promotion.
-
GitLab Runner in Kubernetes
Helm deployment, Docker-in-Docker with emptyDir for TLS certs, per-job resource limits in runner config, registration token via ESO, S3 cache for fast builds.
-
Running Kafka on Kubernetes with the Bitnami chart
Bitnami Kafka chart: KRaft mode without ZooKeeper, persistent storage, external access, SASL/SCRAM authentication, Prometheus metrics, and kafka-exporter ServiceMonitor.
-
APISIX as a Kubernetes ingress: global rules, TLS, and OpenTelemetry
ApisixGlobalRule for Elasticsearch logging, OpenTelemetry tracing, gzip, and real-ip. ApisixTls for wildcard certs. Two ingress classes: internal and external.
-
Prometheus exporters in Kubernetes: kafka, mongodb, redis, and more
HelmRelease configs for kafka-exporter, mongodb-exporter, and redis-exporter. RabbitMQ built-in Prometheus support. helm-exporter for tracking chart versions. ServiceMonitor patterns for each.
-
node-exporter alerting: clocks, disks, and inotify limits
PrometheusRule alerts for clock skew, CPU saturation, memory pressure, disk fullness prediction, inotify watch exhaustion, and network error rates — with practical thresholds from production.
-
SonarQube in Kubernetes: deployment and GitLab CI integration
SonarQube HelmRelease with PostgreSQL backend (CloudNativePG), GitLab OAuth, ESO for credentials, sonar-scanner-cli job in GitLab CI, GIT_DEPTH:0 for blame.
-
Kafka alerting in Kubernetes: from broker down to consumer lag
PrometheusRule alerts for Kafka: BrokerDown, UnderReplicatedPartitions, OfflinePartitions, consumer group lag, disk fill prediction, and JVM memory pressure with kafka-exporter metrics.
-
SealedSecrets: storing encrypted kubeconfigs in git
Bitnami SealedSecrets encrypt cluster credentials for safe git storage. Sealing workflow, namespace binding, update process, and controller troubleshooting.
-
Writing good PrometheusRules: structure, labels, and unit tests
PrometheusRule anatomy with required helm-chart/namespace labels, Grafana deep-link annotations, humanizeTimestamp and humanizeDuration in alert messages, and promtool rule unit test framework.
-
imagePullSecrets at scale: Kustomize secretGenerator across namespaces
Kustomize secretGenerator for Docker registry credentials, namespace-targeted patches, service account imagePullSecrets injection, and managing secrets across many namespaces without duplication.
-
Elastic APM Server in Kubernetes
APM Server HelmRelease from the elastic chart, OTLP/gRPC ingestion, Elasticsearch credentials via ESO, number_of_replicas:0 for single-node, bootstrap in the ES script.
-
External Secrets Operator with HashiCorp Vault: JWT auth and KV v2
ESO ClusterSecretStore with Vault JWT authentication via Kubernetes service accounts, ExternalSecret syncing KV v2 paths, refresh interval, secret templating, and ESO vs SealedSecrets tradeoffs.
-
Bootstrapping k3s clusters with Ansible and Flux
Ansible handles OS prep, k3s install, and Cilium setup. Flux bootstrap via ansible-playbook. Spoke kubeconfigs sealed and committed.
-
Sealed Secrets: the full ceremony from install to cluster migration
kubeseal workflow: controller install, offline sealing with pub cert, cert backup procedures, rotating controller keys, and re-sealing all secrets when migrating to a new cluster.
-
HashiCorp Vault: bootstrap, unseal, and Kubernetes auth
Vault HA with Raft storage, operator init ceremony, manual unseal vs YC KMS auto-unseal, Kubernetes auth method setup, KV v2, policies, and ESO role binding.
-
FluxCD hub-and-spoke: one cluster to manage them all
FluxCD v2 hub-and-spoke: hub cluster runs Flux controllers, spoke clusters are managed via spec.kubeConfig in HelmRelease and Kustomization. KubeConfig secret setup, dependsOn across clusters.
-
Hubble: network observability built into Cilium
Hubble UI and Relay setup in Cilium, flow inspection with hubble observe, DNS query visibility, policy verdict monitoring, and building Grafana dashboards from Hubble Prometheus metrics.
-
Kustomize base/custom/patch: DRY Helm values across environments
Three-tier Kustomize layout for six clusters: shared base, optional custom components, env-specific delta patches. The directory-only rule and kubeConfig JSON patch.
-
Cilium as kube-proxy replacement with L2 LoadBalancer on k3s
k3s installation without kube-proxy, Cilium Helm values for kube-proxy replacement mode, L2 announcement CiliumLoadBalancerIPPool and CiliumL2AnnouncementPolicy, and BGP alternative comparison.
-
Headlamp: a Kubernetes UI that doesn't require kubectl
Headlamp deployed as a HelmRelease with OIDC via GitLab, ApisixRoute for ingress, RBAC bindings from GitLab groups, multi-cluster kubeconfig in a Secret.
-
Ansible playbook for k3s: sysctl, kernel modules, and Cilium bootstrap
Idempotent Ansible playbook: sysctl settings for Kubernetes networking, inotify limits, swap disable, kernel modules, k3s server installation with --flannel-backend=none, and Cilium bootstrap via helm.
-
FluxCD hub-and-spoke: managing multiple Kubernetes clusters from one git repo
Flux runs on one hub cluster and deploys to five spokes via kubeConfig secrets. The four Kustomization objects per environment, dependsOn ordering, and daily operations.
No posts match the selected filters.