tag: kubernetes
- pinned Eight years in DevOps: an engineer's path from IT support to Kubernetes A DevOps career path: 8 years from IT support to a 25+ cluster AWS/EKS reference architecture — Sirena GDS, a greenfield startup, FluxCD at MixVel, MTS fintech, Red Rose, now Finstar.
- One Evening, Three Problems: Mail Down, Memory Full, Wrong Docker Arch Incident log from a 4GB single-node k0s box: snappymail OOMKilled loop, 163% memory overcommit, swap thrashing disguised as disk I/O, and a docker buildx arch mismatch.
- Traefik → APISIX on a single node: what broke during the ingress migration A direct cutover from Traefik to APISIX on bare-metal k0s: hostNetwork port conflicts, a broken --set with commas, an addPrefix bug causing 404s on assets, and eight more findings.
- Bare-metal k8s: keep Cilium L2 LoadBalancer & node maintenance reliable Make Cilium L2 LoadBalancer IPs stable on bare-metal Kubernetes, tune node networking, and keep upgrades and reboots from breaking reachability.
- Kubespray: a Dedicated Control Plane and Two Workers to Offload a Saturated Node Splitting an overloaded single Kubernetes node into a dedicated control plane plus two workers with Kubespray v2.31: inventory, Cilium, and workload isolation.
- k3s on Two VMs: Choosing a Load Balancer and Storage for a Test Cluster Installing k3s server+agent on two VMs, comparing MetalLB vs Cilium L2 for LB, and choosing a StorageClass: local-path, NFS Provisioner, or Longhorn — with pitfalls explained for each.
- Wildcard TLS with ACME and cert-manager: Automating Let's Encrypt via Yandex Cloud DNS How ACME and DNS-01 work, why wildcard certs require it, and how to automate Let's Encrypt in Kubernetes with cert-manager and the Yandex Cloud DNS webhook in a FluxCD GitOps setup.
- CoreDNS as Authoritative DNS in Kubernetes: Corefile and Zones How CoreDNS becomes authoritative for cluster.local, how to add custom zones via the file plugin or stub forwarding, and what to do when DNS breaks.
- Full observability on one VPS: metrics, logs, alerts, dashboards — 452 MiB requested How VictoriaMetrics, VictoriaLogs, Promtail, vmalert, and Grafana fit together on a 2-core VPS: architecture, the full resource bill, and repeating patterns.
- OpenTelemetry as APM replacement: one SDK, any backend How OpenTelemetry replaces vendor APM tools (Elastic, Datadog, Dynatrace) with one SDK, one protocol, and a Grafana stack for zero licensing cost.
- vmsingle: one VictoriaMetrics binary instead of the whole Prometheus stack VictoriaMetrics single-node on k0s: embedded scraper with static targets, cAdvisor RBAC, hostPath PV, and a stable ClusterIP to survive CoreDNS restarts.
- Shadowsocks proxy in Kubernetes Running shadowsocks-libev in a Kubernetes pod as a NodePort service. Cipher selection, UDP relay, Secret management, and the setup script.
- How this site is built FastAPI, k0s, Traefik, WireGuard, and plain Markdown — a tour of the infrastructure behind this personal site.
- Rotating Tor HTTP proxy in Kubernetes Running zhaow-de/rotating-tor-http-proxy in k0s: N Privoxy+Tor circuits, HAProxy round-robin, stats UI on NodePort 30444.
- GitLab CI multi-environment pipeline: workflow rules and deploy gates workflow.rules to suppress duplicate pipelines, stages for dev/test/demo/preprod/prod, per-environment deploy jobs extending a shared .deploy anchor, manual prod gate with user restrictions, Grafana deploy annotations.
- Helm chart config pattern: Files.Get, tpl, and the config/ directory Storing appsettings.json as a consul-template HCL source in config/, loading rendered output into ConfigMaps with Files.Get + tpl, subPath volumeMounts, and checksum annotations for forced rolling restarts.
- consul-template in the GitLab CI deploy job Using consul-template -once to render Vault secrets into Helm chart config/ before helm upgrade. HCL template patterns for per-service and shared secrets, CA cert rendering, and why this beats ESO for CI-time secret injection.
- Debugging Flux reconciliation: a field guide Where to look when Flux doesn't reconcile: get all, describe, force reconcile, suspend/resume, common error patterns for Kustomization and HelmRelease.
- abot: Alertmanager proxy for Telegram and Mattermost notifications A custom Go service between Alertmanager and notification channels: fan-out, retry, HTTP proxy support, per-cluster routing.
- Multi-cluster Prometheus: remote_write to central VictoriaMetrics Spoke Prometheus instances remote_write to a central VictoriaMetrics on the infra cluster. externalLabels.cluster, writeRelabelConfigs for traffic reduction, cross-cluster dashboards.
- IKEv2 and WireGuard running in separate Kubernetes namespaces Why VPN pods run inside k0s with hostNetwork and privileged mode, and the tradeoffs that come with it.
- Blackbox exporter and Prometheus Probes for endpoint monitoring Using prometheus-blackbox-exporter and Probe CRDs to monitor HTTP endpoints, TCP ports, and TLS certificate expiry across clusters.
- Ansible for multi-cluster k3s management Inventory structure, node preparation role, k3s install, Cilium bootstrap before Flux, kubeconfig extraction with host substitution. Idempotent playbooks for OS-level changes.
- Adding a new environment to a FluxCD hub-and-spoke cluster Full checklist: provision cluster, seal kubeconfig, create Kustomize overlays, patches, routes, and register in the Flux root kustomization.
- Yandex Cloud Managed Kubernetes: specifics vs on-prem k3s What YC manages for you, storage classes, node groups, CCM service accounts, no Cilium L2 needed, kubeconfig via YC CLI, and Flux patch differences.
- Terraform Operator: running Terraform apply from inside Kubernetes GalleyBytes terraform-operator runs Terraform as a CRD workload. State on a PVC, SSH key reused from Flux, outputs written to object status.
- Self-hosted Docker registry inside the k0s cluster registry:2 on NodePort 30500, no external dependencies for deploys, and automatic cleanup of old image tags.
- LimitRange, ResourceQuota, and VPA: controlling resource usage LimitRange injects default requests/limits. ResourceQuota caps namespace totals and blocks cloud LoadBalancers in dev. VPA in Off mode gives right-sizing recommendations.
- Proxy mesh in Kubernetes: Tor exit nodes and geo-routing with hola-proxy Running Tor rotating proxies and country-specific hola-proxy instances in a Kubernetes namespace, with TCP health probes and internal routing.
- Kustomize configMapGenerator and secretGenerator Generating ConfigMaps and Secrets from files with content-hash rolling updates, disableNameSuffixHash for Helm, env file sources, and HelmRelease valuesFrom.
- Automatic TLS with cert-manager, Let's Encrypt, and Traefik IngressRoutes ClusterIssuers, HTTP-01 challenges, multi-SAN certificates, and addPrefix middleware for subdomain routing.
- OpenTelemetry bridge: APISIX → otel-collector → Elasticsearch APM APISIX sends OTLP/HTTP traces; APM Server wants gRPC. An otel-collector deployment bridges the two. fullnameOverride is critical.
- Flux Image Update Automation: auto-bump image tags in git ImageRepository scans the registry, ImagePolicy selects the right tag (semver, regex), ImageUpdateAutomation commits the updated tag back to git via marker annotations.
- Helm charts declared in k0s config: Flannel, Traefik, cert-manager, Prometheus k0s has a built-in Helm controller. Declare charts in k0s.yaml and they install automatically on startup — no FluxCD required.
- Kubernetes RBAC patterns: per-namespace roles in GitOps Aggregated ClusterRoles for developer and devops, RoleBindings scoped per namespace managed by Flux, ClusterRoleBinding for monitoring read-only access.
- Trivy + SonarQube: security scanning in a GitLab CI pipeline Filesystem and live Kubernetes cluster scans with Trivy. SARIF for the repo scan, JSON-to-external-issues conversion for k8s findings, imported into SonarQube.
- GitLab CI for an infra repo: lint, validate, and flux reconcile Pipeline stages for a Kubernetes infra repository: yamllint, kubectl kustomize per environment, flux reconcile on merge, Telegram/Mattermost notifications.
- cert-manager: wildcard certificates with DNS-01 challenge ClusterIssuer with ACME DNS-01 via Yandex Cloud DNS webhook. Certificate CR for *.dev.test.antonnovikov.com, sealed distribution to spoke clusters.
- Cilium L2 LoadBalancer on bare-metal k3s No cloud controller, no MetalLB — Cilium's built-in L2 announcements assign LoadBalancer IPs on bare-metal k3s using ARP.
- Cilium NetworkPolicy: default-deny and DNS-aware rules Default-deny ingress per namespace, allow from APISIX, CiliumNetworkPolicy toFQDNs for external APIs, L7 HTTP path rules, cluster-wide Prometheus scrape policy.
- Helm chart testing: lint, unit tests, and ct in CI helm lint, helm-unittest for template assertions, chart-testing (ct) for changed-chart detection, CI/CD stub values files, and what to actually test in your Helm charts.
- env-view: a custom Helm chart for per-environment service dashboards A static nginx page rendered from Helm values — every cluster gets a links page listing all service URLs, managed by FluxCD.
- CloudNativePG: production PostgreSQL in Kubernetes CNPG operator: streaming replication with automatic failover, WAL archival to S3, scheduled base backups, point-in-time recovery, PgBouncer connection pooling, and monitoring with PrometheusRule.
- Elasticsearch Index Lifecycle Management ILM policy hot→delete for vector-* indices: rollover at 20GB or 7 days, delete after 30 days. Index template, write alias bootstrap, idempotent bootstrap script.
- Loki and LogQL: logs without Elasticsearch overhead Grafana Loki in SingleBinary mode, Grafana Alloy as promtail replacement, LogQL stream selectors, JSON parsing, rate queries, and trace ID correlation between logs and Jaeger.
- Alertmanager routing: Telegram notifications with inhibition rules Alertmanager route tree with severity-based routing, Telegram bot receiver, inhibition rules to suppress cascading alerts, grouping configuration, silence API, and amtool for testing routing decisions.
- External Secrets Operator + HashiCorp Vault: Kubernetes auth flow ESO authenticates to Vault using a Kubernetes ServiceAccount token. ClusterSecretStore, ExternalSecret, and the bootstrap steps for Vault Kubernetes auth.
- kube-prometheus-stack: the full monitoring setup kube-prometheus-stack HelmRelease with Prometheus persistence and retention tuning, Grafana admin secret, ServiceMonitor for custom apps, sidecar dashboard loading, and storage sizing estimates.
- ELK in Kubernetes: Elasticsearch, APM Server, and Kibana ELK stack via Helm in FluxCD, Elastic APM agent in .NET with zero-code auto-instrumentation, index lifecycle management, and correlating logs with APM service maps in Kibana.
- Grafana datasource and dashboard provisioning via ConfigMap additionalDataSources in kube-prom-stack values, sidecar dashboard discovery from ConfigMaps with grafana_dashboard label, dashboards in git, folder structure.
- Vector: log pipeline from Kubernetes to Elasticsearch Vector DaemonSet, VRL transforms for JSON parsing and health-check filtering, sensitive field redaction, ServiceMonitor for pipeline metrics, and dual sink to Elasticsearch plus S3 archive.
- Elasticsearch in Kubernetes: HelmRelease, ingest pipelines, and bootstrap Deploying Elasticsearch via Flux from a git-bundled chart, then bootstrapping index templates, ingest pipelines, and replica settings with a shell script.
- OpenTelemetry in .NET: traces, metrics, and logs with minimal boilerplate OTel SDK setup in Program.cs, Prometheus exporter for /metrics, custom ActivitySource spans, custom Meter counters and histograms, OTLP exporter, and OTel Collector HelmRelease.
- Helm values-per-environment: the -f layering pattern Base values.yaml with production defaults, minimal per-env override files, FluxCD HelmRelease inline values, dry-run template verification, and why to avoid --set in production pipelines.
- Prometheus recording rules: pre-aggregating expensive queries PrometheusRule with record: rules for CPU/memory/HTTP rate. Naming convention level:metric:operations. SLO burn rate multi-window pre-computation.
- Self-hosted Docker registry in Kubernetes: setup, auth, and cleanup registry:2 on a NodePort, PVC storage, insecure-registries config for Docker hosts, CI runner push workflow, imagePullSecret, and weekly garbage collection CronJob for deleted blobs.
- VictoriaLogs + Vector: replacing EFK with a lightweight log stack Vector DaemonSet routes app logs to Elasticsearch and error-level logs to VictoriaLogs. VRL transforms, deduplication, and the VictoriaLogs Elasticsearch-compatible sink.
- GitLab CI Docker deploy pipeline: build, push, and rolling update Full GitLab CI pipeline for containerized apps: Docker build with BuildKit layer caching, push to private registry, kubectl rollout with SHA-pinned image tags, multi-environment promotion, and registry cleanup job.
- VictoriaMetrics Operator: VMAgent, VMSingle, VMServiceScrape VMSingle for 6-month retention, VMAgent as scraper with remote_write only, VMServiceScrape as drop-in for ServiceMonitor, VMRule, Grafana datasource setup.
- DefectDojo in Kubernetes: security findings as a first-class CI gate DefectDojo HelmRelease, Trivy and SonarQube report upload via API, product and engagement model, SLA enforcement in CI that fails pipelines on unacknowledged high-severity findings.
- Alertmanager routing to Telegram and Mattermost Custom Go templates for alert messages, severity-mapped emoji, cluster labels in group_by, and repeat intervals that don't wake you up every 30 minutes.
- APISIX as Kubernetes ingress: rate limiting, JWT routing, and Lua plugins APISIX Ingress Controller with ApisixRoute, limit-count for rate limiting per consumer, JWT claim-based routing for A/B testing, serverless Lua plugin for request transformation, canary releases.
- Traefik Middleware for security headers: HSTS, CSP, and frame protection Traefik Middleware resources for HSTS, X-Frame-Options, X-Content-Type-Options, Referrer-Policy, Permissions-Policy, and Content-Security-Policy. IngressRoute wiring and testing headers with curl.
- Flux Notification Controller: commit status, Telegram, Mattermost Provider and Alert CRDs: GitLab commit status on every reconcile, generic webhook to abot for Telegram fan-out, per-namespace Alert objects for environment-scoped noise.
- Cilium network policy: default-deny and workload isolation Default-deny CiliumNetworkPolicy per namespace, selective allow rules for DNS, ingress controller, inter-service traffic, FQDN-based egress policies, and L7 HTTP policy for path-level control.
- kube-prometheus-stack on multi-cluster: spoke Prometheus, central Grafana Grafana on one cluster, Prometheus on all six. Spoke kube-prom-stack with Grafana disabled, externalLabels.cluster for cross-cluster dashboards.
- Trivy Operator in Kubernetes: automated vulnerability scanning Trivy Operator deployment via Helm, VulnerabilityReport and ConfigAuditReport CRDs, Prometheus metrics integration, alerting on HIGH/CRITICAL CVEs, and Trivy in GitLab CI with SARIF upload.
- GitLab Runner in Kubernetes Helm deployment, Docker-in-Docker with emptyDir for TLS certs, per-job resource limits in runner config, registration token via ESO, S3 cache for fast builds.
- Running Kafka on Kubernetes with the Bitnami chart Bitnami Kafka chart: KRaft mode without ZooKeeper, persistent storage, external access, SASL/SCRAM authentication, Prometheus metrics, and kafka-exporter ServiceMonitor.
- APISIX as a Kubernetes ingress: global rules, TLS, and OpenTelemetry ApisixGlobalRule for Elasticsearch logging, OpenTelemetry tracing, gzip, and real-ip. ApisixTls for wildcard certs. Two ingress classes: internal and external.
- Prometheus exporters in Kubernetes: kafka, mongodb, redis, and more HelmRelease configs for kafka-exporter, mongodb-exporter, and redis-exporter. RabbitMQ built-in Prometheus support. helm-exporter for tracking chart versions. ServiceMonitor patterns for each.
- node-exporter alerting: clocks, disks, and inotify limits PrometheusRule alerts for clock skew, CPU saturation, memory pressure, disk fullness prediction, inotify watch exhaustion, and network error rates — with practical thresholds from production.
- SonarQube in Kubernetes: deployment and GitLab CI integration SonarQube HelmRelease with PostgreSQL backend (CloudNativePG), GitLab OAuth, ESO for credentials, sonar-scanner-cli job in GitLab CI, GIT_DEPTH:0 for blame.
- Kafka alerting in Kubernetes: from broker down to consumer lag PrometheusRule alerts for Kafka: BrokerDown, UnderReplicatedPartitions, OfflinePartitions, consumer group lag, disk fill prediction, and JVM memory pressure with kafka-exporter metrics.
- SealedSecrets: storing encrypted kubeconfigs in git Bitnami SealedSecrets encrypt cluster credentials for safe git storage. Sealing workflow, namespace binding, update process, and controller troubleshooting.
- Writing good PrometheusRules: structure, labels, and unit tests PrometheusRule anatomy with required helm-chart/namespace labels, Grafana deep-link annotations, humanizeTimestamp and humanizeDuration in alert messages, and promtool rule unit test framework.
- imagePullSecrets at scale: Kustomize secretGenerator across namespaces Kustomize secretGenerator for Docker registry credentials, namespace-targeted patches, service account imagePullSecrets injection, and managing secrets across many namespaces without duplication.
- Elastic APM Server in Kubernetes APM Server HelmRelease from the elastic chart, OTLP/gRPC ingestion, Elasticsearch credentials via ESO, number_of_replicas:0 for single-node, bootstrap in the ES script.
- External Secrets Operator with HashiCorp Vault: JWT auth and KV v2 ESO ClusterSecretStore with Vault JWT authentication via Kubernetes service accounts, ExternalSecret syncing KV v2 paths, refresh interval, secret templating, and ESO vs SealedSecrets tradeoffs.
- Bootstrapping k3s clusters with Ansible and Flux Ansible handles OS prep, k3s install, and Cilium setup. Flux bootstrap via ansible-playbook. Spoke kubeconfigs sealed and committed.
- Sealed Secrets: the full ceremony from install to cluster migration kubeseal workflow: controller install, offline sealing with pub cert, cert backup procedures, rotating controller keys, and re-sealing all secrets when migrating to a new cluster.
- HashiCorp Vault: bootstrap, unseal, and Kubernetes auth Vault HA with Raft storage, operator init ceremony, manual unseal vs YC KMS auto-unseal, Kubernetes auth method setup, KV v2, policies, and ESO role binding.
- FluxCD hub-and-spoke: one cluster to manage them all FluxCD v2 hub-and-spoke: hub cluster runs Flux controllers, spoke clusters are managed via spec.kubeConfig in HelmRelease and Kustomization. KubeConfig secret setup, dependsOn across clusters.
- Hubble: network observability built into Cilium Hubble UI and Relay setup in Cilium, flow inspection with hubble observe, DNS query visibility, policy verdict monitoring, and building Grafana dashboards from Hubble Prometheus metrics.
- Kustomize base/custom/patch: DRY Helm values across environments Three-tier Kustomize layout for six clusters: shared base, optional custom components, env-specific delta patches. The directory-only rule and kubeConfig JSON patch.
- Cilium as kube-proxy replacement with L2 LoadBalancer on k3s k3s installation without kube-proxy, Cilium Helm values for kube-proxy replacement mode, L2 announcement CiliumLoadBalancerIPPool and CiliumL2AnnouncementPolicy, and BGP alternative comparison.
- Headlamp: a Kubernetes UI that doesn't require kubectl Headlamp deployed as a HelmRelease with OIDC via GitLab, ApisixRoute for ingress, RBAC bindings from GitLab groups, multi-cluster kubeconfig in a Secret.
- Ansible playbook for k3s: sysctl, kernel modules, and Cilium bootstrap Idempotent Ansible playbook: sysctl settings for Kubernetes networking, inotify limits, swap disable, kernel modules, k3s server installation with --flannel-backend=none, and Cilium bootstrap via helm.
- FluxCD hub-and-spoke: managing multiple Kubernetes clusters from one git repo Flux runs on one hub cluster and deploys to five spokes via kubeConfig secrets. The four Kustomization objects per environment, dependsOn ordering, and daily operations.
No posts match the selected filters.