tag: devops
- pinned Eight years in DevOps: an engineer's path from IT support to Kubernetes A DevOps career path: 8 years from IT support to a 25-cluster AWS/EKS reference architecture — Sirena GDS, a greenfield startup, FluxCD at MixVel, MTS fintech, Red Rose, now Finstar.
- Bare-metal k8s: keep Cilium L2 LoadBalancer & node maintenance reliable Make Cilium L2 LoadBalancer IPs stable on bare-metal Kubernetes, tune node networking, and keep upgrades and reboots from breaking reachability.
- Lost the only SSH key on a Yandex Cloud fleet: getting back in Why detaching a boot disk fails, the snapshot-based rescue technique that works instead, and scaling it to 80 hosts with Ansible without losing quorum.
- Kubespray: a Dedicated Control Plane and Two Workers to Offload a Saturated Node Splitting an overloaded single Kubernetes node into a dedicated control plane plus two workers with Kubespray v2.31: inventory, Cilium, and workload isolation.
- k3s on Two VMs: Choosing a Load Balancer and Storage for a Test Cluster Installing k3s server+agent on two VMs, comparing MetalLB vs Cilium L2 for LB, and choosing a StorageClass: local-path, NFS Provisioner, or Longhorn — with pitfalls explained for each.
- CoreDNS as Authoritative DNS in Kubernetes: Corefile and Zones How CoreDNS becomes authoritative for cluster.local, how to add custom zones via the file plugin or stub forwarding, and what to do when DNS breaks.
- OpenTelemetry as APM replacement: one SDK, any backend How OpenTelemetry replaces vendor APM tools (Elastic, Datadog, Dynatrace) with one SDK, one protocol, and a Grafana stack for zero licensing cost.
- Shadowsocks proxy in Kubernetes Running shadowsocks-libev in a Kubernetes pod as a NodePort service. Cipher selection, UDP relay, Secret management, and the setup script.
- NuGet library CI: versioning from branch name and dual-registry publish dotnet pack with version from branch name, push to GitLab Package Registry via CI_JOB_TOKEN, selective publish to Artifactory for stable releases with manual gate and user restriction.
- GitLab CI multi-environment pipeline: workflow rules and deploy gates workflow.rules to suppress duplicate pipelines, stages for dev/test/demo/preprod/prod, per-environment deploy jobs extending a shared .deploy anchor, manual prod gate with user restrictions, Grafana deploy annotations.
- Helm chart config pattern: Files.Get, tpl, and the config/ directory Storing appsettings.json as a consul-template HCL source in config/, loading rendered output into ConfigMaps with Files.Get + tpl, subPath volumeMounts, and checksum annotations for forced rolling restarts.
- Beszel: lightweight server monitoring with Telegram alerts A Go binary that fits in 50 MiB and sends Telegram alerts when the server goes down. Why not Grafana.
- consul-template in the GitLab CI deploy job Using consul-template -once to render Vault secrets into Helm chart config/ before helm upgrade. HCL template patterns for per-service and shared secrets, CA cert rendering, and why this beats ESO for CI-time secret injection.
- Debugging Flux reconciliation: a field guide Where to look when Flux doesn't reconcile: get all, describe, force reconcile, suspend/resume, common error patterns for Kustomization and HelmRelease.
- Multi-cluster Prometheus: remote_write to central VictoriaMetrics Spoke Prometheus instances remote_write to a central VictoriaMetrics on the infra cluster. externalLabels.cluster, writeRelabelConfigs for traffic reduction, cross-cluster dashboards.
- Blackbox exporter and Prometheus Probes for endpoint monitoring Using prometheus-blackbox-exporter and Probe CRDs to monitor HTTP endpoints, TCP ports, and TLS certificate expiry across clusters.
- Ansible for multi-cluster k3s management Inventory structure, node preparation role, k3s install, Cilium bootstrap before Flux, kubeconfig extraction with host substitution. Idempotent playbooks for OS-level changes.
- Adding a new environment to a FluxCD hub-and-spoke cluster Full checklist: provision cluster, seal kubeconfig, create Kustomize overlays, patches, routes, and register in the Flux root kustomization.
- Deploying to k0s with a shell script instead of CI/CD One shell script, a self-hosted Docker registry, and envsubst. No GitHub Actions, no ArgoCD — here's why that's enough.
- Yandex Cloud Managed Kubernetes: specifics vs on-prem k3s What YC manages for you, storage classes, node groups, CCM service accounts, no Cilium L2 needed, kubeconfig via YC CLI, and Flux patch differences.
- Terraform Operator: running Terraform apply from inside Kubernetes GalleyBytes terraform-operator runs Terraform as a CRD workload. State on a PVC, SSH key reused from Flux, outputs written to object status.
- LimitRange, ResourceQuota, and VPA: controlling resource usage LimitRange injects default requests/limits. ResourceQuota caps namespace totals and blocks cloud LoadBalancers in dev. VPA in Off mode gives right-sizing recommendations.
- Helm charts declared in k0s config: Flannel, Traefik, cert-manager, Prometheus k0s has a built-in Helm controller. Declare charts in k0s.yaml and they install automatically on startup — no FluxCD required.
- Kubernetes RBAC patterns: per-namespace roles in GitOps Aggregated ClusterRoles for developer and devops, RoleBindings scoped per namespace managed by Flux, ClusterRoleBinding for monitoring read-only access.
- Trivy + SonarQube: security scanning in a GitLab CI pipeline Filesystem and live Kubernetes cluster scans with Trivy. SARIF for the repo scan, JSON-to-external-issues conversion for k8s findings, imported into SonarQube.
- Setting up a Debian VPS for k0s: sysctl, kernel modules, and the install What 01-prepare-vm.sh actually does: disabling swap, tuning inotify limits, loading kernel modules, and installing k0s single-node.
- GitLab CI for an infra repo: lint, validate, and flux reconcile Pipeline stages for a Kubernetes infra repository: yamllint, kubectl kustomize per environment, flux reconcile on merge, Telegram/Mattermost notifications.
- cert-manager: wildcard certificates with DNS-01 challenge ClusterIssuer with ACME DNS-01 via Yandex Cloud DNS webhook. Certificate CR for *.dev.test.antonnovikov.com, sealed distribution to spoke clusters.
- GitLab CI DinD: Docker builds with BuildKit caching Docker-in-Docker setup with TLS, registry layer caching with mode=max, registry mirror to bypass DockerHub rate limits, multi-arch builds with QEMU, and Trivy scan in the same pipeline.
- env-view: a custom Helm chart for per-environment service dashboards A static nginx page rendered from Helm values — every cluster gets a links page listing all service URLs, managed by FluxCD.
- CloudNativePG: production PostgreSQL in Kubernetes CNPG operator: streaming replication with automatic failover, WAL archival to S3, scheduled base backups, point-in-time recovery, PgBouncer connection pooling, and monitoring with PrometheusRule.
- Elasticsearch Index Lifecycle Management ILM policy hot→delete for vector-* indices: rollover at 20GB or 7 days, delete after 30 days. Index template, write alias bootstrap, idempotent bootstrap script.
- External Secrets Operator + HashiCorp Vault: Kubernetes auth flow ESO authenticates to Vault using a Kubernetes ServiceAccount token. ClusterSecretStore, ExternalSecret, and the bootstrap steps for Vault Kubernetes auth.
- Grafana datasource and dashboard provisioning via ConfigMap additionalDataSources in kube-prom-stack values, sidecar dashboard discovery from ConfigMaps with grafana_dashboard label, dashboards in git, folder structure.
- Vector: log pipeline from Kubernetes to Elasticsearch Vector DaemonSet, VRL transforms for JSON parsing and health-check filtering, sensitive field redaction, ServiceMonitor for pipeline metrics, and dual sink to Elasticsearch plus S3 archive.
- Elasticsearch in Kubernetes: HelmRelease, ingest pipelines, and bootstrap Deploying Elasticsearch via Flux from a git-bundled chart, then bootstrapping index templates, ingest pipelines, and replica settings with a shell script.
- Helm values-per-environment: the -f layering pattern Base values.yaml with production defaults, minimal per-env override files, FluxCD HelmRelease inline values, dry-run template verification, and why to avoid --set in production pipelines.
- Prometheus recording rules: pre-aggregating expensive queries PrometheusRule with record: rules for CPU/memory/HTTP rate. Naming convention level:metric:operations. SLO burn rate multi-window pre-computation.
- Self-hosted Docker registry in Kubernetes: setup, auth, and cleanup registry:2 on a NodePort, PVC storage, insecure-registries config for Docker hosts, CI runner push workflow, imagePullSecret, and weekly garbage collection CronJob for deleted blobs.
- VictoriaLogs + Vector: replacing EFK with a lightweight log stack Vector DaemonSet routes app logs to Elasticsearch and error-level logs to VictoriaLogs. VRL transforms, deduplication, and the VictoriaLogs Elasticsearch-compatible sink.
- Multi-stage Dockerfile for .NET: build once, run lean Three-stage pattern with restore/build/runtime layers, cache optimization by separating project files from source code, non-root user, readOnlyRootFilesystem, BuildKit registry caching in GitLab CI.
- VictoriaMetrics Operator: VMAgent, VMSingle, VMServiceScrape VMSingle for 6-month retention, VMAgent as scraper with remote_write only, VMServiceScrape as drop-in for ServiceMonitor, VMRule, Grafana datasource setup.
- Alertmanager routing to Telegram and Mattermost Custom Go templates for alert messages, severity-mapped emoji, cluster labels in group_by, and repeat intervals that don't wake you up every 30 minutes.
- Flux Notification Controller: commit status, Telegram, Mattermost Provider and Alert CRDs: GitLab commit status on every reconcile, generic webhook to abot for Telegram fan-out, per-namespace Alert objects for environment-scoped noise.
- Terraform modules for PostgreSQL RBAC: roles, users, and default privileges Reusable Terraform modules for PostgreSQL using the cyrilgdn/postgresql provider: db_role, db_user, default_privileges, S3 backend state locking, and GitLab CI pipeline for plan/apply with env promotion.
- GitLab Runner in Kubernetes Helm deployment, Docker-in-Docker with emptyDir for TLS certs, per-job resource limits in runner config, registration token via ESO, S3 cache for fast builds.
- Running Kafka on Kubernetes with the Bitnami chart Bitnami Kafka chart: KRaft mode without ZooKeeper, persistent storage, external access, SASL/SCRAM authentication, Prometheus metrics, and kafka-exporter ServiceMonitor.
- Prometheus exporters in Kubernetes: kafka, mongodb, redis, and more HelmRelease configs for kafka-exporter, mongodb-exporter, and redis-exporter. RabbitMQ built-in Prometheus support. helm-exporter for tracking chart versions. ServiceMonitor patterns for each.
- SonarQube in Kubernetes: deployment and GitLab CI integration SonarQube HelmRelease with PostgreSQL backend (CloudNativePG), GitLab OAuth, ESO for credentials, sonar-scanner-cli job in GitLab CI, GIT_DEPTH:0 for blame.
- SealedSecrets: storing encrypted kubeconfigs in git Bitnami SealedSecrets encrypt cluster credentials for safe git storage. Sealing workflow, namespace binding, update process, and controller troubleshooting.
- Writing good PrometheusRules: structure, labels, and unit tests PrometheusRule anatomy with required helm-chart/namespace labels, Grafana deep-link annotations, humanizeTimestamp and humanizeDuration in alert messages, and promtool rule unit test framework.
- imagePullSecrets at scale: Kustomize secretGenerator across namespaces Kustomize secretGenerator for Docker registry credentials, namespace-targeted patches, service account imagePullSecrets injection, and managing secrets across many namespaces without duplication.
- HashiCorp Vault: bootstrap, unseal, and Kubernetes auth Vault HA with Raft storage, operator init ceremony, manual unseal vs YC KMS auto-unseal, Kubernetes auth method setup, KV v2, policies, and ESO role binding.
- FluxCD hub-and-spoke: one cluster to manage them all FluxCD v2 hub-and-spoke: hub cluster runs Flux controllers, spoke clusters are managed via spec.kubeConfig in HelmRelease and Kustomization. KubeConfig secret setup, dependsOn across clusters.
- Headlamp: a Kubernetes UI that doesn't require kubectl Headlamp deployed as a HelmRelease with OIDC via GitLab, ApisixRoute for ingress, RBAC bindings from GitLab groups, multi-cluster kubeconfig in a Secret.
- Ansible playbook for k3s: sysctl, kernel modules, and Cilium bootstrap Idempotent Ansible playbook: sysctl settings for Kubernetes networking, inotify limits, swap disable, kernel modules, k3s server installation with --flannel-backend=none, and Cilium bootstrap via helm.
- FluxCD hub-and-spoke: managing multiple Kubernetes clusters from one git repo Flux runs on one hub cluster and deploys to five spokes via kubeConfig secrets. The four Kustomization objects per environment, dependsOn ordering, and daily operations.
No posts match the selected filters.