Yandex Cloud Managed Kubernetes: specifics vs on-prem k3s

Published: 2026-04-26

The infra runs two types of Kubernetes clusters: on-prem k3s (dev, test) and Yandex Cloud Managed k8s (sre, loadgds, demo). They use the same Flux configuration, but several layers differ.

What YC manages for you

YC Managed k8s abstracts away the most painful operational tasks: etcd backup/restore, certificate rotation, and OS patching on masters.

Storage classes

YC provides two storage classes out of the box:

bashcopykubectl get storageclass
# NAME                  PROVISIONER
# yc-network-hdd        disk-csi-driver.mks.ycloud.io
# yc-network-ssd        disk-csi-driver.mks.ycloud.io

Both are backed by network block storage — not local disks. This means:

• Volumes survive node replacement (safe for auto-upgrade).
• Throughput is limited vs local NVMe.
• Volumes are zone-specific — a PVC created in ru-central1-a can only attach to a pod scheduled in ru-central1-a.

SSD for databases and Prometheus:

yamlcopystorageSpec:
  volumeClaimTemplate:
    spec:
      storageClassName: yc-network-ssd
      resources:
        requests:
          storage: 20Gi

HDD for logs and archive storage:

yamlcopypersistence:
  storageClass: yc-network-hdd
  size: 100Gi

Node groups

YC clusters are defined with node groups instead of single nodes:

bashcopyyc managed-kubernetes node-group create \
  --cluster-name sre \
  --name default \
  --platform-id standard-v3 \
  --memory 16 \
  --cores 4 \
  --disk-type network-ssd \
  --disk-size 100 \
  --fixed-size 3 \
  --location zone=ru-central1-a

For autoscaling (Cluster Autoscaler is pre-installed on YC clusters):

bashcopy  --auto-scale min=2,max=6,initial=2

Node group auto-upgrade replaces nodes one by one during maintenance windows. This is transparent to pods because PVCs re-attach to the new node.

Service accounts for the cloud controller

On YC, you assign a service account at cluster creation time. The YC CCM uses this SA to:

• Create/delete Network Load Balancers when Service type: LoadBalancer is applied.
• Manage node labels and taints.
• Handle route tables for pod networking.

bashcopyyc iam service-account create k8s-ccm
yc iam role-assignment add \
  --role editor \
  --service-account-name k8s-ccm \
  --folder-id ${YC_FOLDER_ID}

yc iam key create \
  --service-account-name k8s-ccm \
  --output k8s-ccm-key.json

The key is mounted into the CCM pod automatically by YC — you don't manage this on managed clusters. On k3s with Cilium, there's no cloud CCM at all.

No Cilium L2 on YC

On YC clusters, CiliumLoadBalancerIPPool and CiliumL2AnnouncementPolicy are not deployed. YC provisions actual network load balancers when a Service of type: LoadBalancer is created.

The Kustomize patches for YC environments exclude the Cilium LB resources:

yamlcopy# projects/sre/kustomization/spoke/kustomization.yaml
resources:
  - ../../../../base/...
# Note: no cilium-lb-pool.yaml here

On-prem clusters have:

yamlcopyresources:
  - ../../../../base/...
  - cilium-lb-pool.yaml        # IP pool for Cilium L2
  - cilium-l2-policy.yaml      # ARP announcement policy

kubeconfig and context

YC clusters need their kubeconfig obtained via the YC CLI:

bashcopyyc managed-kubernetes cluster get-credentials sre --external
kubectl config rename-context yc-sre sre-k8s

The --external flag uses the cluster's external endpoint (requires whitelisting the management machine's IP in the cluster's security group). For CI/CD, use --internal and run from within the same VPC:

bashcopyyc managed-kubernetes cluster get-credentials sre --internal

For automated pipelines, authenticate with a service account key:

bashcopyyc config set service-account-key sa-key.json
yc managed-kubernetes cluster get-credentials sre --external

Differences in Flux patches

YC-specific patches in projects/sre/kustomization/hub/patches/:

yamlcopy# apisix-int.yaml — no static IP needed, YC assigns one
spec:
  values:
    service:
      type: LoadBalancer
      # no loadBalancerIP — YC auto-assigns

# kube-prom-stack.yaml — SSD storage
spec:
  values:
    prometheus:
      prometheusSpec:
        storageSpec:
          volumeClaimTemplate:
            spec:
              storageClassName: yc-network-ssd
    alertmanager:
      alertmanagerSpec:
        storage:
          volumeClaimTemplate:
            spec:
              storageClassName: yc-network-ssd

On-prem patches use local-path storageClass and a static LoadBalancer IP from the Cilium pool.

Cost considerations

On YC Managed k8s, you pay for:

• Control plane: fixed hourly rate regardless of node count.
• Nodes: per-vCPU and per-GB-RAM per hour.
• Network Load Balancers: per LB + per GB traffic.
• Storage: per GB per hour (SSD is ~3x the price of HDD).

To reduce costs:

• Use services.loadbalancers: "0" ResourceQuota in dev namespaces.
• Use HDD for non-latency-sensitive storage.
• Pre-emptible (spot) nodes for non-critical workloads.

Cluster creation via Terraform

The YC provider for Terraform has a yandex_kubernetes_cluster resource:

hclcopyresource "yandex_kubernetes_cluster" "sre" {
  name        = "sre"
  description = "SRE monitoring cluster"
  network_id  = yandex_vpc_network.main.id

  master {
    version = "1.28"
    zonal {
      zone      = "ru-central1-a"
      subnet_id = yandex_vpc_subnet.main.id
    }
    public_ip = true
  }

  service_account_id      = yandex_iam_service_account.k8s.id
  node_service_account_id = yandex_iam_service_account.k8s_nodes.id
}

resource "yandex_kubernetes_node_group" "default" {
  cluster_id = yandex_kubernetes_cluster.sre.id
  name       = "default"

  instance_template {
    platform_id = "standard-v3"
    resources {
      memory = 16
      cores  = 4
    }
    boot_disk {
      type = "network-ssd"
      size = 100
    }
  }

  scale_policy {
    fixed_scale {
      size = 3
    }
  }

  allocation_policy {
    location {
      zone = "ru-central1-a"
    }
  }
}

What can go wrong

FluxCD patches for on-prem don't apply to YC clusters. On-prem clusters may use Cilium L2 LoadBalancerIP fields or node selectors that don't exist in YC. Always create YC-specific overlays that remove these fields. Check with kubectl kustomize fluxcd/projects/sre/ | grep loadBalancerIP.

Service account doesn't have k8s.clusters.agent role. The YC service account needs specific IAM roles to push kubeconfig updates to the cluster. Without container.clusters.agent, kubectl returns 403. Assign via yc iam service-account add-access-binding.

Node group doesn't scale down. YC node autoscaler respects pod disruption budgets and local PVCs. Pods with hostPath volumes block scale-down. Prefer network-attached PersistentVolumeClaims (YC network-ssd) over hostPath on YC.

Summary

• YC Managed Kubernetes replaces kube-apiserver, etcd, and control-plane management; node groups are billed separately
• No Cilium L2 on YC — use YC's cloud load balancer (service.type: LoadBalancer) instead of LoadBalancerIP
• Default StorageClass is yc-network-ssd; on-prem overlays using hostPath need YC-specific patches
• kubeconfig for YC clusters is generated via yc managed-kubernetes cluster get-credentials and stored as a SealedSecret for Flux
• Same Flux base manifests work across on-prem and YC via per-environment Kustomize patches