← weblog / 2026-07-06-yandex-cloud-lost-ssh-key-disk-rescue
2026-07-06·yandex-cloudansibledisaster-recoverylinuxdevops·9 min read

Lost the only SSH key on a Yandex Cloud fleet: getting back in

Published: 2026-07-06

An admin leaves, and their key turns out to be the only one cloud-init ever wrote onto dozens of production VMs. No OS Login, no second key, no password anyone remembers — just a password hash in an old secrets dump that doesn't turn back into a login. ssh answers Permission denied (publickey), and the first instinct is to push a new key through instance metadata. That didn't work. The second instinct is to attach the disk to another VM and fix it there. That didn't work either, for a different reason. Here's what actually did, and how it turned into automation for a whole fleet instead of one machine.

Why the obvious fixes don't work

Adding a key with yc compute instance add-metadata --metadata-from-file ssh-keys=... is the first thing everyone tries, and it's right maybe seventy percent of the time. On instances built from an image with a live guest agent polling metadata, the key lands in authorized_keys within a minute. The problem is that VMs provisioned by hand through cloud-init user-data usually apply that block once, at first boot, and never read it again. Push a new key through metadata on one of those and ssh -vvv will happily offer it — the server just never picked it up.

The second instinct — attach the disk elsewhere, chroot in, add a user — is the right idea. The direct version of it isn't:

$ yc compute instance detach-disk --name my-host --disk-id fhmxxxxxxxxxxxxxxxxx
ERROR: rpc error: code = InvalidArgument
desc = Request validation error: Can't detach boot disk

Yandex Cloud won't let you detach a disk while it's an instance's boot disk, stopped or not. That's a platform rule, not a CLI quirk. So the fix has to work on a copy of the disk, not the disk itself.

What works: snapshot instead of the live disk

Stop the instance, snapshot its boot disk, and build a new disk from that snapshot — it isn't anyone's boot disk yet, so it can be attached anywhere. Attach that new disk as a secondary disk to a rescue VM, mount it, chroot in, add a recovery user with a key and passwordless sudo. Detach it, delete the old instance (leave the disk alone — that's the rollback), and recreate the instance on the fixed disk with the same name and the same internal IP.

The original disk is never detached from anything — only ever read through a snapshot. What comes out the other end is a different instance object wearing the same name and IP, on a disk identical to the original plus one added user.

Snapshot and the fixed disk

bashFOLDER_ID=b1xxxxxxxxxxxxxxxxxx
HOST=my-host

DISK_ID=$(yc compute instance get --name $HOST --folder-id $FOLDER_ID --full --format json \
  | python3 -c "import json,sys;print(json.load(sys.stdin)['boot_disk']['disk_id'])")

yc compute instance stop --name $HOST --folder-id $FOLDER_ID

SNAP_ID=$(yc compute snapshot create --name ${HOST}-snap-final --disk-id $DISK_ID \
  --folder-id $FOLDER_ID --format json | python3 -c "import json,sys;print(json.load(sys.stdin)['id'])")

Don't hardcode the new disk's size — it has to be at least as large as the source, and disks across a real fleet aren't uniform (more on this below, because it bit me):

bashSRC_SIZE_GB=$(yc compute disk get --id $DISK_ID --folder-id $FOLDER_ID --format json \
  | python3 -c "import json,sys;print(-(-int(json.load(sys.stdin)['size'])//1073741824))")

NEW_DISK=$(yc compute disk create --name ${HOST}-boot-fixed --source-snapshot-id $SNAP_ID \
  --zone ru-central1-a --type network-hdd --size $SRC_SIZE_GB --folder-id $FOLDER_ID --format json \
  | python3 -c "import json,sys;print(json.load(sys.stdin)['id'])")

Rescue VM, mount, recovery user

A small VM in the same zone, never itself a recovery target — just a shell you trust:

bashyc compute instance attach-disk --name rescue-vm --disk-id $NEW_DISK --folder-id $FOLDER_ID

On the rescue VM:

bashmkdir -p /mnt/rescue
mount /dev/disk/by-id/virtio-${NEW_DISK}-part2 /mnt/rescue

mkdir -p /mnt/rescue/home/devops/.ssh
chroot /mnt/rescue useradd -m -s /bin/bash -G sudo devops 2>/dev/null || true
echo "ssh-ed25519 AAAA... devops" > /mnt/rescue/home/devops/.ssh/authorized_keys
chmod 700 /mnt/rescue/home/devops/.ssh
chmod 600 /mnt/rescue/home/devops/.ssh/authorized_keys
chroot /mnt/rescue chown -R devops:devops /home/devops
echo "devops ALL=(ALL) NOPASSWD:ALL" > /mnt/rescue/etc/sudoers.d/devops
umount /mnt/rescue

/dev/disk/by-id/virtio-${NEW_DISK}-part2 isn't an arbitrary path — it's there to avoid the race condition in the next section. More on why below.

Moving the instance onto the fixed disk

bashADDR=$(yc compute instance get --name $HOST --folder-id $FOLDER_ID --format json \
  | python3 -c "import json,sys;print(json.load(sys.stdin)['network_interfaces'][0]['primary_v4_address']['address'])")
SUBNET=$(yc compute instance get --name $HOST --folder-id $FOLDER_ID --format json \
  | python3 -c "import json,sys;print(json.load(sys.stdin)['network_interfaces'][0]['subnet_id'])")

yc compute instance detach-disk --name rescue-vm --disk-id $NEW_DISK --folder-id $FOLDER_ID
yc compute instance delete --name $HOST --folder-id $FOLDER_ID

yc compute instance create --name $HOST --zone ru-central1-a \
  --network-interface subnet-id=$SUBNET,address=$ADDR \
  --use-boot-disk disk-id=$NEW_DISK,auto-delete=false \
  --memory 4 --cores 4 --folder-id $FOLDER_ID

address=$ADDR pins the same internal IP the instance had before — DNS, other services' config, firewall rules don't notice a thing. Verify:

bashssh devops@$ADDR 'echo ok; sudo whoami'

On one machine this took about fifteen minutes, most of it waiting on a 100 GB snapshot. Then it turned out there wasn't just one machine.

One missing key is rarely on one machine

I checked the rest of the fleet the same way I'd tracked down the original alert — walked every instance's metadata and looked for the same departed admin's key in user-data. Out of 118 instances in the folder, 80 had it. Databases, queues, caches, a bastion, and — worse — the VPN gateways and routers that are the network path to everything else.

Running the same script against all 80 at once is a bad idea for two reasons. First, some of them are replicated clusters (Cassandra, RabbitMQ, etcd) where taking down more than one node at a time risks the quorum. Second, some of them aren't Linux at all (router appliances) or sit behind a managed database control plane — chroot doesn't apply there and shouldn't be attempted.

I grouped everything by what breaks if it goes wrong, not by service type:

yaml# inventory/hosts.yml
all:
  children:
    quorum_sensitive:      # replicated clusters — strictly one node at a time
      hosts:
        db-node-01: {}
        db-node-02: {}
        db-node-03: {}
    independent:            # no cross-host dependency — safe in parallel
      hosts:
        cache-node-01: {}
        bastion-01: {}
    network_path:            # this IS the path to everything else — manual, last
      hosts:
        vpn-gw-01: {}
        router-01: {}

yaml- name: Recover access — quorum-sensitive
  hosts: quorum_sensitive
  serial: 1                 # don't touch node N+1 until node N is back in the cluster
  tasks:
    - include_tasks: tasks/recover_one.yml
    - name: wait for cluster health
      command: ssh devops@{{ recovered_addr }} "cluster-status-check"
      delegate_to: localhost
      register: check
      until: check.rc == 0
      retries: 60
      delay: 10

- name: Recover access — independent
  hosts: independent
  serial: "100%"
  tasks:
    - include_tasks: tasks/recover_one.yml

# network_path isn't referenced by any play. Those hosts get done one at a time,
# by hand, only after a fallback access path is confirmed working. Breaking your
# own VPN gateway mid-recovery cuts you off from everything else you're fixing.

Each task looks up its own zone, subnet, disk size, and IP from the hostname at runtime — the inventory only needs the name. That keeps the blast radius visible at a glance: whatever's listed gets touched, and the grouping tells you exactly what's safe to parallelize and what isn't.

The race condition when one rescue VM serves a whole zone

With one extra disk attached to the rescue VM at a time, finding it in lsblk is fine — grab the last partition. It stops being fine at scale: several hosts in the same zone processed together, sharing one rescue VM instead of spinning up dozens. Two concurrent attach-disk calls followed by two scripts both grabbing "the last partition in lsblk" will grab the same disk. One host's fix lands on another host's filesystem.

Yandex Cloud exposes every attached disk at a deterministic path keyed to its own ID, regardless of attach order:

bash$ ls -la /dev/disk/by-id/ | grep virtio
virtio-fhmlrpr6g54rfs0dal0p       -> ../../vdb
virtio-fhmlrpr6g54rfs0dal0p-part1 -> ../../vdb1
virtio-fhmlrpr6g54rfs0dal0p-part2 -> ../../vdb2

Which is where virtio-${NEW_DISK}-part2 in the mount step above comes from. I didn't take this on faith — attached a disk, looked at by-id, and the symlink was exactly where it should be. The fleet-scale script picks the largest partition under that ID (in practice the root filesystem) instead of guessing from lsblk:

bashBASE="/dev/disk/by-id/virtio-${NEW_DISK}"
for i in $(seq 1 15); do compgen -G "${BASE}*" >/dev/null 2>&1 && break; sleep 1; done

BEST=""; BEST_SIZE=0
for p in ${BASE}*; do
  [ -e "$p" ] || continue
  SZ=$(blockdev --getsize64 "$p" 2>/dev/null || echo 0)
  [ "$SZ" -gt "$BEST_SIZE" ] && { BEST_SIZE=$SZ; BEST="$p"; }
done
[ -z "$BEST" ] && { echo "FATAL: by-id device not found, refusing to guess" >&2; exit 1; }

If the symlink isn't there after 15 seconds, this fails loudly instead of guessing. A loud failure beats a silent write to the wrong disk.

What actually broke on the real run

The first host outside the initial test — a bastion with a 1 TB disk — cost half an hour. yc compute disk create refused immediately:

ERROR: rpc error: code = InvalidArgument
desc = Request validation error: Disk size must be greater or equal than 1024.0 GB

The reason was dumb and entirely mine: I'd hardcoded --size 100, based on the first couple of machines, which happened to have 100 GB disks. Not this one. Fixed by the SRC_SIZE_GB step above — always read the size off the source disk, never hardcode it.

Second problem, same bastion, same 1 TB disk, this time the snapshot:

ERROR: operation (id=...) poll fail: rpc error: code = Unavailable
desc = error reading from server: read tcp ...: read: operation timed out
...14m26s

The CLI gave up after fourteen and a half minutes of polling and reported failure. By then the instance was already stopped. First instinct was to retry. Good thing I checked first:

bashyc operation get <operation-id>

done: true, snapshot READY — the operation had kept working server-side for almost forty more minutes after the CLI stopped listening. Retrying with the same snapshot name would have failed on "already exists" and left the bastion down for nothing, twice as long. Since then, for anything over a hundred gigabytes, I check yc operation get before deciding whether to retry or just continue from the next step by hand.

Summary

  • Adding a key through instance metadata only helps if something is actually still polling for it — hand-provisioned VMs often apply user-data once and never again
  • A boot disk can't be detached, running or stopped — work from a snapshot-derived copy instead
  • The shape of it: stop → snapshot → new disk from the snapshot → mount and fix on a rescue VM → delete the old instance, recreate it on the fixed disk with the same name and IP
  • Always read the new disk's size off the source — a size that works on one host will fail on another
  • When several hosts share one rescue VM, locate the disk by /dev/disk/by-id/virtio-<disk-id>, never by lsblk order
  • At fleet scale, group hosts by what breaks, not by service type — handle the network path (VPN/routers) last, by hand
  • A "failed" long-running CLI operation on a big disk may have finished server-side anyway — check yc operation get before retrying