Self-hosted Docker registry inside the k0s cluster

Published: 2026-06-09

There's no DockerHub account, no GHCR, no ECR in this setup. Images are pushed to a Docker Registry v2 running as a pod inside the same k0s cluster that runs the site. Here's the setup and the tradeoffs.

Why self-host

No external dependency for deploys: the deploy script pushes and the cluster pulls from the same server.
No rate limits (DockerHub limits anonymous pulls to 100/6h).
No secrets needed for registry auth — it's internal only, firewalled from the internet.
Images never leave the VPS.
No egress costs for image pulls.

The obvious downside: if the registry pod dies, you can't deploy. But on a single node, if the pod dies, you probably have bigger problems already.

What runs

05-setup-registry.sh deploys registry:2 (the official Docker Registry v2 image) in a registry namespace. Single replica with a hostPath volume at /opt/registry-data:

yamlapiVersion: apps/v1
kind: Deployment
metadata:
  name: registry
  namespace: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: registry
  template:
    spec:
      containers:
        - name: registry
          image: registry:2
          ports:
            - containerPort: 5000
          env:
            - name: REGISTRY_STORAGE_DELETE_ENABLED
              value: "true"
            - name: REGISTRY_HTTP_ADDR
              value: "0.0.0.0:5000"
          volumeMounts:
            - mountPath: /var/lib/registry
              name: data
      volumes:
        - name: data
          hostPath:
            path: /opt/registry-data
            type: DirectoryOrCreate

REGISTRY_STORAGE_DELETE_ENABLED: true is required for the deploy script to delete old image manifests via the registry's HTTP API. Without it, DELETE /v2/<name>/manifests/<digest> returns 405.

NodePort exposure

The registry is exposed on port 30500 via a NodePort service:

yamlapiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: registry
spec:
  type: NodePort
  selector:
    app: registry
  ports:
    - port: 5000
      targetPort: 5000
      nodePort: 30500

From the local machine (which has kubectl configured), pushing looks like:

bashdocker push 91.184.248.13:30500/antonnovikov-site:abc1234

The port is only accessible from machines with the firewall rule. The VPS firewall allows 30500 from the deploy machine's IP only.

Check what's in the registry:

bash# List repos
curl http://91.184.248.13:30500/v2/_catalog

# List tags for a repo
curl http://91.184.248.13:30500/v2/antonnovikov-site/tags/list

Configuring Docker to trust the registry

Since port 30500 is plain HTTP (no TLS), Docker needs to be told to allow it as an insecure registry.

On the local machine (macOS Docker Desktop):

json// ~/.docker/daemon.json
{
  "insecure-registries": ["91.184.248.13:30500"]
}

Or via Docker Desktop → Settings → Docker Engine → add to the JSON.

On the k0s worker (same node), containerd needs the same allowance. k0s handles this automatically when you specify insecureRegistries in the k0s config:

yaml# /etc/k0s/k0s.yaml
spec:
  images:
    defaultPullPolicy: IfNotPresent
  extensions:
    helm: ...
# In containerd config section:
  containerd:
    config: |
      [plugins."io.containerd.grpc.v1.cri".registry.mirrors."91.184.248.13:30500"]
        endpoint = ["http://91.184.248.13:30500"]

Or directly in /etc/containerd/config.toml if managing containerd separately.

Cleaning up old images

After every successful deploy, 06-deploy-site.sh queries the registry API to delete all tags except the current one:

bashREGISTRY="91.184.248.13:30500"
IMAGE_NAME="antonnovikov-site"
IMAGE_TAG="$COMMIT_SHA"

TAGS=$(curl -s "http://${REGISTRY}/v2/${IMAGE_NAME}/tags/list" | jq -r '.tags[]?')

for tag in $TAGS; do
    if [ "$tag" != "$IMAGE_TAG" ]; then
        DIGEST=$(curl -s -I \
            -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
            "http://${REGISTRY}/v2/${IMAGE_NAME}/manifests/${tag}" \
            | grep -i "Docker-Content-Digest" | awk '{print $2}' | tr -d '\r')

        if [ -n "$DIGEST" ]; then
            curl -s -X DELETE \
                "http://${REGISTRY}/v2/${IMAGE_NAME}/manifests/${DIGEST}" \
                && echo "Deleted $tag ($DIGEST)" || echo "Failed to delete $tag"
        fi
    fi
done

Note: the registry's delete API only soft-deletes manifest references. To reclaim disk space, run the garbage collector:

bash# Find the registry pod
REGISTRY_POD=$(kubectl get pods -n registry -l app=registry -o name | head -1)

# Run GC
kubectl exec -n registry $REGISTRY_POD -- \
    registry garbage-collect /etc/docker/registry/config.yml

For automation, add this as a CronJob after deploy:

yamlapiVersion: batch/v1
kind: CronJob
metadata:
  name: registry-gc
  namespace: registry
spec:
  schedule: "0 3 * * *"  # 3am daily
  jobTemplate:
    spec:
      template:
        spec:
          containers:
            - name: gc
              image: registry:2
              command:
                - registry
                - garbage-collect
                - /etc/docker/registry/config.yml
              volumeMounts:
                - mountPath: /var/lib/registry
                  name: data
          volumes:
            - name: data
              hostPath:
                path: /opt/registry-data
          restartPolicy: OnFailure

Security considerations

This setup is HTTP (no TLS), which is acceptable because:

Port 30500 is firewalled to the deploy machine's IP only.
Images never traverse the public internet on push.
The registry has no authentication — adding auth would require TLS.

For a multi-developer setup, add TLS via cert-manager and basic auth:

yamlenv:
  - name: REGISTRY_AUTH
    value: htpasswd
  - name: REGISTRY_AUTH_HTPASSWD_REALM
    value: "Registry Realm"
  - name: REGISTRY_AUTH_HTPASSWD_PATH
    value: /auth/htpasswd
  - name: REGISTRY_HTTP_TLS_CERTIFICATE
    value: /certs/tls.crt
  - name: REGISTRY_HTTP_TLS_KEY
    value: /certs/tls.key

Disk usage monitoring

bash# Check total registry data size
du -sh /opt/registry-data/

# Watch disk usage on the node
df -h /opt/

A Prometheus alert on high disk usage on the registry hostPath mount is worth adding to avoid a full disk killing registry pods.

What can go wrong

imagePullBackOff after registry pod restart. If the registry pod restarts and its data PVC is healthy, pulls will resume. But if hostPath data is gone (node replaced or path changed), cached layers are lost. Pods that reference images already pulled to nodes will keep running; new pods won't start. Always back up /opt/registry-data/ before node maintenance.

Registry disk full, all pushes fail. Registry v2 doesn't auto-prune. Run registry garbage-collect to remove unreferenced layers: kubectl exec -n registry deploy/registry -- /bin/registry garbage-collect /etc/docker/registry/config.yml. Add a Prometheus alert on the hostPath mount's disk usage.

Push fails with "unauthorized". The registry is protected by htpasswd basic auth. Ensure the client is logged in: docker login <registry-host>:<port>. Check the htpasswd secret in Kubernetes matches the one used to generate credentials.

Summary

Docker Registry v2 runs as a pod in k0s with a hostPath volume at /opt/registry-data
TLS is terminated by Traefik; the registry gets a Let's Encrypt certificate via the shared wildcard cert
htpasswd-based basic auth stored as a Kubernetes Secret — no external auth service needed
Registry v2 has no auto-pruning: run registry garbage-collect periodically or disk fills up
All images in the setup use a single registry hostname; no DockerHub, no GHCR, no ECR

weblog