Wildcard TLS with ACME and cert-manager: Automating Let's Encrypt via Yandex Cloud DNS

Published: 2026-06-17

A wildcard certificate covers every subdomain under a single domain — *.dev.antonnovikov.com terminates TLS for prometheus.dev.antonnovikov.com, kibana.dev.antonnovikov.com, and any other service without a per-route certificate. Getting and renewing wildcard certs requires DNS-01 challenge — no HTTP endpoint involved, only a DNS TXT record written via API. This post covers how ACME and DNS-01 work, how cert-manager automates the full lifecycle in Kubernetes, and how to wire it into a FluxCD hub-and-spoke GitOps setup using Yandex Cloud DNS.

The problem: static certs in git

The naive setup works until it doesn't. Generate a wildcard cert manually, commit the .crt and .key files to the repository, copy them into a dozen namespaces via secretGenerator, mount in ingress. Repeat every 90 days.

In a real cluster with 10+ services and multiple environments this means:

Cert files sitting in git (plaintext or sealed — either way, renewed manually)
A calendar reminder to renew before expiry
Kustomize secretGenerator replicating one cert to 10–15 namespaces per environment
One missed renewal = TLS errors across the entire environment

The current dev environment setup in fluxcd/projects/dev/kustomization/spoke/secrets/kustomization.yaml has exactly this: the same wildcard cert duplicated into 15 secretGenerator blocks — one per namespace. Every cert rotation requires updating two files, committing them, and reconciling across all environments.

The fix is automating the full lifecycle: ACME challenge, cert issuance, renewal, and distribution across namespaces — without committing cert material to git at all.

How ACME works

ACME (Automatic Certificate Management Environment, RFC 8555) is the protocol Let's Encrypt uses to issue certificates without human involvement. The client (cert-manager) creates an account with the ACME server, requests a certificate, completes a challenge to prove domain ownership, and receives the signed cert.

Three challenge types exist:

HTTP-01 — the ACME server fetches http://<domain>/.well-known/acme-challenge/<token>. Requires port 80 reachable from the internet. Cannot be used for wildcard certs.

DNS-01 — the client creates a TXT record _acme-challenge.<domain> with a token value. The ACME server queries DNS to verify it. Works for wildcard certs. Requires DNS API access.

TLS-ALPN-01 — TLS handshake on port 443 with a special ALPN extension. Niche use case, not relevant here.

For *.dev.antonnovikov.com, DNS-01 is the only option. Let's Encrypt explicitly requires it for wildcard issuance — there is no workaround. The LE documentation states: "Wildcard certificates can only be issued via DNS-01 challenge type."

The full ACME flow for a wildcard:

cert-manager              Let's Encrypt             Yandex Cloud DNS
     │                         │                           │
     │── POST /new-order ───────▶                          │
     │◀── challenge: DNS-01 ───│                           │
     │                         │                           │
     │── create TXT record ────────────────────────────────▶
     │   _acme-challenge.dev.antonnovikov.com = <token>         │
     │                         │                           │
     │── POST /challenge ───────▶                          │
     │                         │── DNS query TXT ──────────▶
     │                         │◀── TXT value ─────────────│
     │◀── cert issued ─────────│                           │
     │                         │                           │
     │── delete TXT record ────────────────────────────────▶

Let's Encrypt certificates expire after 90 days. cert-manager renews them 30 days before expiry by default (configurable via spec.renewBefore). The renewal triggers a new ACME flow automatically — no human action required.

cert-manager: the ACME client for Kubernetes

cert-manager is the standard Kubernetes controller for certificate lifecycle management. It introduces three main CRDs:

ClusterIssuer — cluster-scoped configuration for a certificate authority. For ACME it holds the server URL, contact email, private key reference, and solver configuration.

Certificate — a request for a cert with specific DNS names, validity period, and key algorithm. References a ClusterIssuer and names a Secret where the TLS material will be stored.

CertificateRequest — a single issuance attempt, created internally by cert-manager when a Certificate needs to be issued or renewed.

When a Certificate resource is created:

cert-manager generates a private key, temporarily stores it
Creates a CSR and submits it to the ACME server via the ClusterIssuer
Completes the challenge (DNS-01: calls the webhook to create a TXT record)
Receives the signed cert, writes it to spec.secretName as tls.crt + tls.key
Monitors expiry, triggers renewal 30 days before

The Secret produced is a standard kubernetes.io/tls Secret — identical in structure to what secretGenerator produced from static files. No changes needed in APISIX, ApisixTls, or any ingress configuration.

DNS-01 with Yandex Cloud

cert-manager's DNS-01 support is extensible via webhooks. A webhook is a small server implementing the solver interface — cert-manager calls it to present and clean up DNS challenges. The webhook interacts with the DNS provider API.

For Yandex Cloud DNS the webhook (from yandex-cloud/cert-manager-webhook-yandex) requires:

A YC service account with the dns.editor role on the DNS zone
An authorized key (JSON format) stored as a Kubernetes Secret
The YC folder ID where the DNS zone lives

Step 1: Create service account and key

bashyc iam service-account create --name cert-manager-dns

FOLDER_ID=$(yc config get folder-id)

yc resource-manager folder add-access-binding \
  --id "$FOLDER_ID" \
  --role dns.editor \
  --service-account-name cert-manager-dns

yc iam key create \
  --service-account-name cert-manager-dns \
  --output /tmp/yc-key.json

Step 2: Seal the key and commit

bashkubectl create secret generic yc-dns-credentials \
  --from-file=authorized-key.json=/tmp/yc-key.json \
  --namespace cert-manager \
  --dry-run=client -o yaml > /tmp/yc-dns-secret.yaml

kubeseal --format yaml \
  --controller-namespace flux-system \
  --kubeconfig .kubeconfigs/dev.yaml \
  < /tmp/yc-dns-secret.yaml \
  > fluxcd/projects/dev/kustomization/spoke/secrets/yc-dns-credentials.yaml

rm /tmp/yc-key.json /tmp/yc-dns-secret.yaml

This sealed secret needs to be added to spoke/secrets/kustomization.yaml alongside the other secrets.

The ClusterIssuer referencing the YC webhook:

yamlapiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@antonnovikov.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - dns01:
          webhook:
            groupName: acme.cloud.yandex.com
            solverName: yandex-cloud-dns
            config:
              folder: <FOLDER_ID>
              serviceAccountSecretRef:
                name: yc-dns-credentials
                key: authorized-key.json

The Certificate for the wildcard:

yamlapiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: dev-wildcard
  namespace: ingress-apisix
spec:
  secretName: dev-wildcard
  dnsNames:
    - "*.dev.antonnovikov.com"
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  duration: 2160h   # 90 days
  renewBefore: 720h # renew 30 days before expiry

After cert-manager issues the cert, ingress-apisix/dev-wildcard exists as a standard TLS Secret — same name the existing ApisixTls in fluxcd/projects/dev/kustomization/routes/apisix-tls-tech.yaml references. No changes needed to the routes overlay.

Implementation in FluxCD

cert-manager deploys as a HelmRelease on each spoke. Since spokes are managed from the hub via kubeConfig, HelmReleases go in fluxcd/custom/apps/cert-manager/.

yaml# fluxcd/custom/apps/cert-manager/helmrelease.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: cert-manager
  namespace: cert-manager
spec:
  interval: 1h
  chart:
    spec:
      chart: cert-manager
      version: "v1.16.*"
      sourceRef:
        kind: HelmRepository
        name: jetstack
        namespace: flux-system
  values:
    crds:
      enabled: true
    dns01RecursiveNameservers: "8.8.8.8:53,1.1.1.1:53"
    dns01RecursiveNameserversOnly: true

dns01RecursiveNameserversOnly: true forces cert-manager to verify TXT records through the specified external resolvers instead of the cluster's internal DNS. Without this, cert-manager may query CoreDNS, which won't know about records in a public zone and validation fails intermittently.

The YC DNS webhook is a separate HelmRelease with a dependsOn on cert-manager:

yaml# fluxcd/custom/apps/cert-manager/webhook-yc.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: cert-manager-webhook-yandex
  namespace: cert-manager
spec:
  interval: 1h
  dependsOn:
    - name: cert-manager
  chart:
    spec:
      chart: cert-manager-webhook-yandex
      sourceRef:
        kind: HelmRepository
        name: cert-manager-webhook-yandex
        namespace: flux-system

Both HelmReleases target the cert-manager namespace, which must exist on the spoke. Add it to fluxcd/projects/{env}/kustomization/spoke/namespaces.yaml before deploying.

Add the required HelmRepositories to fluxcd/base/helmrepositories.yaml:

yaml---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: jetstack
  namespace: flux-system
spec:
  interval: 1h
  url: https://charts.jetstack.io
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: cert-manager-webhook-yandex
  namespace: flux-system
spec:
  interval: 1h
  url: https://yandex-cloud.github.io/cert-manager-webhook-yandex
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: emberstack
  namespace: flux-system
spec:
  interval: 1h
  url: https://emberstack.github.io/helm-charts

The ClusterIssuer and Certificate are spoke-level resources — they go in fluxcd/projects/{env}/kustomization/spoke/ and are applied to the spoke cluster via {env}-spoke Kustomization. They require cert-manager to be running, so the HelmRelease in {env}-apps must reconcile first. This is already handled by the existing dependsOn chain: {env}-spoke waits for nothing, and {env}-apps HelmReleases are independent — but cert-manager must be healthy before applying the ClusterIssuer. In practice, Flux will retry until cert-manager CRDs are available.

Distributing one cert to many namespaces

cert-manager creates the cert in one namespace (ingress-apisix). The current secretGenerator approach copies it to 15 namespaces. Replacing this requires a Secret replication mechanism.

kubernetes-reflector by EmberStack does exactly this: it watches annotated Secrets and mirrors them into specified namespaces automatically.

yaml# fluxcd/custom/apps/reflector/helmrelease.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: reflector
  namespace: kube-system
spec:
  interval: 1h
  chart:
    spec:
      chart: reflector
      version: ">=7.0.0"
      sourceRef:
        kind: HelmRepository
        name: emberstack
        namespace: flux-system

With reflector installed, annotate the Certificate to enable auto-replication:

yamlapiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: dev-wildcard
  namespace: ingress-apisix
spec:
  secretName: dev-wildcard
  secretTemplate:
    annotations:
      reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
      reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
      reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: >-
        kube-system,observability,headlamp,elasticsearch,kafka,minio,queue,db,redis,app,aurora,reportportal
  dnsNames:
    - "*.dev.antonnovikov.com"
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  duration: 2160h
  renewBefore: 720h

Reflector watches for these annotations and creates mirror Secrets in each listed namespace. When cert-manager renews the primary cert, reflector automatically updates all mirrors within seconds. No manual steps, no secretGenerator blocks, no cert files in git.

Migrating from static cert files

The existing secretGenerator entries in spoke/secrets/kustomization.yaml can be removed once cert-manager is issuing and reflector is distributing.

Migration sequence:

Add cert-manager namespace to spoke/namespaces.yaml
Include the cert-manager and reflector HelmReleases in the hub overlay for the target environment
Add the sealed yc-dns-credentials secret to spoke/secrets/kustomization.yaml
Deploy ClusterIssuer and Certificate resources to spoke/
Wait for the certificate to be issued:

bashkubectl get certificate -n ingress-apisix --context=dev-k8s
# NAME                       READY   SECRET             AGE
# dev-wildcard   True    dev-wildcard    2m

Verify reflector has created mirrors in all target namespaces:

bashkubectl get secret dev-wildcard -A --context=dev-k8s

Remove all secretGenerator blocks for dev-wildcard from spoke/secrets/kustomization.yaml
Delete dev-wildcard.crt and dev-wildcard.key from the repository
Commit and reconcile

The ApisixTls in routes/apisix-tls-tech.yaml does not need to change — it still references dev-wildcard in ingress-apisix.

What can go wrong

DNS propagation delay — DNS-01 requires the TXT record to be visible to Let's Encrypt's validating resolvers before verification succeeds. The dns01RecursiveNameservers setting ensures cert-manager also waits for the record to appear on public resolvers. If validation times out, cert-manager retries automatically. Check progress:

bashkubectl describe certificaterequest -n ingress-apisix --context=dev-k8s
kubectl logs -n cert-manager -l app=cert-manager --context=dev-k8s | grep -i "error\|challenge"

YC webhook auth failure — if the service account key is invalid or the dns.editor role is missing, the webhook returns an error and the challenge stalls. Check:

bashkubectl logs -n cert-manager -l app.kubernetes.io/name=cert-manager-webhook-yandex \
  --context=dev-k8s

Reflector not syncing — if a target namespace didn't exist when reflector first saw the annotation, the mirror is not created. After the namespace appears, reflector will pick it up on the next sync cycle (default: 1 minute). Can be forced by adding or toggling any annotation on the source Secret.

APISIX not picking up renewed cert — APISIX reloads TLS via the ApisixTls controller. Usually it picks up the updated Secret without a restart. If it doesn't:

bashkubectl rollout restart deployment apisix -n ingress-apisix --context=dev-k8s

Let's Encrypt rate limits — the production API limits to 5 duplicate certificates per domain per week. During initial setup always use the staging endpoint first:

yamlspec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory

Staging certs are not trusted by browsers but the issuance flow is identical. Switch to production only after confirming everything works end to end.

Summary

ACME automates the full TLS certificate lifecycle — no manual steps after initial wiring
Wildcard certs require DNS-01; HTTP-01 cannot be used for *.domain.com
cert-manager implements ACME natively in Kubernetes via ClusterIssuer and Certificate CRDs
The YC DNS webhook handles TXT record creation and cleanup via the YC DNS API
kubernetes-reflector distributes one cert to many namespaces via annotations — replaces secretGenerator with static files
No cert material is stored in git; everything lives as Kubernetes Secrets managed by cert-manager
Renewal happens automatically 30 days before expiry — no calendar reminders, no incidents from expired certs

weblog