Certificates, DNS & Front Doors

Every lab service has exactly one public name — a name on the lab’s own domain, rhels.com — and one certificate story: Let’s Encrypt, issued by cert-manager, via DNS-01. This chapter is the map of both.

Part A — How it is set up

The hard constraint

The lab sits behind a residential connection whose provider permanently blocks inbound ports 80 and 443. Consequences that are not negotiable and should never be re-litigated in a plan:

• HTTP-01 certificate validation can never work here.
• Nothing in the lab is reachable from the public internet by design; names resolve, but the addresses behind them are tailnet addresses, so only tailnet members can connect.

The issuer

• ClusterIssuer letsencrypt-prod on the tooling plane (GitOps-managed; adopted 2026-10-24 together with a DNS-01 self-check that verifies issuance against public resolvers).
• DNS-01 via Cloudflare, using a dedicated API token scoped to the lab’s zone (read + DNS write on that zone only). The token lives in a Secret in the cert-manager namespace — placed by the owner, never in git — and is also recorded in Vault (secret/lab/cloudflare/api-token).
• The token has an expiry (early January 2027); a renewal reminder is set. If lab certificates start failing to renew, check this token first.
• Renewal is otherwise automatic: cert-manager renews before expiry, and consumers pick up the new Secret on their normal restart/sync.

The front doors

The tooling plane’s services are fronted by the Tailscale operator and an ingress controller, giving each service its name:

DNS for these names is Cloudflare, DNS-only records pointing at tailnet addresses. The same zone also carries the tailnet’s own machine records, so lab names and machine names resolve in one place. Login mechanisms per service are listed in the showcase’s service directory; the identity chain behind them is identity.md.

The CRC side

• CRC routes keep the cluster’s own apps-crc.testing domain and the certs OpenShift serves for it; cert-manager does not run on CRC today. Giving CRC workloads real names is a researched, undecided step — see the custom-domain note in networking.md.
• Tailnet machine names use Tailscale-issued certificates where a service needs TLS under a machine name.
• Vault’s own listener certificate is lab-internal CA material mounted from a cluster Secret (see vault.md). Replacing ad-hoc internal CA material like this with a real internal CA is exactly the planned Vault PKI step — vault-pki.md.

Part B — How to use it

• Get a certificate for a new service: the normal path is the shared ingress — terminate at the ingress like the other front doors and no per-service Certificate is needed. A service that terminates its own TLS (an L4 service, or a pod that must present the cert itself) gets its own cert-manager Certificate in its namespace naming the letsencrypt-prod issuer and its front-door name — the LDAP Certificate (ldap-rhels-tls) is the working example. Either way the request lives in git.
• Check a certificate: kubectl get certificate -A on the tooling cluster shows Ready state and renewal time; kubectl describe certificate <name> -n <ns> shows the cert-manager events when one is stuck. A certificate that will not issue is almost always the DNS-01 path: check the Cloudflare token’s expiry (Part A) before debugging anything else.
• Renewals: automatic — cert-manager renews before expiry and consumers pick up the new Secret on their normal restart/sync. A consumer still serving the old cert after a renewal is a consumer that has not restarted; bounce it through its GitOps (rollout), not by hand-editing the Secret.
• Add a front door (a new rhels.com name for a service): the three-part pattern — ingress wiring in git, the Cloudflare DNS record, and the certificate path above — is written out in tools/tailscale-cloudflare.md Part B. Add the new name to the front-door table in Part A in the same change.