Vault & Secrets Delivery

HashiCorp Vault is the lab’s secrets store and, since the 2026-10-24 migration, the single source of truth for lab credentials: 59 KV entries hold everything from the Harbor admin password to the Grafana Cloud tokens. Git holds paths and policies, never values.

Part A — How it is set up

Deployment

• Where: namespace vault on the tooling cluster — Vault 2.0.4, standalone (single node), installed from the platform repo’s Helm values. It is deliberately not Argo-managed; a rebuild is a manual, documented procedure.
• Storage: file storage on a 5 GiB persistent volume with a retain-on-delete policy both ways — deleting the release does not delete the secrets.
• TLS: the listener is TLS-only and, since the 2026-10-25 cutover, serves a Let’s Encrypt certificate for vault.rhels.com mounted from a cluster Secret; clients verify it against public roots — nothing pins the old internal CA anymore. The UI/API are reached through the lab front door as vault.rhels.com (see certificates.md).

Seal model — read this before a reboot surprises you

Vault seals at every tooling-host boot. Sealed means running but useless: every read fails until the owner unseals it.

• The unseal key and the root token exist only in the owner’s macOS Keychain. No agent ever sees the unseal key.
• After a reboot, the owner runs a bootstrap script on the tooling host that pulls the key from the Keychain and unseals. Until then, every Vault-dependent flow (including secrets sync into CRC) is paused — that is the design, not an outage. Runbook 11 covers the sequence.
• Agent write gate: since 2026-10-24 the owner also places the root token as a mode-600 file on the tooling host. Agents perform Vault writes by running the Vault CLI inside the Vault pod with that token. The token file is a standing delegation — it exists because the owner put it there, and the owner can revoke the gate by removing it.

What lives where

One KV-v2 mount, secret, in two trees:

• secret/platform/… — the two original proof tenants (vault-proof-app1, vault-proof-app2).
• secret/lab/… — the working tree, grouped by consumer: agents, apps, argocd, cloudflare, cursor, github, google, google-workspace, grafana, grafana-cloud, harbor, health-suite, ldap, memory, openshift, servicenow, ssh, tailscale.

Docs and issues may name paths freely; values never leave Vault.

Delivery into clusters: the VSO tenant pattern

CRC runs the Vault Secrets Operator (Argo CD app crc-vault-secrets-operator) with one VaultConnection named default pointing back at the tooling-plane Vault. Each consumer is a tenant bundle in the GitOps repo:

1. a dedicated service account in the consumer’s namespace, bound so Vault can review its token;
2. a VaultAuth (Kubernetes auth method, one Vault role per tenant);
3. a VaultStaticSecret naming exactly one KV path;
4. on the Vault side, a policy and auth role scoped to that one path — a tenant can read its own secret and nothing else.

Current tenants:

2026-11-01: CRC runs VSO 1.6.2, upgraded from 1.5.1 to fix the stale SecretSynced condition bug (upstream #1323). After the upgrade, 18/18 VaultStaticSecrets synced; GitOps PR #128 merged and lab-monorepo#104 closed. This supersedes the previous undiagnosed stale proof-tenant status note.

Backup & restore (summary)

Vault’s file-storage volume is the lab’s one irreplaceable state, so it is the one thing with a backup discipline:

• The whole state is a directory tree — portable as a tar archive, restorable into a same-version Vault by untarring into the data path while stopped, then unsealing with the owner’s key.
• Archives live on the tooling host under dated backup folders, with an off-box copy of the current archive on the agent VM (checksum-verified identical on both sides).
• Current archive: 2026-10-24-postmigration (215 files, taken after the migration below); its restore drill passed the same day — the restored tree diffed 215/215 identical, and a throwaway Vault booted on it sealed-and-healthy. An unsealed read of the new archive was not part of the drill (no agent holds the key); the full-unseal proof rests on the identical October 18 drill. The rule: a backup counts only with a same-day drill, and a migration-scale change is not “backed up” until a fresh archive says so. The full classification of every volume in both clusters is disaster-recovery.md.

Let’s Encrypt cutover — completed 2026-10-25

Staged 2026-10-24, applied 2026-10-25 on the owner’s word. A Let’s Encrypt certificate for vault.rhels.com was issued into Secret vault-server-le-tls (Certificate of the same name, in local-k8s-gitops clusters/docker-desktop/rhels-ingress/vault-certificate.yaml, mirroring the LDAP pattern; issuer Let’s Encrypt, valid 2026-10-24 → 2027-01-22, full chain in tls.crt, no ca.crt key). The four Helm values edits recorded in clusters/docker-desktop/vault/values.yaml were applied by manual helm upgrade (the release’s StatefulSet uses the OnDelete strategy, so vault-0 was deleted by hand to roll it); the owner had authorized the unseal in advance, and Vault was unsealed from the key file the owner placed on the tooling host. The listener now presents the LE chain on :8200 (verified against public roots, name-checked), and every consumer was repointed in the same window: the CRC VSO default connection uses https://vault.rhels.com with no CA pin, and reconcile.py / bootstrap_vault.py default to the same address with public-root verification (reconcile.py verify-auth passes for both proof tenants).

Two operational notes from the execution:

• The in-pod Vault CLI dials the loopback address, which the DNS-only LE certificate cannot validate — ~/bin/vx therefore sets VAULT_TLS_SERVER_NAME=vault.rhels.com for its in-pod calls.
• After the connection’s address changed, the pre-existing VaultStaticSecrets kept a stale client binding and kept reporting sync errors against the old address (an operator restart did not clear it). Deleting each VaultStaticSecret and letting Argo CD recreate it re-registered its client; all 13 then reported SYNCED. The destination Secrets are not touched by the deletion.

Who consumed the endpoint before the cutover (the mini’s tailnet HTTPS endpoint, port 8200, internal CA pinned) — the repoint targets for the checklist below:

Cutover order (owner-gated):

0. Preconditions + addressing decision. Owner present; a fresh Vault backup archive exists; Certificate vault-server-le-tls Ready. The LE cert is valid only for vault.rhels.com, so every consumer must stop addressing the tailnet name in the same window. Default plan: consumers move to https://vault.rhels.com (the :443 front door). A :8200-direct variant additionally needs DNS/LB wiring for that name to the Vault Service (the LDAP pattern) — a separate decision, not an improvisation.
1. Apply the STAGED values edit (four lines) and helm upgrade manually. vault-0 restarts; Vault comes up sealed.
2. Owner unseals (runbook 11).
3. Verify in-pod: a vx read of a known path; the listener presents the LE chain for vault.rhels.com.
4. Repoint CRC VSO in git (connection.yaml): new address, drop caCertSecretRef (public roots now verify). Argo syncs; watch every VaultStaticSecret return to SYNCED (the proof tenants’ pre-existing quirk in Part A is the baseline, not a regression).
5. Repoint the scripts: VAULT_ADDR=https://vault.rhels.com and stop pinning the internal CA in reconcile.py / bootstrap_vault.py; run reconcile.py verify-auth.
6. Prove end to end: trigger the CRC login-tests and delivery CronJobs (both consume VSO-delivered credentials) and run the health suite.
7. Rollback is the same window run in reverse: restore the four values lines, helm upgrade, owner unseals, revert the consumer edits. The internal bundle (tls.yaml) is never modified, so rollback needs no certificate work.

Part B — How to use it

• After any tooling-host reboot: Vault is sealed and every Vault-dependent flow is paused until the owner unseals it (runbook 11). Agents do not unseal; report the sealed state and wait.
• Read or write a secret (agents): on the tooling host, use the Vault CLI inside the Vault pod with the owner-placed token file (the agent write gate in Part A). Read the path you need; when writing, create a new KV version — never edit history, never echo a value into a log, an issue, or these docs. Paths may be named freely; values never leave Vault.
• Give a workload a secret: do not hand it a token. Add a VSO tenant bundle (Part A) — service account, VaultAuth, VaultStaticSecret, and a Vault policy scoped to that one path — in the consumer’s GitOps, and let the operator deliver a native Kubernetes Secret. Check delivery by the VaultStaticSecret’s SYNCED condition and by comparing the delivered Secret’s content hash, never by printing it.
• Rotate a credential: write the new value to Vault first, then update the consumer’s other copy (an Actions secret, a fixture env file) from Vault — Vault is the source the copies are cut from. Runbook 6 is the sequence. The Harbor admin rotation (2026-10-24) is the worked example.
• Humans: the UI/API are at vault.rhels.com (token auth). The unseal key stays in the owner’s Keychain — by the owner’s decision (2026-10-24) there is deliberately no offline escrow copy; losing the Keychain copy means losing Vault, and the backup archives are encrypted-by-seal against exactly that key.

The 2026-10-24 migration — state

A 54-item inventory of every lab credential was migrated into the layout above. Closed by that work:

• Harbor’s admin password was rotated off the default that had been committed to git; the GitOps repo was scrubbed (history grep for the old value: zero), and CI now reads its Harbor password from a GitHub Actions secret.
• A committed literal in an example repository-credential file was replaced with an obvious placeholder.
• Grafana Cloud access was re-cut into scoped tokens (write-only for the log sink, read-only for verification) — see observability.md.

Closed after the migration, same day:

• The on-disk Google credential export files that Vault duplicated were shredded on 2026-10-24, after digest-verifying each one against its Vault copy (five files). The two live working copies the tooling still uses were kept — they are the tool’s own config, and they too are escrowed in Vault.

Still open, all awaiting an owner decision rather than engineering:

• Two credentials exist in divergent tooling-plane/CRC variants (ServiceNow token, agentops GitHub token — both variants are live and both are stored); the owner picks the canonical one.
• One Google OAuth client has two different secrets on disk; Vault holds both as successive versions and the owner confirms the live one in the Google console.
• The tooling-plane Grafana’s admin credential exists only inside Grafana itself — not yet recorded in Vault.
• A retired Grafana Cloud push token awaits revocation in the UI (its value is unrecoverable, so revocation is the only cleanup).