Disaster Recovery & Rebuild

The lab’s doctrine is rebuild over repair: both clusters are treated as disposable, and git is the source of truth. This chapter is the honest version of that doctrine — what a cold rebuild actually takes, which state cannot come back from git, and where the manual seams are. Verified against live state on 2026-10-24.

The stateful inventory

Everything else in both clusters is stateless and reconciles from git. The complete persistent-volume list, classified:

Tooling plane (Mac mini, Docker Desktop Kubernetes)

The agentops observability stack (Prometheus/Grafana/Alertmanager) has no volumes at all — its state is in-pod by construction, its configuration in git.

Workload cluster (CRC)

The image inventory — what a rebuild cannot pull

A rebuilt cluster re-pulls almost everything from public registries. The exceptions below (enumerated from every live pod and every GitOps manifest on 2026-10-24) are the images whose reference does not resolve publicly; each needs its build path run once before its workload can start.

Tooling plane (Mac mini):

Workload cluster (CRC):

Everything else in both clusters resolves from a public registry (quay.io, docker.io, registry.k8s.io, ghcr.io, the Red Hat registries, MCR) and needs no local action.

The one true backup: Vault

Because Vault’s volume is the single irreplaceable, it is the single thing with a backup procedure:

• The file-storage backend makes Vault’s whole state a directory tree — portable as a tar archive, restorable into a same-version Vault by untarring into the data path while stopped, then unsealing with the owner’s key.
• Archives live on the tooling host, outside the cluster, under ~/vault-backups/ (dated folders), with an off-box copy of the current archive kept on the agent VM (checksum-verified identical).
• Automated daily backup (since 2026-10-24): a scheduled job (vault-backup-daily, ~02:18 America/Chicago) runs ~/workspace/vault-backups/vault-backup.sh, which takes the same tar archive every day. The *-daily archives are kept 14 days on both the tooling host and the agent VM, sha256-verified identical, so the worst case is losing one day of Vault changes. The first daily archive (2026-10-24-daily, 219 files) passed the same structural restore drill as the milestone archives — restored tree identical (219/219) and a throwaway Vault booted on it to Sealed=true. The daily job reports only on failure.
• Status (2026-10-24): the current milestone archive is 2026-10-24-postmigration, taken after the secrets migration (215 files), and its restore drill passed the same day — the restored tree diffed file-for-file identical to the live tree, 215/215. The previous archive (2026-10-18-step1, 62 files) passed the same drill on its day. The discipline this implies: a backup counts only with a same-day restore drill, and any change on the scale of the 2026-10-24 migration is not “backed up” until a fresh archive + drill says so.

Cold rebuild order

From bare hosts. Steps marked [manual] are the seams — everything else is root-app sync.

1. [manual] Hosts. Tooling host with Docker Desktop Kubernetes enabled; the CRC host with the CRC bundle set up; both on the tailnet. No cluster exists yet, so nothing in git can do this.
2. [manual] Tooling-plane Argo CD. First install of Argo CD, then apply the root application (argocd/applications/root.yaml in rhels/local-k8s-gitops). From here the plane self-assembles: foundation, cert-manager, ingress, Tailscale operator, runners. Two owner-placed secrets gate full sync: the Cloudflare API token (cert-manager namespace) and the Tailscale operator’s credentials.
3. [manual] Vault. Helm-install from the platform repo’s values (clusters/docker-desktop/vault/) — it is deliberately not an Argo app. Restore the backup archive into the data volume (or initialize empty), then the owner unseals from the Keychain (see vault.md). Then [manual] run vault-access/scripts/reconcile.py from the platform repo with Vault admin access: it ensures the KV mounts and reconciles every namespace-scoped Vault policy and Kubernetes-auth role, seeds the proof tenants, and verifies the CRC auth path end to end. The script is in git; running it is a human/agent step — Argo CD cannot.
4. Identity. The agentops-ldap app syncs from rhels/lab-agentops; the directory seeds from its module. vault.md records where its credentials of record live, and identity.md Part A documents exactly what the seed restores. Since the 2026-10-24 codification the seed restores the full directory as it stands — every user (including the health-suite test user), group, and membership. The one deliberate exception is the app1002-prod group, which is intentionally unseeded (a test fixture) and arrives through the ldap-groups/ request path instead.
5. Harbor. The tooling-harbor app installs the chart once two manual pieces exist: [manual] the pre-created harbor-admin Secret (value taken from Vault), and [manual] Harbor’s LDAP auth mode, which is Harbor-internal configuration set through its own API/UI — the chart values do not carry it. The exact values to re-enter are collected as a rebuild checklist in tools/harbor.md.
6. CRC. [manual] Create the VM (crc start; the pull secret of record is in Vault), install the OpenShift GitOps operator, and apply the bootstrap root application (clusters/crc/openshift-gitops/bootstrap/). The cluster then self-assembles: Secrets Operator and the vault-access tenants (step 3 must be done — tenants cannot sync against a sealed or unreconciled Vault), Grafana, the Prometheus stack, Kyverno, the logging pipeline, the agent platform, the demo apps. For the routine delete-and-rebuild drill this whole step is scripted — see “CRC recovery: the timed procedure” below.
7. Front doors and verification. The rhels-ingress apps publish the service names once cert-manager and the Tailscale operator are up (see certificates.md). Close out with the health suite (runbook 12’s delivery check plus the login checks): a rebuild is done when the suite is green, not when the syncs are.

CRC recovery: the timed procedure

Step 6 above, expanded into the drill that is actually run. CRC is deleted and rebuilt on purpose, repeatedly, to keep the recovery time measured and shrinking. The executable form is clusters/crc/openshift-gitops/bootstrap/recover.sh in rhels/local-k8s-gitops — one command from the agent VM, and it needs nothing from the owner: the pull secret comes from Vault, Vault unseals through the documented Keychain path if it is sealed, and every other credential is placed from Vault by the script or by a VaultStaticSecret. The script prints a per-phase timing table; the phases and their gates:

Two bootstrap details the first drills taught, both now scripted: the logging app’s controller Role in openshift-logging is applied once by the script (Argo cannot grant its own rights in a namespace it has none in yet; the Role itself lives in git at clusters/crc/logging/operator/argocd-manager-rbac.yaml), and the Grafana OAuthClient is re-applied by a PostSync Job from the Vault-delivered secret (an OAuthClient carries its secret as a literal field, so the object itself can never be committed).

What the 2026-10-24 optimization pass closed

The first timed baseline (below) exposed four self-inflicted delays; all four are fixed in git, so a rebuild no longer meets them:

• Images: the log-producer is pulled from Harbor like everything else (see the image inventory); no build happens during recovery.
• The Prometheus operator’s admission secret was created by hand; when a rebuild missed it, the operator sat in ContainerCreating for 43 minutes. A self-healing Job in clusters/crc/observability/ now creates the self-signed secret iff absent, before the operator starts.
• UID pins: namespace UID ranges are re-allocated on every rebuild, and the monitoring/kyverno pins in git went stale — two manual mid-drill fix commits. fix-uid-pins.sh (phase 7) makes it one scripted step.
• Hand-applied objects are GitOps now: the Cluster Logging operator + ClusterLogForwarder, the cluster OAuth object (its two IdP secrets arrive from Vault — the LDAP bind secret is renamed to the bindPassword key by a Vault Secrets Operator transformation), and two agentops secrets (agentops-snow-token, agentops-github) that a rebuild silently lost.

Acceptance battery

A recovery is done when all of these hold, not when the syncs look quiet:

• Every Argo CD Application Synced and Healthy.
• Every VaultStaticSecret reports secretSynced: True.
• The Kyverno verifier gate: the harbor-verifier VaultStaticSecret in the kyverno namespace is synced, and the private-Harbor verify proof is live in app000-health PolicyReports — signed image PASS, unsigned control FAIL (“no signatures found”). (The credential itself is mini-plane state outside git; phase 8b is what puts it back.)
• Both group-sync CronJobs (group-sync, openshift-group-sync) run green as manually triggered Jobs.
• The lab health suite (in rhels/lab-agentops, health-suite/) runs green against the rebuilt cluster.

Drill log

What “GitOps” does not cover — the seam list, collected

• Cluster creation (both planes) and host setup.
• Argo CD’s own first install on each cluster, and applying each root application once.
• Vault: install, unseal, and the reconcile script run.
• Owner-placed secrets that other things consume: Cloudflare token, Tailscale operator credentials, harbor-admin.
• Harbor’s internal LDAP auth configuration.
• The CRC tunnel into the agent environment (its service definition lives in the agent VM’s own config repo; the tunnel’s SSH key is recorded in Vault).
• CRC rebuild seams closed by scripting on 2026-10-24 (covered by recover.sh, listed here so they are not rediscovered as gaps): the 11 Argo repository secrets (regenerated from the Vault escrow), the Grafana OAuthClient (PostSync Job from the Vault-delivered secret), and the logging app’s controller Role (applied once at bootstrap; Argo cannot grant its own rights).

Everything on this list is a known, accepted seam — the failure mode to avoid is discovering one mid-incident, which is why they are written here.