SRE Assessment — IaC, DR, Upgrades, SLO/SLA

Assessed 2026-10-29 by four read-only audits (IaC, disaster recovery, upgrades, SLO/SLI) against the live mini and pro clusters, the git trees, and the health-suite reports. CRC was not started for this assessment; everything CRC-side is git-defined or last-known and is labelled as such. Every number on this page is labelled measured (observed live or in a dated record), inferred (derived from measured data), or unproven (no evidence found — a restore that was never drilled is stated as never drilled, not estimated away).

Headlines:

• IaC: 53 of 88 audited units are fully git-defined — 60% (measured for mini/pro, inferred for CRC). Counting content that is in git but hand-delivered, 86% (same basis). The debt is delivery automation and host/DB state, not missing manifests.
• DR: the rebuild-from-git story is drill-proven where exercised (CRC 27m24s, Vault milestones, pro standups). The worst gap is concentrated: the Vault unseal key exists only on the mini — losing that host makes the off-host Vault archives undecryptable. Harbor and StackRox Central, both promoted to pro on 2026-10-29, have no backup at all.
• Upgrades: the most overdue item is not an upgrade but a migration — ingress-nginx v1.15.1 is the final release of a retired project (repo archived 2026-03-24) with a critical unpatched CVE, and it is the mini’s front door. Behind it: Prometheus v2.55.1 (security support ended 2024-12-03), Logstash 8.17.3 (minor EOL 2025-08-12), Vault 2.0.4 (predates the October 2026 security-fix line), and Docker Desktop 4.96.3 on the mini (one release behind a credential-in-logs security fix).
• SLOs: of the proposed SLOs below — 4 MET, 6 MISSED, 5 UNMEASURABLE with today’s measurement. The misses are mostly measurement defects (CRC stopped, inconsistent infra-down gating), but two are real: the LDAP bind matrix at 93.8% and the front-door sweep at 92.9%.

This page supersedes the placement details in the Disaster Recovery chapter (verified 2026-10-24, pre-migration: Harbor, the observability trio, and the logging stack have since moved to pro).

1. IaC coverage

Denominator: 88 units = 48 live Argo CD Applications (measured) on the mini’s Argo CD (29 destined for the mini, 19 for pro) + ~28 git-defined CRC Applications (inferred count; CRC stopped) + 12 named non-Argo platform components. At audit time the mini’s Argo CD showed 48 Applications, 47 Synced/Healthy, 1 OutOfSync/Healthy (pro-harbor) (measured).

Exceptions and drift

Cheapest wins: merge + run the Harbor LDAP script (#5); write one idempotent pro Serve assertion script (#6); LaunchAgent-or-retire pf-ldap (#8).

2. Disaster recovery

RTO/RPO are inferred unless a drill measured them. “Last proven restore” means evidence — a dated drill or restore record — not a belief.

Host-loss blast radius

• Mini dies — the big one. Vault (primary and its unseal key), OpenLDAP (the IdP behind Dex, ACS SSO, Harbor LDAP, CRC OAuth), Argo CD (control plane for all three clusters), the front door + cert-manager, the retained Harbor rollback PVCs, lab-docs. Pro’s workloads keep running but unmanaged and unhealed. The Spark VM’s Vault archive copy survives as sealed ciphertext with no key anywhere off the dead host — every Vault-escrowed rebuild step (CRC pull secret, Argo repo creds, Harbor/Grafana admin, pro cluster token) then fails and must be re-created from original sources, several of which are Vault itself.
• Pro dies — painful, survivable. StackRox Central (ACS dark for the mini + CRC sensors), Harbor (now the primary registry — a CRC recover.sh run cannot complete its image pulls without it), pro logging, the observability trio, the agentops platform, pro CI runners. The mini plane (Vault, LDAP, Argo) survives, which is what makes this recoverable. Accepted losses: Harbor and Central return empty; logs/metrics gone by design. Full sequence inferred at ~half a day, dominated by image re-pushes — never run end-to-end as a drill (unproven as a whole); every individual app was proven at standup.

3. Upgrade posture

Current versions are measured live unless noted. “Latest” claims are vendor/endoflife.date figures checked 2026-10-29/30; where a latest could not be verified it says unproven rather than guessing.

Cross-cutting finding (inferred): everything stood up in October 2026 is within 1–3 patches of latest. Everything inherited from the original mini observability manifests (Prometheus 2.55.1, Grafana 11.3.1, AlertManager 0.27.0, Logstash 8.17.3) was copied to pro unchanged during the migration — the migration modernised placement, not versions.

4. SLOs, error budgets, and the internal SLA

How the SLIs are computed today (measured, from the suite code)

• Auth suite (report.py): availability = passes / executed runs, infra-down excluded per target. Windows: 394 runs, 2026-10-15 → 2026-10-30 (15 days, not 30 — the “last 30 days” table is currently identical to all-runs).
• Service suite (service_report.py): passes / executed, infra-down and pending excluded; 152 runs over 5 days only. It applies a flat 99% to every check — treated as a real SLO, that would produce 17 simultaneous false “MISSED” verdicts.
• Critical inconsistency (measured + inferred): the auth suite gates CRC checks on the tunnel and marks a dead tunnel infra-down (excluded); the service suite calls the CRC API directly and records a refusal as fail. That one difference manufactures most of the 34–65% service numbers while CRC is intentionally stopped. Likewise, in-flight E2E runs graded fail instead of pending inflate the miss.

Definitions adopted by this assessment (proposed)

1. Two denominators, always. Conditional SLI (infra-down excluded) answers “does it work when its host is up?”; unconditional SLI (passes / all attempts) answers “could a user actually use it?”. Report both.
2. Declared-down exclusion — honestly. A check is excluded only if the target was in a pre-declared stopped/maintenance state (a declared_status.yaml committed before or at the stop; a third status declared-down, counted and reported separately). Retroactive edits are not allowed; undeclared down stays a failure in the unconditional SLI. CRC — stopped since 2026-10-26 by owner decision — is exactly this case: CRC SLOs apply only to declared-up windows. If declared-down exceeds 50% of a window, the verdict is UNMEASURABLE, not MET.
3. 99% everywhere is abandoned — unachievable for browser-through-tunnel checks, too lax for a negative security control, meaningless for a headroom gauge.
4. Minimum-n rule: no MET/MISSED verdict on <100 executed runs in the window or <14 days of declared-up data — UNMEASURABLE instead.

SLO table (targets over rolling 30 days)

Current values are measured over the windows that exist today — auth suite 15 days, service suite 5 days — not a full 30. The statuses below are the assessment’s baseline call on that data; rows below the minimum-n rule (front doors n=99, log delivery n=62, oauth-only n≈73, and every service-suite row on its 5-day window) are provisional until a full window accrues, and are marked as called on short windows rather than silently treated as settled.

Latency SLOs explicitly not proposed: all CRC browser flows (median 18–41 s is tunnel + Chromium + OAuth settle waits — a latency SLO there would optimise the test harness), the LDAP matrix median (12.6 s is SSH/kubectl exec overhead), the sweep median (a sum of sequential probes), E2E stage “durations” (observer runtime), and the Vault median (SSH overhead). The only user-meaningful latency numbers today are the Harbor API and, loosely, MCP roundtrip (2.6 s median, PDI-tainted).

Error budgets and burn policy (proposed)

30 days = 43,200 minutes; for an hourly synthetic, the probe count is the operative budget.

Internal SLA (proposed) — one-operator lab

RHELS Lab — Internal Service Commitments (draft, not a contract)

Who this covers. The lab owner (operator) commits to the app/platform teams and agents who build on the lab, and to the owner’s own projects. There is one operator. There is no on-call rotation, no NOC, and no vendor behind this.

What is committed (rolling 30 days, declared-up windows): Vault available & unsealed 99%; Harbor registry auth/API 99%; mini Argo CD login 99%; *.rhels.com front doors 99% per vhost; LDAP binds 99%. CRC/OpenShift platform, when declared up: console/Argo/Grafana auth 95%. MCP roundtrip 95% (PDI-dependent); dual-Loki log delivery 95%. Security controls (wrong-password rejection, oauth-only posture, signed-image policy) are not percentage commitments: a failure is an incident, full stop.

Support reality. Best-effort during the operator’s waking hours, America/Chicago. Alerts route to GitHub issues via the alert bridge — notification, not a guaranteed page. Target response: same day for Vault/Harbor/LDAP/front-door incidents; next 1–2 days for CRC, MCP, pipeline, and log-delivery issues. No response-time guarantee outside waking hours, weekends, or the operator’s absence.

Exclusions — these do not count against the commitments:

1. Declared stops & maintenance — e.g. CRC stopped by owner decision since 2026-10-26; the chess-time CRC auto-stop (Mon/Wed evenings); any stop declared in advance in the lab repo.
2. Host hardware / host OS — host failure, sleep, or power; Docker Desktop failure on the host.
3. Connectivity the lab doesn’t control — home ISP (including its inbound 80/443 block), Tailscale/tailnet, the egress proxy, power at the house.
4. External services — ServiceNow PDI (incl. its sleep/wake), Grafana Cloud, Let’s Encrypt, GitHub Actions availability.
5. Tenant/demo workloads — the app1001-series and namespace-request experiments; reported, not masked, and not covered.

What “support” does not include. Data recovery beyond the existing Vault backup regime; guaranteed E2E pipeline turnaround; any availability claim for a service with no SLI.

Review. SLOs, budgets, and this SLA are reviewed when (a) a budget is exhausted, (b) CRC’s declared status changes, or (c) 30 days of clean declared-up data exists — whichever comes first. Changes are owner-approved and committed to the lab repo.

CI runners, as a commitment: best-effort with queue transparency — the proposed 95%-of-jobs-start-≤5-min objective becomes a commitment only once queue wait is actually measured (today it is not recorded at all).

Measurement gaps — cheapest fix first

1. Declared-status record (declared_status.yaml + a third status in both suites). Blocks every honest CRC verdict. Do first — it unblocks half the table.
2. Consistent infra-down gating in the service suite — preflight the CRC API once, as the auth suite preflights the tunnel.
3. E2E run-level composite SLI — join stages by the run id already on disk; in-flight runs count pending until the 90-minute timeout (this also fixes the known observer defect where mid-cycle runs grade FAIL).
4. Harbor push/pull success from real CI — report job-level step conclusions from the Actions API; login at 99.4% does not prove pulls work.
5. CI runner queue wait — the Actions API timestamps on runs the trigger already dispatches.
6. Cert headroom as a number — the check already fetches every cert’s expiry; emit days-left and aggregate min/p25. Report-side only.
7. Front-door second vantage + per-vhost rows — the probe code already exists; run it from a second host.
8. Real LDAP bind latency (on-mini) — time the bind calls inside the existing on-mini script instead of timing the SSH around them.
9. Per-sink log-delivery SLI + delivery lag — the check already queries both Lokis separately; record per-sink results.
10. Vault API latency + backup age as separate SLIs — emit backup age in days; tighten the freshness threshold from 14 d to 2 d.
11. Argo census split (mini vs CRC) — the data is already fetched per cluster; mini GitOps health currently has no clean trend.
12. Step-level p95 for Harbor ldap_login — the composite p95 is dominated by the deliberate wrong-password step (2 s median), masking real login-latency drift.

5. Where the fixes live

The gaps above are tracked as a checklist backlog in rhels/lab-monorepo (label spark): Vault unseal-key escrow decision, Harbor + StackRox backups, the ingress-nginx migration, Prometheus 3.13 LTS, Logstash 8.19.x, the Vault security line, Docker Desktop 4.97.0 on the mini, the mini root app + fork fix, the pro Serve assertion script, pf-ldap, the mini-Argo and LDAP-seed rebuild drills, Vault daily restore drills, a GitHub mirror/export, classic-ARC retirement, the OpenLDAP image migration, the Vector bump before the next CRC start, observability version alignment on pro, the stale DR/CRC-GitOps docs, and the twelve measurement gaps. Nothing on this page was changed silently — assessment and definitions only.