RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · QA Use Cases

QA Use Cases — CRC / OpenShift

Tool page: workload-cluster.md. CRC (OpenShift 4.22.7) runs on the m4 and hosts the workload plane: OpenShift GitOps, the agentops platform’s second instance, the service-health E2E surface. CRC is intentionally STOPPED at the time of writing — every story here is framed when CRC is running, and a stopped CRC makes each story not-run, never failed. Start discipline: the CRC-start checklist (goal record) governs, including its preconditions (not inside the Mon/Wed evening chess auto-stop window; mini side healthy first — CRC consumes the mini’s LDAP, Vault, and Harbor).

Admin stories

CRC-A1 — Cluster and console reachable, end to end [CLI]

  • Persona: Platform admin.
  • Steps: after a start, prove the tunnel path end to end: console health and API health through the tunnel — a bound port alone never counts (source: CRC-start checklist, “Tunnel proof end-to-end”).
  • Expected: console at console.crc.rhels.com and the API both answer healthy through the tunnel.
  • Assert: console health endpoint → 200; API /healthz → 200; both fetched through the tunnel address with proxy bypass, per the checklist’s probe form.

CRC-A2 — OAuth providers: LDAP primary, htpasswd break-glass [browser]

  • Persona: Platform admin.
  • Steps: open the console login; confirm the agentops-ldap provider is offered and works (the IDN-U1 bind’s console form); confirm the htpasswd developer account still exists as break-glass.
  • Expected: both providers present; developer is break-glass only — the daily path is LDAP (source: service-directory.md — console row; identity.md, break-glass paths).
  • Assert: console login page lists agentops-ldap; oc get oauth cluster providers ⊇ {agentops-ldap, htpasswd}; a developer login succeeds (used only by this check, on the admin’s word — the fixture cadence is per-start, not per-run; see Gaps).

CRC-A3 — OpenShift group-sync projection freshness [config/API]

  • Persona: Platform admin.
  • Steps: read the openshift-group-sync CronJob’s last run and the projected OpenShift groups.
  • Expected: the projection runs every 30 minutes (rfc2307) over the recorded 9-group whitelist, no prune; projected memberships equal the directory’s (source: lab record — periodic group sync delivery 2026-10-14; identity.md, “Projection into OpenShift”).
  • Assert: last successful run age <= 35m; OpenShift group set == the whitelist set; sampled group’s users == the LDAP group’s members.

CRC-A4 — OpenShift GitOps census [config/API]

  • Persona: Platform admin.
  • Steps: the ARG-A1 census, CRC edition: every Application in openshift-gitops.
  • Expected: all Synced/Healthy, including the platform apps (observability, logging, vault-access, security, agentops) (source: tools/argocd.md — CRC layout).
  • Assert: per ARG-A1: every item Synced + Healthy; the census names its exceptions.

CRC-A5 — Vault Kubernetes auth from CRC [config/API]

  • Persona: Platform admin.
  • Steps: confirm a CRC tenant’s VSO delivery is live (VLT-U2’s CRC leg) and the Vault auth role for CRC service accounts authenticates.
  • Expected: tenants SYNCED; the auth path the namespace chart wires (vault-auth ServiceAccount + VaultAuth, role convention <bsn>-role) works for a created namespace (source: vault.md, tenant pattern; namespace-service.md, step 8).
  • Assert: per VLT-U2 on the CRC side; a token issued via the k8s auth method for a tenant SA can read exactly its one path (VLT-U3’s negative applies here too).

CRC-A6 — The E2E cycle is the platform’s own story [config/API]

  • Persona: Platform admin (the 6-hourly trigger runs this persona unattended).
  • Steps: read the latest E2E run manifest and the observer’s stage table.
  • Expected: the cycle grades its stages — runner, build-sign, alert-delivery, alert-bridge, vault-secret, kyverno, readiness, logs, metrics, cert — and a completed run reaches 18/18 stage passes, as the proof cycles did (source: E2E design + observer record; proof runs 2026-10-25/26 graded 18/18).
  • Assert: latest completed manifest: every stage present with a verdict; known defect to assert around: an hourly grading that lands mid-cycle can mis-grade in-flight stages (lab-agentops#27) and step-B pushes can hit non-fast-forward (#29) — both are tracked defects, reported as such, not suite noise.

Status — 2026-11-01: latest cycle 2026-11-01 06:32Z, manifest e2e-20261101063208: ALL run-scoped stages PASS (observer runs 064509Z/074459Z). The in-flight grading pair fix is rhels/lab-agentops PR #70, still OPEN (not merged/fixed): V-E2E-01..05 PASS, V-E2E-06 NOT-RUN. The 06:32Z cycle was bracketed, not overlapped, by observer runs; a one-shot observation is scheduled for the 2026-11-01 12:32Z cycle. rhels/lab-agentops#28 remains OPEN; V-AD-01/02/03 unposted for the latest cycle.

CRC-A7 — Recovery is a drill, and the drill is timed (DR) [CLI]

  • Persona: Platform admin. Goal: the cluster’s disaster recovery is a proven procedure, not a hope.
  • Preconditions: an owner-authorized drill window — this story runs on drill days, not per-run.
  • Steps: execute recover.sh (in rhels/local-k8s-gitops) end to end: preflight (incl. Vault unseal via the sanctioned path), snapshot, delete, start, bootstrap, convergence, acceptance battery.
  • Expected: the full recovery completes at the proven scale — baseline ~95 minutes was drilled down to 27m24s (Drill 3, 2026-10-25) — with the acceptance battery green (source: disaster-recovery.md, the DR chapter’s CRC section; recover.sh record).
  • Assert: drill wall-clock recorded as the measured value (regression = materially above the 27–30m band); acceptance battery exit 0; the drill report is filed with the run’s records.

User stories

CRC-U1 — oc login as a directory user [CLI]

  • Persona: dev1.
  • Steps: oc login against the cluster API via the agentops-ldap provider (from the m4 — the recorded working path).
  • Expected: login succeeds as dev1 with dev1’s groups (the CRC-A3 projection made visible) (source: lab record — “oc login as dev1 works”).
  • Assert: oc whoami → dev1; oc groups contains dev1’s projected groups.

CRC-U2 — Work inside my own namespace [CLI]

  • Persona: dev1, in an MCP-created app1001 namespace (MCP-U1’s product).
  • Steps: deploy the dummy workload; read it back.
  • Expected: the workload runs; the namespace carries the BSN labels, the ResourceQuota from the request, and the app1001-nonprod RoleBinding with admin (non-prod) (source: namespace-service.md, step 8).
  • Assert: pod phase == Running; quota hard values == the request file’s; RoleBinding roleRef name == admin.

CRC-U3 — Other BSNs’ namespaces are closed (negative) [CLI]

  • Persona: dev1 reaching for another segment’s namespace.
  • Steps: attempt to read/patch resources in a namespace belonging to a BSN whose groups dev1 is not in.
  • Expected: refused by RBAC; production bindings, where they exist for dev1’s own groups, are view — never admin (source: namespace-service.md — “admin in non-production, view in production”).
  • Assert: oc auth can-i update deployments -n <foreign-ns> → no; for a prod namespace of dev1’s own BSN: can-i get pods → yes, can-i update → no.

Gaps

  • Framing gap (by design): every story here is conditional on a running CRC; the suite must treat “stopped” as a first-class not-run state with the checklist as its source — the hourly suite’s infra-down grading is the existing mechanism.
  • Manual-only: CRC-A7 (drills are owner-authorized, destructive-by-design events) and CRC-A2’s htpasswd leg (break-glass credentials are exercised sparingly, on the admin’s word). Both are deliberately manual; everything else codes.
  • TBD: no documented expectation for CRC’s own resource headroom (the m4’s CRC sizing — 8 CPU / 18 GiB — is a record, not a threshold with an alert behind it).