RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · QA Use Cases

QA Use Cases — Vault

Tool page: vault.md · PKI: vault-pki.md · Credential locations: Service Directory. Vault is the lab’s single source of truth for credentials: one KV-v2 mount (secret), standalone on the mini, deliberately not Argo-managed (source: vault.md Part A). Each story’s Assert line is the code-facing contract a suite implements verbatim.

Admin stories

VLT-A1 — Seal state is known and correct [config/API]

  • Persona: Platform admin. Goal: prove Vault’s seal state is a known, intentional state — never a surprise discovered by a consumer.
  • Preconditions: Vault pod Running on the mini.
  • Steps: query seal status; if the mini rebooted since the last run, confirm the sealed interval was reported and the owner unseal (runbook 11) was the path back. Agents never unseal.
  • Expected: unsealed in steady state; sealed only between a tooling-host boot and the owner’s unseal — that interval is design, not outage (source: vault.md, “Seal model”; standing owner rule).
  • Assert: GET /v1/sys/seal-status → 200, sealed == false (steady state); run-over-run, any sealed == true sample must be followed by an owner-unseal event, never by an agent action.

VLT-A2 — KV integrity spot-check [config/API]

  • Persona: Platform admin. Goal: the paths every other tool’s stories depend on exist.
  • Steps: read metadata (never values into logs) for the spot set: Harbor admin, Grafana admin, ACS Central admin, directory admin, ARC GitHub App paths.
  • Expected: every spot path exists with a current version; a missing path is drift — the Service Directory promises it (source: service-directory.md; vault.md KV layout).
  • Assert: for each path p in the spot set: GET /v1/secret/metadata/lab|platform/…p → 200, data.current_version >= 1. The spot set lives in the check’s data manifest, not in this prose.

VLT-A3 — Internal CA (PKI) health [config/API]

  • Persona: Platform admin. Goal: the lab’s internal CA still stands behind the service-health issuer.
  • Steps: read the pki/ mount and root CA; confirm role lab-health-issuer; confirm ClusterIssuer vault-lab Ready.
  • Expected: mount enabled, root valid, role allows health.lab, issuer Ready (source: vault-pki.md — status BUILT 2026-10-25).
  • Assert: GET /v1/sys/mounts/pki → type == "pki"; GET /v1/pki/roles/lab-health-issuer → allowed_domains contains health.lab; kubectl get clusterissuer vault-lab → READY == True.

VLT-A4 — Backup freshness [CLI]

  • Persona: Platform admin. Goal: the one irreplaceable store has a current, drill-proven backup.
  • Steps: list dated archives on the tooling host and the off-box copy on the agent VM; compare checksums; date the newest archive against the last migration-scale change and its drill record.
  • Expected: newest archive post-dates the last migration-scale change, has a same-day restore drill on record, checksums match (source: vault.md — “a backup counts only with a same-day drill”).
  • Assert: sha256(archive@mini) == sha256(archive@vm); archive_date >= last_migration_change_date; drill record exists for that archive (the drill-record lookup is the manual residue — see Gaps).

VLT-A5 — Policy and role audit [config/API]

  • Persona: Platform admin. Goal: every Kubernetes-auth role is scoped to exactly one KV path, per the tenant pattern.
  • Steps: list policies and k8s-auth roles; read each tenant role’s bound policy.
  • Expected: no policy grants broader access than its one tenant path; the live tenant list matches the documented tenant table (source: vault.md, “Delivery into clusters”).
  • Assert: for each tenant role r: policy(r) parses to exactly one path "secret/data/<tenant>" stanza (plus its metadata twin); tenant set == documented set.

User stories

VLT-U1 — Token login reaches the UI [browser]

  • Persona: Platform admin (human). Goal: the human path into Vault works through the front door.
  • Steps: open vault.rhels.com, sign in with a token.
  • Expected: UI loads over the Let’s Encrypt chain (public roots, no internal-CA pin); the secret mount is browsable (source: vault.md, LE cutover 2026-10-25).
  • Assert: page title contains Vault; post-login URL matches /ui/vault/secrets; TLS chain verifies against system roots.

VLT-U2 — A tenant reads its own secret [config/API]

  • Persona: A VSO tenant workload (a production tenant, e.g. the CRC agentops tenant). Goal: delivery works end to end for exactly the tenant’s path.
  • Steps: read the tenant’s VaultStaticSecret status; compare the delivered Kubernetes Secret’s content hash (never its value).
  • Expected: SYNCED and the delivered Secret present (source: vault.md, tenant table + Part B).
  • Assert: kubectl get vaultstaticsecret <t> → .status.conditions[Ready] == True; delivered Secret exists; sha256(secret.data) == recorded delivery hash (hash only).

VLT-U3 — A tenant is denied outside its policy (negative) [config/API]

  • Persona: The same tenant, reaching beyond its grant.
  • Steps: with the tenant’s role token, attempt to read another tenant’s KV path.
  • Expected: denied, no data returned (source: vault.md — “a tenant can read its own secret and nothing else”).
  • Assert: GET /v1/secret/data/lab/harbor/admin with tenant token → 403.

VLT-U4 — Sealed Vault refuses reads (negative) [config/API]

  • Persona: Any consumer. Preconditions: a sealed window (post-reboot, pre-unseal — never induced by the suite).
  • Steps: attempt a KV read with a valid token while sealed.
  • Expected: the read fails as sealed; consumers pause rather than falling back to stale or plaintext copies (source: vault.md, “Seal model”).
  • Assert: GET /v1/secret/data/<any> while seal-status.sealed == true → 503 with sealed error; the suite records this story as not-run outside a natural sealed window (it never seals Vault to test it).

Gaps

  • Known-failure: the two proof tenants’ VaultStaticSecrets report SYNCED=False while otherwise healthy — undiagnosed (vault.md). VLT-U2 names a production tenant; the proof tenants stay an explicit known-failure until diagnosed.
  • TBD: no documented cadence for a PKI issuance round-trip (mount health is proven indirectly by the service-health cert stage when CRC runs).
  • Manual-only: VLT-A4’s drill-record half — “a same-day restore drill is on record” is a human attestation in the docs/issue record; the archive and checksum halves are coded. Everything else in this set is code-expressible.