RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · QA Use Cases

QA Use Cases — ServiceNow PDI + sims + group-sync

Tool page: tools/servicenow-pdi.md. The PDI (dev377665, a free developer instance) is the namespace platform’s real backend: the BSN registry table (u_bsn_registry), the request queue (u_ldap_group_request), and the “Create LDAP Group” catalog item whose Business Rule commits a group file to rhels/lab-agentops (ldap-groups/), which the ldap-group-apply workflow reconciles into OpenLDAP. A simulator (snow-sim) with the same API shape stands in for exercises. Platform work goes through the Table and Catalog APIs as the service user mid.lab — never the classic UI (source: servicenow-pdi.md; lab record — API-first pattern).

Admin stories

SNO-A1 — Registry and queue tables reachable [config/API]

  • Persona: Platform admin (as mid.lab).
  • Steps: Table-API read of u_bsn_registry and u_ldap_group_request.
  • Expected: both answer; the registry holds the demo BSN records (app1001/1002/1003 family) with their segment, support-group, and cluster fields populated (source: servicenow-pdi.md — “The BSN registry”; lab record: 9-column table, 3 seed records).
  • Assert: GET /api/now/table/u_bsn_registry → 200, result count >= 3, every record has non-empty segment + support group; queue table → 200.

SNO-A2 — mid.lab role posture is the intended one [config/API]

  • Persona: Platform admin.
  • Steps: read mid.lab’s role grants.
  • Expected: mid_server + catalog — and not admin. Honest current state: mid.lab still carries a temporary admin grant because the queue table was API-created and has no ACLs; the owner is creating the three ACLs (read/write/create for mid_server), after which the admin grant is removed and re-verified. Until then this story reports the grant as a known-failure with an owner, not a pass (source: lab record — the parked least-privilege downgrade, owner route chosen 2026-10-24).
  • Assert: role set for mid.lab == {mid_server, catalog} — currently expected to FAIL on the extra admin role; the regression diff must keep naming it until the ACLs land.

SNO-A3 — Keep-alive freshness [config/API]

  • Persona: Platform admin. Goal: the instance never approaches the reclaim rule (>90 days old and no login in 10 days).
  • Steps: read the keep-alive job’s last successful run (weekly, Saturday nights).
  • Expected: last login well inside the 10-day window — the weekly cadence exists precisely because a monthly one would not beat the limit (source: servicenow-pdi.md — “The reclaim rule that governs everything”).
  • Assert: now - last_keepalive_success <= 8 days; PDI root responds 200 (an instance that needs waking is a finding, a reclaimed one is an incident).

SNO-A4 — Simulator parity [config/API]

  • Persona: Platform admin.
  • Steps: with SNOW_BACKEND=sim on a scratch MCP instance, run get_bsn_details for the same BSNs as SNO-A1; compare shapes.
  • Expected: the sim answers with the same record shape (fields the MCP consumes), so exercises off the real PDI stay honest (source: servicenow-pdi.md — “a snow-sim service simulates the same API locally”).
  • Assert: field-name set of sim records == field-name set of PDI records for the consumed fields; values may differ, shape may not.

SNO-A5 — Group-sync write-back health [config/API]

  • Persona: Platform admin.
  • Steps: read recent queue rows; compare their states against the directory.
  • Expected: rows progress committed → done; a row stuck at committed points at the apply workflow, not the PDI; a PDI outage never fails the apply itself — the write-back retries next run (source: servicenow-pdi.md).
  • Assert: for the latest N rows in state done, the named group exists in LDAP (cross-check per IDN-A3’s mechanics); zero rows older than one apply cycle sit in committed without a corresponding workflow failure on record.

User stories

SNO-U1 — Catalog request becomes a real group [config/API]

  • Persona: A team member needing a new LDAP group.
  • Steps: submit the “Create LDAP Group” catalog item (API) for a throwaway group name; wait for the Business Rule’s git commit; wait for the ldap-group-apply workflow run; search the directory.
  • Expected: a REQ exists; one file ldap-groups/<group>.yaml lands in rhels/lab-agentops with the commit SHA recorded on the queue row (state committed); the workflow creates the group with the requester as first member; the row is written back to done (source: servicenow-pdi.md Part B — “Request an LDAP group”; identity.md — group creation from git). Proven end-to-end repeatedly in the POC era (REQ0010001–REQ0010005).
  • Assert: queue row state transitions to done; ldapsearch (cn=<group>) → 1 entry whose member contains the requester’s DN; the throwaway group is then removed by the documented removal path (the story cleans up after itself).

SNO-U2 — Request state is readable without the UI [config/API]

  • Persona: The same requester, checking progress.
  • Steps: Table-API read of the queue row by group name at each stage of SNO-U1.
  • Expected: committed means “file landed, apply pending/running”; done means “verified in LDAP” — the two states are never conflated (source: servicenow-pdi.md — “Check a request’s state”).
  • Assert: at the moment the git commit exists but the workflow has not finished, state == committed (not done); after write-back, done.

SNO-U3 — A malformed group request is refused (negative) [config/API]

  • Persona: A requester (or the apply script) with a bad group name.
  • Steps: submit/commit a group file whose name violates the apply script’s allowlist.
  • Expected: the apply refuses it (allowlist: lowercase alphanumerics plus ._-, bounded length) and no directory entry appears; existing groups are never modified by the apply — “existing groups are left exactly as-is” (source: lab record — scripts/apply_ldap_groups.py allowlist + idempotence; identity.md).
  • Assert: workflow step for the bad file → failure with the allowlist named; ldapsearch for the bad name → 0 entries; a pre-existing group’s member set is byte-equal before/after the run.

Gaps

  • Known-failure (owned): SNO-A2 — the mid.lab admin grant awaits the owner’s three ACLs; the story is written to fail honestly until then.
  • TBD: the PDI has no documented expectation for catalog item content beyond the group flow (fields, approvals) — stories assert the flow, not the form.
  • TBD: snow-sim persistence/metrics (a §3.12 backlog item in the design record) — no expectation exists for the sim beyond API shape (SNO-A4).
  • Manual-only: none.