RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · QA Use Cases

QA Use Cases — MCP server (Namespace as a Service)

Tool page: tools/namespace-service.md; backend: tools/servicenow-pdi.md. The rule every story hangs on: the agent is the front door, never the authority — all validation is deterministic and server-side, re-run at plan and at creation (source: namespace-service.md). Endpoint: https://mcp.rhels.com/mcp. Since 2026-10-29 the server runs on pro (the front is unchanged); both instances (mini-fronted pro + CRC, when running) run the same code from rhels/lab-agentops with SNOW_BACKEND selecting the real PDI or the simulator.

Admin stories

MCP-A1 — Tool list intact [config/API]

  • Persona: Platform admin.
  • Steps: connect, list tools.
  • Expected: the tool list contains the nine documented tools — get_bsn_details, verify_requester, list_eligible_clusters, check_namespace_name, ensure_ldap_group, plan_namespace, create_namespace_request, get_request_status, list_my_namespaces — plus the login session tool the flow begins with (source: namespace-service.md, “The nine tools” + flow step 2).
  • Assert: MCP tools/list name set == the manifest set stored as data; the manifest is pinned from the live server at suite build (see Gaps — the book’s “nine” and the login tool make the raw count ambiguous in prose, which is exactly why the set lives as data).

MCP-A2 — Backend mode is the recorded mode [config/API]

  • Persona: Platform admin.
  • Steps: read the server’s effective SNOW_BACKEND; call get_bsn_details for a known BSN (app1001) and confirm the record’s provenance.
  • Expected: the platform instances run against the real PDI (pdi) — both have since 2026-10-15; sim is for exercises only (source: namespace-service.md intro; servicenow-pdi.md).
  • Assert: get_bsn_details(app1001) → record matches the PDI registry row (app name “Payments Demo”, segment, groups) — a sim-shaped answer on a platform instance is drift.

MCP-A3 — Session issuance and identity binding [config/API]

  • Persona: Platform admin probing with fixture accounts.
  • Steps: login as dev1; call a discovery tool with the token; call again with a tampered/absent token; let a token age past its TTL (or inspect the TTL constant).
  • Expected: token issued on a real LDAP bind, opaque, 8-hour TTL; identity derives from the token alone; no token → refused; wrong password → 401 (source: namespace-service.md, step 2 of the flow).
  • Assert: login(bad password) → 401; tool call without token → refusal; tool call with token → caller’s identity == dev1 in the server-side audit counters (per-tool call and validation-failure metrics exist and increment — source: “Audit is the PR” + Prometheus counters).

MCP-A4 — Server holds no cluster-admin [config/API]

  • Persona: Platform admin. Goal: the security model’s load-bearing claim.
  • Steps: read the MCP server’s ClusterRole; confirm its only write path is git via the GitHub API.
  • Expected: a read-only ClusterRole (status aggregation); no cluster write verbs; the GitHub token arrives as an out-of-band Secret, never committed (source: namespace-service.md, “Security model”).
  • Assert: rendered ClusterRole rules contain no create, update, patch, or delete verbs on workload resources; repo grep for token-shaped strings in services/mcp-server/ → zero (the CI gitleaks gate is the standing control).

User stories

MCP-U1 — Happy path: BSN to namespace via PR [config/API]

  • Persona: dev1 through an app-team agent.
  • Preconditions: a fresh suffix on app1001’s non-prod cluster.
  • Steps: login → get_bsn_details(app1001) → verify_requester → list_eligible_clusters → check_namespace_name → plan_namespace (defaults 500m / 512Mi) → confirm → create_namespace_request → merge the PR → get_request_status.
  • Expected: the PR adds exactly one requests/app1001-<suffix>.yaml; after merge + sync, status reports the namespace live, the chart-rendered ResourceQuota and the app1001-nonprod RoleBinding (admin in non-prod) in place; a dummy pod runs (source: namespace-service.md — the worked request; the six-hour E2E cycle runs this story unattended under the standing authorization on lab-monorepo #30).
  • Assert: PR file set == {requests/app1001-<suffix>.yaml}; post-merge: namespace exists with BSN labels; RoleBinding roleRef == admin (nonprod) for group app1001-nonprod; get_request_status stage 1 == complete. (Stages 2–3 are MCP-U3’s clocks, not this story’s gate.)

MCP-U2 — Non-member is refused, and nothing is written (negative) [config/API]

  • Persona: dev2 (valid directory user, not in app1001-devsupport).
  • Steps: login → verify_requester(app1001) → plan_namespace(app1001, …) → attempt create_namespace_request.
  • Expected: verify fails in plain language naming the group to join; plan fails; create refuses — and no commit or PR results. “Proven negatively” is the documented standard (source: namespace-service.md, “Security model”).
  • Assert: verify/plan responses carry a failure naming app1001-devsupport; git log on rhels/lab-agentops shows no new commit and no new PR from the attempt (compare HEAD + open-PR count before/after).

MCP-U3 — Status tells the three clocks apart [config/API]

  • Persona: dev1, checking the MCP-U1 request later.
  • Steps: get_request_status at intervals after creation.
  • Expected: three completions reported separately — namespace live; team can log in (group exists and a group-sync cycle carried it); workloads can read secrets (VaultAuth present; the per-BSN Vault policy lands on its own clock). A pending group shows as pending with its ticket number, not as an error (source: namespace-service.md, “three completions, three clocks”).
  • Assert: response contains three distinct stage states; no stage borrows another’s value (e.g. stage 2 != complete while the group is absent from the directory).

MCP-U4 — Injection cannot reach the request file (negative) [config/API]

  • Persona: dev1, adversarially (the app1001-secure fixture pattern).
  • Steps: plan/create with hostile quota strings (YAML payloads in cpu/memory).
  • Expected: strict quantity validation refuses; nothing is committed. This is the regression story for the POC’s one critical finding (source: namespace-service.md, “Injection hardening is regression-tested”).
  • Assert: create → validation failure; HEAD unchanged; the committed-file grep for the payload → zero.

Gaps

  • Known behavior change (2026-10-29): since the server moved to pro, get_request_status’s Argo/namespace stages read pro’s API and degrade to not-found for mini-fulfilled namespaces (git/LDAP/ServiceNow stages stay exact). MCP-U3’s Assert must be read with that scope until the design call lands — recorded on lab-monorepo#42.
  • TBD: no documented expectation for plan freshness (how long a confirmed plan stays creatable before re-planning is required).
  • Manual-only: none for the coded stories; the conversational variant of MCP-U1 (a human chatting with an agent) stays a periodic manual exercise by design — the book’s own test standard is “a scripted MCP client run and a conversational agent run” (namespace-service.md).