RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · QA Use Cases

QA Use Cases — CI / ARC runners

Tool page: tools/github-actions-runners.md. Post-migration state (2026-10-29, lab-monorepo#42 phase 3): Actions Runner Controller scale sets on pro — an org-level set named arc-runner-set (max 1) and one named mini-k8s (max 2). As of 2026-11-01: lab-monorepo#102 closed; all ARC runners are on pro. The mini stack was removed, including its actions-runner-system namespace. Runner labels are unchanged. Two facts define the contract:

  • Scale-set runners carry only the set name as a label — no self-hosted label — so lab workflows target a single label: runs-on: mini-k8s. The set kept the mini-k8s name on pro deliberately; a rename to pro-k8s is a tracked follow-up (source: phase-3 record, PR #65 — the label blocker the review caught).
  • GitHub-hosted runners “cannot reach the tailnet, the directory, or either cluster — for lab work they are not a fallback, they are a category error” (source: github-actions-runners.md).

Admin stories

CIR-A1 — Scale sets registered and listening [config/API]

  • Persona: Platform admin.
  • Steps: query the GitHub org’s Actions runner scale sets; read the ARC controller/listener pods on pro.
  • Expected: both sets registered to the rhels org; listeners healthy; the mini stack is removed and its actions-runner-system namespace is absent (source: lab-monorepo#102, 2026-11-01).
  • Assert: GitHub API: scale-set list contains {arc-runner-set, mini-k8s}; pro: listener pods Running; mini: actions-runner-system namespace NotFound; no summerwind CRDs, ClusterRoles, bindings or webhooks remain.

CIR-A2 — The label contract holds across workflows [CLI]

  • Persona: Platform admin. Goal: no workflow can silently target a runner that does not exist.
  • Steps: grep every runs-on in the lab’s repos (local-k8s-gitops, lab-agentops, lab-docs-source, lab-monorepo workflows).
  • Expected: every lab-touching job uses runs-on: mini-k8s (single label) or the arc-runner-set name; the old two-label form [self-hosted, mini-k8s] appears nowhere — it can never match a scale-set runner. This defect bit for real: the docs publish workflow queued ~15 minutes on the old form (fixed in lab-docs-source PR #17) (source: phase-3 record; docs publish record 2026-10-29).
  • Assert: grep for self-hosted, mini-k8s and self-hosted,mini-k8s across workflow files → 0 matches; every runs-on value ∈ the manifest set.

CIR-A3 — Auth is the GitHub App, escrowed in Vault [config/API]

  • Persona: Platform admin.
  • Steps: confirm the ARC GitHub App credential exists in Vault (metadata only) and the runner auth path references it; confirm the retired classic-PAT path is gone or explicitly in its recorded interim state.
  • Expected: App credentials at the platform ARC path in Vault (they exist — verified during the Service Directory build); the burned classic PAT’s revocation is tracked on lab-monorepo#46, whose end state is App-only auth (source: service-directory.md; lab record — #46 reframe).
  • Assert: Vault metadata read on the ARC App path → 200 with the three recorded fields present (names only); the interim state, while it lasts, diffs loudly in the run report rather than passing silently.

CIR-A4 — Capacity ceilings are the recorded ones [config/API]

  • Persona: Platform admin.
  • Steps: read both scale sets’ max-runners from the GitOps values on pro.
  • Expected: arc-runner-set max 1, mini-k8s max 2 — the sizing that fit pro’s request-floored node (source: phase-3 record).
  • Assert: rendered scale-set specs: maxRunners == 1 and == 2 respectively; live GitHub-side counts agree.

User stories

CIR-U1 — Smoke workflow lands on a pro runner and goes green [config/API]

  • Persona: A developer dispatching arc-smoke-test.
  • Steps: dispatch the workflow; watch the run.
  • Expected: the job is picked up by a pro scale-set runner (runner name carries the set prefix) and completes green. Proven at delivery: smoke green ×4 on pro runners (source: phase-3 proof record).
  • Assert: run conclusion == success; the job’s runner name matches ^(arc-runner-set|mini-k8s)-; runner host is pro’s cluster (pod namespace arc-runners).

CIR-U2 — A real pipeline runs end to end [config/API]

  • Persona: A developer pushing to a pipeline repo.
  • Steps: dispatch the service-health signed pipeline.
  • Expected: build (buildah) → Harbor push → cosign sign → Trivy gate → digest artifact, all green on a pro runner — twice consecutively was the phase-3 proof bar (source: phase-3 record — runs green ×2).
  • Assert: two consecutive runs: conclusion == success and the digest artifact exists on both.

CIR-U3 — A wrong label queues visibly, never silently (negative) [config/API]

  • Persona: A developer who typos a label.
  • Steps: dispatch a probe workflow with a nonexistent runs-on label; observe for the queue window; cancel it.
  • Expected: the job stays Queued with no runner assigned — visible in the Actions UI/API the whole time, never dispatched to a wrong runner and never reported as lost. The book’s diagnosis order is the story’s other half: “a job stuck at Queued is almost always: the cap slot is busy, the labels drifted, or the controller is down — in that order” (source: github-actions-runners.md Part B).
  • Assert: after the window: run status == queued, runner_name empty; after cancel: status == cancelled; CIR-A1 still green (the probe did not disturb the sets).

Gaps

  • Retirement verified (2026-11-01): all consumer runs after mini removal used pro runners; no workflow labels changed (lab-monorepo#102).
  • TBD: no documented queue-wait SLO (how long Queued is normal at cap); CIR-U3 asserts visibility, not a duration.
  • Manual-only: none.