RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · QA Use Cases

QA Use Cases — Front door

Sources: certificates.md, networking.md, tools/tailscale-cloudflare.md, service-directory.md. One front door: the mini’s ingress-nginx behind the Tailscale operator’s LoadBalancer, cert-manager with ClusterIssuer letsencrypt-prod, DNS-01 only via a zone-scoped Cloudflare token (the home ISP blocks inbound 80/443 — HTTP-01 is not a fallback, it is impossible). Most fronts now proxy to workloads on pro; the door itself did not move.

Admin stories

FDO-A1 — Every certificate Ready and far from expiry [CLI]

  • Persona: Platform admin.
  • Steps: list every Certificate across the mini, pro, and (when running) CRC; read Ready and expiry.
  • Expected: all Ready; every cert more than 30 days from expiry at check time (LE lifetimes are ~90 days, so 30 days of slack means renewal machinery, not luck, is keeping them alive) (source: QA policy per this section’s commissioning brief; certificates.md — the issuer and front doors).
  • Assert: for each Certificate: READY == True and status.notAfter - now > 30d; the measured value per cert (days remaining) is recorded for the diff — a shrinking minimum across runs is a regression even above the line.

FDO-A2 — Issuer posture: DNS-01 only [config/API]

  • Persona: Platform admin.
  • Steps: read ClusterIssuer letsencrypt-prod on the tooling plane (and CRC’s, when running).
  • Expected: Ready; solvers contain DNS-01 (Cloudflare) and no HTTP-01 solver anywhere — an HTTP-01 appearance is drift against the ISP constraint, not an improvement (source: certificates.md Part A).
  • Assert: issuer READY == True; spec.acme.solvers[*].dns01 present; spec.acme.solvers[*].http01 absent; cert-manager’s recursive-nameserver overrides (the documented DNS-01 propagation fix) present in the controller args.

FDO-A3 — Front census: every public name answers [config/API]

  • Persona: Platform admin.
  • Steps: probe every front door in the census: argo, harbor, vault, grafana, prom, alerts, docs, mcp, bridge, acs, id (plus ldap as an L4/LDAPS probe, and the CRC names when CRC runs).
  • Expected: each name returns its service (200, or the service’s own login redirect); the backends behind the proxies are the post-migration homes (Harbor, Grafana, Prometheus, MCP on pro) — the door answers for the right backend, which the per-tool stories verify in depth (source: certificates.md front-door table; service-directory.md).
  • Assert: per name: HTTP status ∈ the name’s manifest expectation (default 200 or 302 to its own login); TLS verify == ok. The health suite’s front-door-sweep observational check already walks this census.

FDO-A4 — Operator and ingress health [CLI]

  • Persona: Platform admin.
  • Steps: read the Tailscale operator and ingress-nginx workloads; confirm the front-door LoadBalancer device exists on the tailnet with its recorded address class.
  • Expected: operator and controller Running; the single front-door device is the ingress’s LoadBalancer — one door, not a drift of per-service devices (source: rhels.com migration record, 2026-10-24).
  • Assert: pods Running in tailscale and ingress-nginx namespaces; exactly one LoadBalancer Service of the tailscale class fronts the ingress.

User stories

FDO-U1 — Each public name loads in a browser [browser]

  • Persona: The owner on a phone (the standing real-world test — this exact check failed once, in 2026-10, and was never explained away).
  • Steps: open each front door from a browser on the tailnet: docs, argo, harbor, grafana, vault, acs, mcp.
  • Expected: every name renders its service’s real first screen (or its login), styled and complete — not a TLS warning, not an unstyled fallback, not a 502 (source: service-directory.md — the directory exists for exactly this walk).
  • Assert: per name: final DOM contains the service’s identifying marker (page title / login button text from the manifest); screenshot on mismatch. FDO-A3’s status codes are necessary but not sufficient — this story is the sufficient one.

FDO-U2 — The Service Directory’s links all resolve [browser]

  • Persona: The owner, using the directory as intended.
  • Steps: from the Service Directory page, follow each service link, then each “Open in Vault ↗” link (after a Vault login).
  • Expected: every service link lands on its service; every Vault link lands on an existing secret page (never Vault’s “not found”) — “if a link ever lands on a ‘not found’ in the UI, the secret moved” (source: service-directory.md).
  • Assert: link-walk: each href returns its expected surface; Vault hrefs resolve to a secret view whose path suffix == the href’s path suffix.

Gaps

  • TBD: no documented per-front latency expectation; FDO-A3 asserts reachability and identity, not timing (the health suite records durations and flags regressions — that diff is the de-facto latency contract until one is written).
  • Known external factor: off-tailnet access from the owner’s phone once failed for reasons suspected to involve the carrier/relay path, never root-caused. FDO-U1 is scoped to on-tailnet browsers; an off-tailnet expectation is TBD.
  • Manual-only: none — FDO-U1/U2 are browser-automatable (the health suite’s harness already drives this class of flow).