RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · QA Use Cases

QA Use Cases — Alert bridge

Sources: the bridge’s GitOps tree (rhels/local-k8s-gitops, clusters/docker-desktop/alert-bridge/, app tooling-alert-bridge), the E2E suite design in rhels/lab-agentops (health-suite/), and the Service Directory. The bridge is the lab’s one machine-delivered alert path: CRC’s Alertmanager → an AlertmanagerConfig in the app000-health namespace → webhook → the bridge at bridge.rhels.com → GitHub issues in rhels/lab-agentops, label e2e-alert, deduplicated by alert fingerprint + E2E run-id. Its GitHub token is Vault-held (see the Service Directory). The bridge has no book page of its own yet — these stories are assembled from its GitOps and the E2E design, and Gaps says so.

Admin stories

BRG-A1 — Webhook reachable and healthy [config/API]

  • Persona: Platform admin.
  • Steps: probe bridge.rhels.com’s health endpoint; read the tooling-alert-bridge app and pod state on the mini.
  • Expected: bridge answers healthy; app Synced/Healthy (source: bridge GitOps tree; service-directory.md).
  • Assert: health probe → 200; Argo app pair == Synced/Healthy (per ARG-A1’s census mechanics).

BRG-A2 — Credential path is the Vault one [config/API]

  • Persona: Platform admin.
  • Steps: confirm the bridge’s GitHub token is delivered from the recorded Vault path (metadata only) — accounting for the recorded interim: the Vault entry currently holds an interim classic token, with a fine-grained issues-only replacement as a tracked follow-up (source: lab record — bridge token caveat).
  • Expected: token present via the Vault path; the interim state, while it lasts, is a named known-finding in the run report — never a silent pass.
  • Assert: Vault metadata read on the bridge path → 200; the bridge pod’s delivered Secret exists and is non-empty (presence only).

BRG-A3 — Dedup state survives a repeat alert [config/API]

  • Persona: Platform admin.
  • Preconditions: BRG-U1 has fired a test alert once.
  • Steps: fire the identical alert again (same fingerprint, same run-id).
  • Expected: no second issue — the bridge updates or ignores the duplicate per its dedup key (fingerprint + run-id) (source: E2E design record).
  • Assert: GET /repos/rhels/lab-agentops/issues?labels=e2e-alert filtered to the test run-id → count stays 1 after the second fire.

User stories

BRG-U1 — A test alert becomes exactly one issue [config/API]

  • Persona: On-call — the human whose signal this is.
  • Preconditions: CRC running (the alert originates from CRC’s Alertmanager via the app000-health AlertmanagerConfig). While CRC is stopped the story records not-run.
  • Steps: run the E2E trigger’s step A (the known-missing image tag — the documented test path, which produces a timed pull failure and a real alert), then watch rhels/lab-agentops.
  • Expected: exactly one new issue, label e2e-alert, carrying the run-id and the failing stage’s identity — within the alert-delivery window the E2E observer grades (source: E2E suite design — trigger step A + bridge dedup; the observer’s alert-delivery and alert-bridge stages grade this path every cycle).
  • Assert: issue count for the run-id == 1; issue labels contain e2e-alert; issue body matches // and names the alert; arrival within the observer’s stage timeout.

BRG-U2 — Recovery closes the loop [config/API]

  • Persona: On-call.
  • Preconditions: BRG-U1’s issue exists; the E2E cycle’s step B has landed the good image.
  • Steps: let the cycle complete; re-read the issue list.
  • Expected: the recovery is visible in the same thread (the observer attributes the run’s stages — runner, build-sign, alert-delivery, alert-bridge, … — and the run’s terminal state is recorded; the on-call reader can tell “alert fired, then recovered” without opening a second tool) (source: E2E observer design — stage-by-stage attribution is the product).
  • Assert: the run’s manifest reaches a terminal state; the observer’s stage table for the run-id shows alert-delivery and alert-bridge resolved (not pending past their timeouts).

Gaps

  • Docs gap (the big one): the bridge has no dedicated book page — no Part A/Part B. Its behavior contract lives in the GitOps tree, the E2E design notes, and one Service Directory row. These stories cite those; a bridge tool page is owed.
  • Known platform defect, recorded: the CRC AlertmanagerConfig conversion-webhook backend objects were live-repaired once and are not in git (first-aid state). BRG-U1 can therefore pass while the config’s source of truth is incomplete — the suite should report that as a standing known-finding beside any green result.
  • TBD: no documented standalone manual test-fire (a synthetic webhook POST that exercises the bridge without a CRC alert). BRG-U1-BR/U2 ride the E2E cycle by design; a bridge-only probe is TBD — no documented expectation.
  • Manual-only: none of the coded stories; the reading experience in BRG-U2 (can a human actually follow the thread) is judged by the on-call persona periodically — that judgment is the manual residue, kept deliberately.