RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · QA Use Cases

QA Use Cases — Grafana + Prometheus + Alertmanager

Tool pages: tools/grafana.md, tools/prometheus-alertmanager.md. Since the 2026-10-29 migration (lab-monorepo#42, item 2) the tooling-plane stack — Prometheus, Grafana, Alertmanager, Pushgateway — runs on pro; the fronts (grafana., prom., alerts.) are unchanged and stay proxied through the mini. Grafana’s single home is pro (owner decision, 2026-10-29). Alerts “page nobody yet” — the UIs are the notification path by the owner’s standing choice (source: prometheus-alertmanager.md).

Admin stories

OBS-A1 — Datasources healthy [config/API]

  • Persona: Platform admin.
  • Steps: health-check every datasource on pro’s Grafana (Prometheus, Loki, Tempo); on CRC Grafana (when CRC runs), its provisioned set plus the Alertmanager datasource.
  • Expected: all healthy; the CRC set is exactly the git-provisioned one (source: grafana.md — “Provisioned, not clicked”).
  • Assert: GET /api/datasources name set == expected set per surface; for each, GET /api/datasources/<id>/health → status == "OK" (or the datasource proxy query up returns a value).

OBS-A2 — The platform dashboards render data [browser]

  • Persona: Platform admin.
  • Steps: open agentops-main on pro’s Grafana; on CRC Grafana, open Lab Test Results (uid lab-test-results).
  • Expected: panels populated with recent series — runs / passed / failed history on Lab Test Results (source: grafana.md; prometheus-alertmanager.md).
  • Assert: each dashboard’s panel queries (issued via the datasource proxy) return >= 1 series with a sample inside the last 60 minutes; a panel returning only empty series is a named failure, not a screenshot judgment.

OBS-A3 — Scrape targets up [config/API]

  • Persona: Platform admin.
  • Steps: read each Prometheus’s active-target list; confirm the recorded tenants are up.
  • Expected: no target down without a named reason; targets arrive only via ServiceMonitors in git (source: prometheus-alertmanager.md).
  • Assert: GET /api/v1/targets?state=active → every target with health != "up" appears in the run’s exception list with a reason; count of down targets is the measured value diffed per run.

OBS-A4 — Alert rules and routes match the design [config/API]

  • Persona: Platform admin.
  • Steps: list PrometheusRules and Alertmanager routes on both tiers.
  • Expected: CRC carries crc-cronjob-consecutive-failures (alerts fire only after two consecutive failed runs); the tooling-plane route keys group-sync escalation off failing_since; no email/webhook receivers exist (source: prometheus-alertmanager.md, “Alerting”).
  • Assert: rule group present with for/expression matching the consecutive-failure pattern in git; Alertmanager config’s receiver set == {default} (UI-only) on CRC and the recorded group-sync route on the tooling tier.

OBS-A5 — Pushgateway freshness [config/API]

  • Persona: Platform admin.
  • Steps: read the group-sync push timestamp on pro’s Pushgateway; on CRC, the test producers’ latest run series.
  • Expected: group-sync pushes land every checker cycle (source: prometheus-alertmanager.md, “Pushgateways”; identity.md).
  • Assert: now - push_time_seconds{job="group-sync"} <= 240s; CRC leg: a result series exists for the most recent scheduled producer run.

User stories

OBS-U1 — Viewer sign-in and read [browser]

  • Persona: dev1 (pro surface: anonymous Viewer is on; CRC surface: directory login).
  • Steps: open grafana.rhels.com; open a demo-app dashboard and Lab Test Results (CRC surface when running).
  • Expected: dashboards render; the CRC login is OpenShift OAuth with the local form disabled. The CRC leg is proven hourly by grafana-openshift-login (source: grafana.md Part B; health-suite README).
  • Assert: GET /api/user (CRC surface) → login == <fixture> and isExternal == true; pro surface: dashboard page returns 200 and panel data per OBS-A2’s query assert.

OBS-U2 — Explore query works [browser]

  • Persona: dev1.
  • Steps: run a PromQL query for a known series and the LogQL query {log_type="application"} against Loki.
  • Expected: both return data; Loki’s documented double-storage (aggregator events stored twice by design) is not mistaken for a fault (source: grafana.md, “Query logs”; logging-pipeline.md).
  • Assert: PromQL instant query → resultType == "vector", length >= 1; LogQL range query → >= 1 stream whose labels are a subset of the allowlist (see LOG-A3).

OBS-U3 — No admin surface for viewers (negative) [browser]

  • Persona: dev2 / anonymous viewer.
  • Steps: attempt server-admin areas (users, datasource edit, alert-rule edit) on both surfaces.
  • Expected: absent or refused; on the CRC surface there is no local login form at all. CRC leg proven by grafana-oauth-only-login (source: grafana.md; health-suite README).
  • Assert: GET /api/admin/users as viewer → 403; datasource PUT as viewer → 403; CRC GET /login → redirect chain contains no password form (the suite’s existing assertion).

Gaps

  • Doc drift: grafana.md still says the tooling-plane Grafana’s admin credential “exists only inside Grafana itself”; it was escrowed to Vault during the 2026-10-29 move (see the Service Directory). The page needs its post-migration pass; these stories follow the Service Directory.
  • TBD: “anonymous Viewer is on” + “the lockdown decision is still open” (grafana.md) — OBS-U3 therefore asserts no admin functions on the pro surface, not no anonymous read. An authenticated-only expectation has no documented basis yet.
  • TBD: no documented staleness threshold for scrape targets (OBS-A3 asserts up/down only).
  • Manual-only: none — OBS-A2 is deliberately written against panel queries, not screenshots, so it codes.