RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · QA Use Cases

QA Use Cases — Kyverno

Sources: rhels/local-k8s-gitops docs/minimal-request-floor.md (the design + proof record) and clusters/crc/security/ (the signed-images policy). Kyverno 1.19.1 runs on all three clusters. Two policies matter: minimal-request-floor (mutate; every in-scope container’s requests set to exactly cpu 10m / memory 32Mi, limits never touched — owner decision 2026-10-26) and, on CRC only, require-signed-images (verifyImages against the Vault-held Harbor robot credential).

Admin stories

KYV-A1 — Policies Ready on all three clusters [CLI]

  • Persona: Platform admin.
  • Steps: on mini, pro, and CRC (when running), read the ClusterPolicy list and readiness.
  • Expected: minimal-request-floor present and Ready on all three; require-signed-images present and Ready on CRC, and absent on pro by design (source: minimal-request-floor.md — “Pro deliberately does not carry CRC’s require-signed-images policy”).
  • Assert: per cluster: kubectl get clusterpolicy minimal-request-floor → READY == True; CRC also require-signed-images → READY == True; pro: require-signed-images → NotFound (expected).

KYV-A2 — Twin-drift guard green [CLI]

  • Persona: Platform admin. Goal: the three floor-policy files cannot diverge silently.
  • Steps: run scripts/check-request-floor-twins.sh in rhels/local-k8s-gitops (also enforced by its PR workflow).
  • Expected: the three policy files’ spec: sections are byte-identical; only header comments may differ (source: minimal-request-floor.md).
  • Assert: script exit code 0.

KYV-A3 — Webhook failure posture per cluster [CLI]

  • Persona: Platform admin. Goal: an admission outage has the designed blast radius on each cluster.
  • Steps: read the Kyverno webhook configurations’ failurePolicy on mini, pro, and CRC.
  • Expected: mini and pro Ignore (chart feature features.forceFailurePolicyIgnore); CRC resource webhooks Fail — the pre-existing posture of its install, unchanged by the floor policy (source: minimal-request-floor.md, “Failure semantics”).
  • Assert: per webhookconfiguration: mini/pro → webhooks[].failurePolicy == "Ignore"; CRC → "Fail".

KYV-A4 — Verifier credential path intact (CRC) [config/API]

  • Persona: Platform admin. Goal: the signed-image verifier can still pull from Harbor.
  • Steps: confirm the harbor-verifier Secret exists in CRC’s kyverno namespace (delivered by the VSO tenant from Vault’s kyverno path) and the policy references it at policy level.
  • Expected: Secret present; wiring is verifyImages.imageRegistryCredentials.secrets on the policy — the controller’s --imagePullSecrets flag does not feed the verifier (proven three ways, 2026-10-25) (source: lab record — Kyverno credential fix).
  • Assert: Secret exists in ns kyverno; rendered policy contains imageRegistryCredentials.secrets: [harbor-verifier]; functional half is KYV-U3.

User stories

KYV-U1 — Big requests are floored, limits preserved (positive) [CLI]

  • Persona: A workload owner (dev1’s team) deploying to a scratch in-scope namespace.
  • Steps: create the proof pod from docs/proof/minimal-request-floor/pods.yaml (requests well above the floor, limits set); read back the admitted spec. Cover the shapes: requests+limits, limits-only, requests-only, neither, plus an initContainer pod.
  • Expected: every shape admitted with requests exactly 10m / 32Mi, limits untouched, QoS Burstable, Running (source: minimal-request-floor.md — probe evidence 2026-10-26/29).
  • Assert: for each admitted pod: every container’s resources.requests == {cpu: "10m", memory: "32Mi"}; resources.limits == submitted limits; status.qosClass == "Burstable".

KYV-U2 — Limit below the floor is refused (negative) [CLI]

  • Persona: The same workload owner, misconfiguring.
  • Steps: apply docs/proof/minimal-request-floor/negative.yaml (a container whose memory limit is below 32Mi).
  • Expected: the API server rejects the pod — a fail-closed control proving the mutation ran and limits interact as designed (source: proof record, all three clusters).
  • Assert: apply → error matching /must be less than or equal to memory limit/ (requests 32Mi > limit); pod count in the scratch namespace unchanged.

KYV-U3 — Signed image admitted, unsigned refused (CRC) [CLI]

  • Persona: A workload owner on CRC. Preconditions: CRC running; the E2E service-health app is the living example.
  • Steps: 1) Deploy the signed lab-health image. 2) Attempt a pod from an unsigned image in an in-scope namespace.
  • Expected: the signed image’s pod runs (its PolicyReport carries a PASS for the image); the unsigned pod is denied / its report FAILs with “no signatures found” — the two outcomes must be distinguishable, which was the point of the 2026-10-25 fix (source: lab record — Kyverno credential fix, service suite 17/17).
  • Assert: PolicyReport for the signed workload: result == pass for the image rule; unsigned attempt: admission error or report result == fail with message matching /no signatures found/. The E2E observer’s kyverno stage runs this story’s essence every cycle.

KYV-U4 — Exempt namespaces keep their requests [CLI]

  • Persona: A platform workload in kube-system (or kyverno, openshift-*).
  • Steps: read back a running exempt-namespace pod’s spec.
  • Expected: its declared requests are unchanged — the floor’s scope exclusions hold (source: minimal-request-floor.md, “Scope”).
  • Assert: sampled exempt pod: resources.requests == its controller’s declared values (not the floor).

Gaps

  • Known residual: one tenant’s PolicyReport fails on an unsigned upstream sidecar image — left untouched as the owner’s future call (lab record). KYV-U3 must use its own scratch workloads, not that tenant, as the unsigned control.
  • TBD: no story asserts PolicyReport freshness (how old a report may be before it stops being evidence) — no documented threshold exists.
  • Manual-only: none — this set is the most fully codable in the section; its proof manifests already exist in git.