RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · QA Use Cases

QA Use Cases — Identity (OpenLDAP + Dex)

Tool page: identity.md. One identity store: OpenLDAP on the mini (dc=agentops,dc=local), consumed directly or through one federation hop (Dex at id.rhels.com, OpenShift OAuth). “If a service grows its own user database, that is a bug to file” (source: identity.md).

Admin stories

IDN-A1 — Seed fidelity: live directory equals the seed [CLI]

  • Persona: Directory admin. Goal: a cold rebuild reproduces the directory exactly as it stands.
  • Steps: dump the live directory (users, groups, members) and diff against the openldap-seed LDIF in rhels/lab-agentops (gitops/modules/ldap/ldap.yaml).
  • Expected: no unexplained differences; the one by-design exception is app1002-prod, deliberately unseeded (it arrives through the request path) (source: identity.md — comparison “re-verified entry-for-entry and member-for-member on 2026-10-26”).
  • Assert: normalized live LDIF == seed LDIF, with the allowlist {app1002-prod} as the only permitted live-only DN set; the diff output is the run’s measured value.

IDN-A2 — LDAPS certificate is the Let’s Encrypt cert [CLI]

  • Persona: Directory admin. Goal: the 636 listener serves the LE certificate from /certs, not a regenerated self-signed one.
  • Steps: probe the listener’s chain; confirm Certificate ldap-rhels-tls Ready; repeat after any openldap pod restart.
  • Expected: chain verifies, subject is the ldap front name. Regression story: on 2026-10-28 the pod regenerated a self-signed cert after a restart; the postStart-hook fix is what this story guards (source: identity.md, “Listeners”; lab record 2026-10-28).
  • Assert: openssl s_client -connect ldap.rhels.com:636 -verify_return_error → Verify return code: 0, subject CN == ldap.rhels.com, issuer matches /Let’s Encrypt/; kubectl -n agentops get certificate ldap-rhels-tls → READY == True.

IDN-A3 — Group-sync freshness [config/API]

  • Persona: Platform admin. Goal: the read-only checker is current and per-group failures stay isolated.
  • Steps: read the group-sync-status ConfigMap; check the checker CronJob’s last-run age; query the Pushgateway series.
  • Expected: status no older than two checker cycles (2-minute cadence); a failing group carries failing_since and fails alone (source: identity.md, “The group machinery”).
  • Assert: now - status.timestamp <= 240s; for every watched group, a status entry exists; the set of failing groups in the status == the set the suite’s own directory probe finds (cross-check, not self-report).

IDN-A4 — PostSync seed Job actually re-ran [CLI]

  • Persona: Directory admin. Goal: guard the known seam — the seed hook runs only when a sync completes.
  • Preconditions: a seed change was pushed since the last run.
  • Steps: compare the openldap-seed Job’s creation time to the seed change’s commit time; read its logs.
  • Expected: a Job run exists after the change. A stale Job is the documented failure mode; the remedy is a direct ldapadd/ldapmodify with git kept as the record (source: identity.md, caveats 2026-10-24).
  • Assert: job.creationTimestamp > seed_change_commit_time and job status Complete; outside a seed-change window the story records not-run.

IDN-A5 — Dex connector health [config/API]

  • Persona: Platform admin. Goal: the federation hop every Argo/ACS login rides on is configured as recorded.
  • Steps: fetch Dex’s discovery document; confirm the LDAP connector and its dedicated bind identity.
  • Expected: discovery served; connector present; bind custody per-consumer, never the directory admin (source: service-directory.md; stackrox.md, “LDAP SSO”).
  • Assert: GET https://id.rhels.com/.well-known/openid-configuration → 200, issuer == "https://id.rhels.com"; Dex’s rendered config contains a connector of type == "ldap" whose bind DN is the recorded per-app bind account.

User stories

IDN-U1 — Bind login as a fixture user [CLI]

  • Persona: dev1. Goal: the primitive every login story in this section stands on.
  • Steps: bind as dev1 over the in-cluster 389 listener and over LDAPS 636.
  • Expected: the bind returns dev1’s DN on both (source: identity.md Part B, “Verify a login works”).
  • Assert: ldapwhoami → exit 0, output == dn:uid=dev1,ou=users,dc=agentops,dc=local on both listeners.

IDN-U2 — Group membership is visible and correct [CLI]

  • Persona: dev1.
  • Steps: search ou=groups for groups whose member is dev1’s DN.
  • Expected: exactly the seeded app1001 memberships (app1001-devsupport, app1001-nonprod); the app1003 segment stays absent by design (source: identity.md — fixtures and the app1003 premise).
  • Assert: returned group CN set == {app1001-devsupport, app1001-nonprod} (plus any group created for dev1 by a later recorded request — additions must cite their request file).

IDN-U3 — Wrong password is rejected (negative) [CLI]

  • Persona: dev1 with a wrong password.
  • Steps: bind with an incorrect password; repeat through the OpenShift console login form.
  • Expected: invalid credentials; a visible console error. The console leg is proven hourly by console-wrong-password (source: health-suite README).
  • Assert: bind → exit non-zero, error matches /Invalid credentials (49)/; console leg per the health suite’s existing check result.

IDN-U4 — New user end-to-end [CLI]

  • Persona: Directory admin creating the next fixture, then the new user. Goal: the documented add-a-user flow works as written, including its limits.
  • Steps: follow identity.md Part B exactly: hash, seed edit, sync, live ldapmodify for membership, verify bind.
  • Expected: the user entry appears from the seed sync; the membership does not appear until the live ldapmodify — that asymmetry is the documented behavior, and the story fails if either half changes silently (source: identity.md, propagation table).
  • Assert: after sync, ldapsearch (uid=<new>) → 1 entry and group member set unchanged; after the ldapmodify, member set contains the new DN; bind as the new user → exit 0. (Runs only when a fixture is actually being added — an on-demand story, not a per-run one.)

Gaps

  • TBD: plaintext LDAP on 389 “is additional, not enforced” (identity.md); no documented condition sets when LDAPS enforcement happens, so an enforcement story has no expectation to assert yet.
  • Docs gap: Dex has no book page of its own; IDN-A5’s expectations are assembled from the Service Directory and the StackRox page.
  • Manual-only: none per-run (IDN-U4 is on-demand by design, not manual — its steps are scripted when invoked).