RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Tool pages

ServiceNow PDI & the MCP servers

The lab’s namespace-provisioning story has an external backend: a ServiceNow Personal Developer Instance (PDI). It holds the service registry and the request queue; the lab’s MCP servers talk to it; and its most concrete trick is turning a catalog request into a git commit that creates an LDAP group.

Part A — How it is set up

The instance

  • A free developer instance, administered by the owner. Its admin credential is a Vault tenant delivered into CRC (secret/lab/servicenow/pdi-admin — vault.md); agent work against it goes through the Table and Catalog APIs, not the classic UI (the API-first pattern is faster and scriptable; browser automation against the PDI is a known dead end here).
  • The reclaim rule that governs everything: ServiceNow reclaims an instance after ~10 days without a login. A weekly keep-alive (Saturday nights — a monthly cadence would not beat a 10-day limit, a lesson recorded when the schedule was first set) signs in and confirms the instance is online. If the PDI is ever found reclaimed, the objects below are recreatable from the build scripts in rhels/lab-agentops; nothing else in the lab depends on PDI data.

What lives there

  • The BSN registry — the lab’s service-number records for the demo apps.
  • The “Create LDAP Group” catalog item. Submitting it fires a Business Rule that commits one file per group to rhels/lab-agentops (ldap-groups/<group>.yaml) through the GitHub Contents API and records the commit SHA on the request’s queue row (u_ldap_group_request, state committed).
  • The queue table u_ldap_group_request — the request’s lifecycle: committed (file landed) → done (group exists). The write-back to done is done by the lab side (below) as the mid.lab service user.
  • The git side of the loop is the ldap-group-apply workflow (identity.md): it reconciles every ldap-groups/ file into OpenLDAP, then best-effort PATCHes the matching queue row to done. LDAP is the source of truth once the group exists; a PDI outage never fails the apply — the write-back retries on the next run.

The MCP servers

  • The MCP server (agent platform) is the agents’ front door for namespace provisioning: front door mcp.rhels.com, deployed on the tooling plane (Argo CD app agentops-mcp-server) with the platform also present on CRC. Agents authenticate with an LDAP bind and receive an opaque session token — agents act through the platform instead of holding cluster credentials.
  • It is backed by the PDI for the registry/catalog/queue flow; a snow-sim service (same repo, services/snow-sim) simulates the same API locally on both planes for exercises that should not touch the real instance.
  • Its companion workloads on CRC (the agentops namespace) include the CMDB and the PDI-backed request apps for the app1001 family — see workload-cluster.md.

Part B — How to use it

  • Request an LDAP group: submit the “Create LDAP Group” catalog item (or, equivalently and often faster, commit the ldap-groups/<group>.yaml file directly — the apply step does not distinguish the author). The group exists when the workflow run is green and a directory search finds it; the queue row reaching done is the ServiceNow-side mirror of the same fact.
  • Check a request’s state: read the queue row through the Table API (u_ldap_group_request by group name): committed means the file landed in git and the apply is pending or running; done means the group was verified in LDAP by the apply’s write-back. A row stuck at committed points at the workflow, not the PDI — go read the ldap-group-apply run.
  • Use the MCP endpoint: authenticate with a directory account (LDAP bind) to get a session token, then call the provisioning tools through it. If a call fails on auth, test the same credentials against the directory itself first (identity.md Part B) — the MCP server has no credentials of its own to be wrong about.
  • PDI etiquette: the keep-alive keeps the instance warm; do not create competing keep-alive jobs, and do not “test” the reclaim policy. Any structural change (new table, catalog item, rule) goes through the API-first build scripts so the instance can be rebuilt if it is ever lost.