RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Operations

The Agent-Run Lab

A personal lab, run the way a production platform is run — because the point of it is the patterns, not the hardware. Three Kubernetes clusters (two on Macs, one OpenShift), a container registry, a secrets vault, a directory, an observability stack, and a CI system — operated day to day by AI agents doing real platform-engineering work, with a human keeping the decisions that should stay human.

This page is the presentation. Everything on it points at a live artifact — a pull request, a test report, a dashboard — that a skeptic can open and inspect. Nothing here is a screenshot of a promise.

The thesis

Agents do the team’s routine work end-to-end; humans keep the one-way doors.

Most AI demonstrations are a chat window producing text. This lab is the opposite: every change an agent makes lands as a pull request with a test plan and a revert plan, is checked by an independent test fleet the agents don’t control, and is graded continuously by observers that report failures honestly — including failures that stay red on purpose, because fixing them needs a human’s authority, not an agent’s initiative.

By the numbers

  • 3 clusters under one GitOps control plane (mini, pro, and an OpenShift cluster)
  • 14 independent QA suites running daily, each reporting a regression diff against its own previous run
  • 27 minutes to rebuild the OpenShift cluster from git, end to end — the one timed drill (October 2026), down from ~95 minutes before the recovery work
  • 1 platform migration (registry, CI runners, observability, and the platform stack moved between machines) executed by agents — the registry’s data verified byte-for-byte after its move, the rest re-proven fresh at standup, rollback snapshots kept
  • Every 6 hours, a fully hands-free end-to-end cycle runs: build, sign, deploy, alert, and file its own issue — nobody touches it, and it publishes its own stage-by-stage verdict, including the stages that fail
  • Every 6 hours, a second test sends a fresh agent in with only the written documentation and grades whether the docs alone can carry a newcomer through the work

Six things you can watch it do

  1. A namespace, requested in plain language. A developer asks for a namespace the way a team would ask. The agent validates the request deterministically — against the service catalog and the directory, on the server side; the language model is never the authority — then opens a pull request. A second agent reviews it. On merge, GitOps provisions the namespace with its quota, access bindings, and secrets wiring. The artifacts: the request, the review, the merge, the live namespace.
  2. The same request, done wrong — on purpose. Ask for a namespace belonging to a service you are not a member of: the system denies it, with the reason stated. Deploy an unsigned container image: where image verification is enforced it is rejected at admission, and everywhere it surfaces as a failed policy report — nothing unsigned passes unnoticed. Guardrails shown as behavior, not as policy slides.
  3. The agents grade themselves. Fourteen test suites run daily; every report leads with what changed since yesterday. The most instructive line is a red one: a least-privilege check on a service account has stayed red by design, because completing it requires a human to create three access rules — and the agents refuse to escalate their own privilege to finish it. That one red check answers “who watches the agents?” better than any diagram.
  4. A real migration, receipts out. The registry, the CI runners, the observability stack, and the platform services moved from one machine to another, executed by agents: the registry’s data verified byte-for-byte after its move, the rest re-proven fresh at standup, rollback snapshots kept, and not one public address changed in the process. Across the whole program, the human made three decisions; everything else is in the pull-request history.
  5. Recovery as a number. The OpenShift cluster was rebuilt from git in 27 minutes in a timed drill, down from ~95 minutes before the recovery work. Delete-and-recreate is a rehearsed procedure here, not a hope — which is exactly what makes experimenting cheap.
  6. The documentation test. Every six hours a fresh agent with zero context receives only the written documentation and must perform the developer journey end to end. Every place it stalls becomes a filed documentation-gap issue, and each run owes one proposal to simplify the flow. “The devil is in the detail” is the objection this lab takes most seriously — so the detail is continuously tested by an agent that knows nothing.

Why mistakes are cheap here (and why that travels)

This lab is not cheap to break because it is a lab. It is cheap to break because of three architectural choices, and all three are portable to an environment where mistakes cost real money:

  • Everything is code. The desired state of the clusters lives in git — 60% of the platform fully git-defined today, measured in the SRE assessment linked below, with the remaining seams named rather than hidden. A mistake is a bad commit, and a bad commit has a revert. Rebuilding is cheap because nothing critical exists only in someone’s hands.
  • Reversibility is graded, and autonomy follows the grade. Tier 1 — reversible and tested: the agent acts alone. Tier 2 — a second agent reviews before anything merges. Tier 3 — credentials, deletions, production approvals: human-only, and silence is never approval. The tiers are written down in a rulebook and enforced by the workflow, not left to someone’s judgment in the moment.
  • Agents cannot mint power. Credentials live in a vault; agents hold scoped, short-lived access and cannot create more. When a credential leaked into a log here, the protocol treated it as burned: it was rotated to an app-based credential with a fraction of the reach, and the final revocation step stayed human-only. The system is built so the worst realistic agent mistake is a pull request a human can reject — not a change already loose in production.

The honest limits

Credibility lives here, so they are stated before anyone has to ask: this pattern does not yet answer formal change-management evidence (the CAB question), regulated-data handling, or support at enterprise scale. Those are real gaps. Inside the lab itself, two services still have no backup, the vault’s unseal key lives on a single host, and image verification is not yet enforced on every cluster — all three are on the record in the SRE assessment, not discovered by auditors. What the lab demonstrates is that the work pattern — agent execution under tiered, auditable guardrails — holds up under continuous testing. The compliance and scale questions are the next layer, not a hidden flaw.

The ask: a 30-day pilot

Pilot the pattern on non-production namespace requests — the request flow that already exists in the day job — with agents drafting the pull requests and humans approving every merge, for 30 days. Measure two things against the current manual flow: cycle time and error rate. The pilot is scoped, reversible, and measurable on purpose: if the numbers don’t beat the manual flow, kill it — with the data in hand.

Go deeper