RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Planes

Tooling Plane

The tooling plane is the control side of the lab: CI runners, GitOps for the tooling host itself, secrets management, TLS automation, the image registry, and observability backends. It runs as Docker Desktop Kubernetes on the tooling host (a Mac mini).

Part A — How it is set up

Namespaces and workloads

NamespaceWorkloadDetail
argocdArgo CDGitOps engine for the tooling plane; reconciles from rhels/local-k8s-gitops. Sign-in is Dex fronting OpenLDAP (see identity.md)
actions-runner-system (removed from mini)Actions Runner Controller on pro2026-11-01: mini stack retired; label mini-k8s unchanged (#102)
agentopsOpenLDAP, Prometheus, Grafana, AlertmanagerThe lab’s identity store and the tooling plane’s own observability stack (GitOps from rhels/lab-agentops). See identity.md and observability.md
vaultHashiCorp Vault 2.0.4The lab’s secrets source of truth — live. Sealed at every host boot; the owner unseals. See vault.md
cert-managercert-managerTLS automation: the letsencrypt-prod ClusterIssuer, DNS-01 via Cloudflare. See certificates.md
ingress-nginx + tailscaleFront doorsTailscale operator + ingress give every service its rhels.com name. See certificates.md
local-path-storagelocal-path provisionerDefault storage class for the tooling cluster
cursor-agentscursor workerAgent worker pod
harborHarbor registryEnterprise image registry — live, rhels/team#107. LDAP auth; the built-in admin is Vault-managed break-glass
observabilityLoki, Tempo, LogstashLog/trace backends — live, rhels/team#107. CRC logs arrive via the standing pipeline (collectors → Logstash + Vector aggregator → dual Loki). See observability.md
lab-docsThis documentationShowcase at /, this book at /agent-docs/, behind the docs front door — see cicd.md

Still planned (not on this plane)

Nothing remaining to install on the tooling plane from rhels/team#107. Remaining #107 work is pipeline/admission, not more tooling-plane backends:

  • Digest pinning into IAC repos (recorded blocked — see cicd.md).
  • Kyverno require-signed-images Audit → Enforce on CRC (final mile).
  • Cluster-wide CRC log forwarding beyond the log-sim scope (see observability.md).
  • Vault PKI as the internal CA — designed, not built (see vault-pki.md).

Operating notes

  • Disk: ~61 GiB free at last inventory. Do not fill it — a second Argo CD install was explicitly rejected for this reason.
  • Memory: tight. The mini ARC cap is retired: all runners are on pro (2026-11-01, lab-monorepo#102).
  • Vault: live and load-bearing — but it seals at every host boot and stays sealed until the owner unseals it from the Keychain. After any tooling-host reboot, expect Vault-dependent syncs to pause; that is the design (see vault.md, runbook 11).
  • Runners: CI jobs that need lab reachability run on pro runners with the unchanged mini-k8s label (2026-11-01, lab-monorepo#102). GitHub-hosted runners cannot reach the private network.

GitOps on the tooling plane

The tooling plane is itself GitOps-managed: Argo CD (in argocd) syncs from the platform repo rhels/local-k8s-gitops using the root-app pattern. Adding a workload to the tooling plane means adding it to that repo, not applying manifests by hand.

Part B — How to use it

  • Reach the cluster: kubectl with the docker-desktop context. From the agent side, the tooling host is reached over SSH and kubectl runs on the host itself (its Homebrew path); kubeconfigs come from the secure channel, never from this book.
  • Day-to-day checks: node and pod health (kubectl get nodes, kubectl get pods -A), the Argo CD UI at argo.rhels.com for sync state (tools/argocd.md), and the Mac Health page for the host underneath it all (tools/grafana.md) — the cluster can be green on a host that is nearly out of disk or memory, so read both.
  • Change something: add or edit the workload in rhels/local-k8s-gitops (or the owning lab-agentops module) and let Argo CD reconcile. Hand-applied manifests on this plane are drift by definition (the AGENTS.md rule for that repo: mutations only through git, minimal, verified).
  • Respect the constraints: disk and memory are the scarce resources (Part A). Before adding a workload, check the free disk and the ARC runner cap; a second GitOps install was rejected on exactly this math.
  • After a host reboot: expect Vault sealed (owner unseals — vault.md, runbook 11), Docker Desktop Kubernetes to come back on its own, and the front doors to follow. The smart-plug power cycle is the owner’s unlock path for a locked host; agents do not drive it.