RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Planes

Networking

Pattern

The tooling host is reachable from the workload cluster over a private tailnet (WireGuard-based); an SSH tunnel exposes the CRC API locally so agents can reach the cluster’s API server and console through loopback.

That is the whole pattern on purpose:

  • Agents never connect to the lab over the public internet.
  • The workload cluster never exposes its API publicly.
  • CI jobs that need cluster reachability run on the self-hosted runners (labels self-hosted mini-k8s) on the tooling plane — GitHub-hosted runners cannot reach the tailnet.
  • Inbound ports 80/443 are permanently blocked by the home ISP, so nothing is ever published by port-forwarding; public names work through tailnet addressing and DNS-01 certificates instead (see certificates.md).

The tunnel is persistent now

The CRC tunnel runs as a persistent service on the agent VM, with a watchdog that checks and restarts it on a 15-minute cadence. This was boot-tested on 2026-10-24: after a deliberate reboot the tunnel returned on its own in about five and a half minutes, with the console and API health endpoints answering — no human step. A down tunnel is therefore usually a transient state: report it only if end-to-end probes (not just a bound port) stay red past a watchdog cycle, and check the CRC host’s own state first — the host sleeps when its power source changes, which no tunnel can fix.

Front doors

Inside the tailnet, services are reached by their rhels.com names rather than by machine names (one ingress + the Tailscale operator on the tooling plane). The name-and-login directory is the showcase’s service directory; the certificate story is certificates.md.

CRC’s domain — researched, not changed

CRC’s route domain is hardcoded by the product; there is no supported knob to rename it, and bundle surgery was rejected (it violates rebuild-over-repair). A 2026-10-24 research pass recommends a second IngressController carrying a rhels.com subdomain for selected apps (per-app route moves, dual-running Grafana, Argo CD last), with the console, OAuth, and API permanently staying on the built-in domain — it needs cert-manager on CRC, which does not exist yet. Decision pending with the owner; nothing is built. Until it is, apps-crc.testing names are the correct ones to document.

What is NOT documented here

By design, these docs never record: tailnet names, IPs, hostnames, SSH paths, tunnel ports, proxy details, or any credentials. If you need those values you get them from the secure channel the human provided (Secure Vault or equivalent), never from this tree, and never write them here.

Consequences for agents

  • If a runbook step says “reach the cluster,” you need the kubeconfig from the secure channel — it is not in this repo.
  • Connectivity checks are read-only: do not change tailnet state, log in interactively, or modify any host/cluster state.
  • If the tunnel is down past a watchdog cycle, report it with the probe evidence; do not attempt to recreate credentials. The tunnel’s SSH key of record lives in Vault (see vault.md).