RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Tool pages

Tailscale & Cloudflare — the name layer

Two external services hold the lab together: Tailscale provides the private network every lab connection rides, and Cloudflare provides the DNS (and registrar) for the names that resolve onto it. Nothing in the lab is reachable from the public internet — names resolve publicly, but the addresses behind them are tailnet addresses, so only tailnet members can connect. (And inbound 80/443 are blocked by the ISP regardless — see certificates.md.)

Part A — How it is set up

Tailscale

  • One tailnet joins the lab’s machines: the tooling host (Mac mini), the CRC host (MacBook Air), the agent VM, and the owner’s devices. MagicDNS is on; HTTPS (Tailscale-issued certificates for machine names) is enabled.
  • The Tailscale operator in the tooling cluster (tailscale namespace, Argo CD app tooling-tailscale-operator) is what publishes the services’ front doors onto the tailnet; the ingress layer in front of it is in certificates.md.
  • A second operator runs in the workload cluster (CRC) since 2026-10-25 (Argo CD app crc-tailscale-operator, same tailscale namespace), so tailnet members can reach the CRC API server and console directly — the Cursor agent’s access path. It runs the APIServer proxy (device crc-operator) and fronts an in-cluster ingress controller (device crc-ingress, chart crc-ingress-nginx). Its OAuth client credential lives in Vault (secret/platform/tailscale/operator-oauth) and reaches the cluster only via the Vault Secrets Operator; nothing is in git. On OpenShift the operator’s shared proxies service account holds the privileged SCC — inherent to tailscaled TUN proxies there; everything else runs under narrower SCCs. Client access is a dedicated service account, cursor-agent, whose token is in Vault (secret/lab/crc/cursor-agent); it currently holds cluster-admin by the owner’s direction — revoke by removing its ClusterRoleBinding in clusters/crc/tailscale/.
  • The lab’s own domains are verified as organisation domains on the tailnet, and the primary lab zone doubles as a tailnet DNS search domain — so short names resolve the way lab members expect.
  • Machine names in the tailnet are load-bearing: lab wiring refers to hosts by their tailnet names. Never rename the tailnet — the rename looks cosmetic and breaks the CRC tunnel, the log pipeline’s cross-plane hop, and the operator wiring in one move.
  • API access for automation is a separate Tailscale API key (connector-managed, values never in git or these docs). It expires in early January 2027; a renewal reminder is set for the same week as the Cloudflare token below. Connectivity checks the agents run against this layer are read-only by standing rule.

Cloudflare

  • One account; five Free-plan zones, four of them registered at Cloudflare Registrar with auto-renew on and transfer locked (renewals fall between March and July 2027). The lab’s primary zone carries the service names (argo, harbor, vault, grafana, prom, alerts, docs, mcp, ldap — the front doors in certificates.md) as DNS-only records pointing at tailnet addresses, plus the tailnet’s own machine records, so lab names and machine names resolve in one place.
  • API access is a token with broad account access, confirmed live in October 2026. It expires on 2027-01-09; a renewal reminder fires on 2027-01-05. Losing it does not take names down (DNS keeps serving), but it freezes DNS edits and the tailnet verification challenges.
  • Separate, zone-scoped DNS-only tokens are what cert-manager uses for DNS-01 challenges — one per cluster (the workload cluster’s was minted in October 2026 for its letsencrypt-prod issuer). Each lives in that cluster’s cert-manager namespace and in Vault, and they are covered in certificates.md. Do not confuse them with the account token: a cert-manager token cannot edit the account, and the account token is never given to a cluster.

Part B — How to use it

  • Resolve a lab name: any tailnet member resolves the front doors through Cloudflare DNS to tailnet addresses. If a name resolves but does not connect, the problem is the tailnet or the service — not DNS; if it does not resolve, check the zone before touching anything else.
  • Add a front door (a new named service): the pattern is three-part and all of it is git except the record itself — (1) the service’s ingress/operator wiring in rhels/local-k8s-gitops (the rhels-ingress apps are the template), (2) a DNS-only A record in the Cloudflare zone pointing at the same tailnet address the other doors use, (3) a Certificate via cert-manager if the service terminates its own TLS (the LDAP pattern) or the shared ingress certificate otherwise. Then add the name to the front-door table in certificates.md and the showcase’s service directory.
  • Renew a token: both expiries land in the first half of January 2027 and both have reminders. Renewal means minting a replacement token of the same scope, updating the connector / secret that holds it, and verifying with one read call — never pasting a token value into an issue, a doc, or git.
  • Reach the CRC API from any tailnet machine: use a kubeconfig whose server is https://api.crc.rhels.com with tls-server-name set to the operator proxy’s Tailscale machine name (the crc-operator device) — the DNS name is the door, but the API proxy presents its Tailscale machine-name certificate (publicly trusted, so no CA file and no insecure flags). A bare curl to the name is reset by the listener’s SNI matching; that is expected, not a fault. The first connection through a cold proxy can take ~30 s of TLS warm-up; steady state is fast.
  • Reach the CRC console: https://console.crc.rhels.com loads the console app under its own Let’s Encrypt certificate. The login flow still hands off to the console’s canonical OAuth address (oauth-openshift.apps-crc.testing), so a full browser login is only reliable from hosts with the lab’s DNS plumbing — API access does not depend on it.
  • Check tailnet health: the Mac metrics tier (tools/prometheus-alertmanager.md) is the ground truth for whether a host is on the network at all; the Tailscale admin console’s machine list is the second source. Report a host down only from fresh evidence, never from a stale check.