RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Tool pages

GitHub Actions — self-hosted runners

Every workflow that touches the lab runs on the lab’s own runners. GitHub-hosted runners cannot reach the tailnet, the directory, or either cluster — so for lab work they are not a fallback, they are a category error.

Part A — How it is set up

2026-11-01: lab-monorepo#102 closed; all ARC runners are on pro. The mini stack was removed, including its actions-runner-system namespace. Runner labels are unchanged.

  • Controller and runner sets: ARC runs on pro; the mini stack is retired. The organisation-level scale sets retain their existing names and labels.
  • Capacity: use the pro scale sets’ configured limits; the former mini cap is no longer a hosting constraint.
  • Who depends on them: the docs pipeline (cicd.md), the ldap-group-apply workflow that creates LDAP groups from git (identity.md), the per-app IAC pipelines of the signed supply chain, and the ServiceNow group-apply loop. Each would fail in a distinctive way on a hosted runner (network unreachable) — that failure mode is the tell that a workflow lost its runs-on labels.
  • Runner image quirks (learned once, recorded forever):
    • python3 exists but there is no pip / ensurepip — workflows bootstrap pip explicitly or avoid it (stdlib-only scripts).
    • Node is not preinstalled — workflows that need it use actions/setup-node (the docs workflow does).
    • Tooling is fetched as static binaries rather than installed (a pinned yq binary is the standing example).
    • Non-interactive Docker on the tooling host cannot reach the macOS keychain credential helper; builds that need anonymous pulls set DOCKER_CONFIG=/tmp/docker-nocreds (the docs image build does exactly this).

Part B — How to use it

  • Target the runners: every lab-touching job starts with

    runs-on: mini-k8s

    If a step must run in a container, the container runs on the runner pod — the ldap-group-apply workflow’s python:3.12-slim job container is the working example.

  • Check runner health: kubectl get pods -A on pro shows live ARC pods; the GitHub organisation’s Actions → Runners view shows the scale set’s registration. A job stuck at “Queued” is almost always one of: the configured capacity is busy, the labels drifted, or the controller is down — in that order.

  • Debug a failed job: read the step log first; if the failure is a network error to a lab name, suspect the runs-on labels before the network. If it is a missing-tool error (pip, node, yq: command not found), it is the runner-image quirks above — fix the workflow, not the runner.

  • Secrets in workflows: values come from GitHub Actions secrets (placed by a human or re-cut from Vault — the Harbor CI secret is the worked example, tools/harbor.md) or from Vault through a Secrets Operator tenant for in-cluster consumers. A credential typed into a workflow file is an incident, not a shortcut.