RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Tool pages

StackRox — security platform on pro

StackRox (the open-source line of Red Hat Advanced Cluster Security) runs on pro, an Intel MacBook Pro (6-core i7, 32 GiB) that joined the lab as a dedicated host on 2026-10-26. One decision shapes everything below: pro has no Argo CD of its own — the Mac mini’s Argo CD is the lab’s single GitOps control plane, and pro’s Docker Desktop Kubernetes is registered in it as an external cluster.

Status: LIVE (2026-10-26). All four Argo Applications are Synced/Healthy; Central shows cluster pro HEALTHY (sensor and collector). The whole StackRox namespace draws ~3.4 GiB of real memory on pro. scripts/check-pro-stackrox.sh passes all stages.

Part A — How it is set up

  • Host & cluster: Docker Desktop on pro (Kubernetes v1.36.1, kind mode), VM sized 22 GiB RAM / 5 CPU / 90 GiB disk. Docker Desktop runs without its privileged helper; the engine answers via the user socket.
  • GitOps: rhels/local-k8s-gitops, tree clusters/pro-desktop/. The mini’s root app syncs root-pro, which delivers the pro-lab AppProject and three Applications — stackrox-operator (wave 0), stackrox-central (wave 1), stackrox-secured (wave 2) — all with destination = pro.
  • The API path: Docker Desktop binds the API to pro’s localhost:6443. On pro, tailscale serve --tcp 6443 republishes it on the tailnet at the pro host’s own tailnet address, port 6443 (raw TCP passthrough). The DD API certificate has no tailnet SAN, so Argo’s cluster entry uses tlsClientConfig.insecure: true — acceptable only on the WireGuard-encrypted, device- authenticated tailnet, with the bearer token still authenticating. The cluster server must never point anywhere else.
  • Cluster identity: ServiceAccount argocd-manager (kube-system, cluster-admin) in pro’s cluster, manifests in clusters/pro-desktop/registration/. Its long-lived token is stored only in Vault (secret/platform/argocd-clusters/pro) and materialized into the mini’s Argo cluster Secret by scripts/sync-pro-cluster-secret.sh — the same Vault-to-Secret script pattern the alert bridge uses, because the mini has no Vault Secrets Operator. scripts/bootstrap-pro-registration.sh performs the whole registration from scratch.
  • StackRox itself: the stackrox-operator Helm chart 4.11.6 (public quay.io/stackrox-io images; no pull secrets, no RHACS images) installs the operator in namespace stackrox. A Central CR runs Central + Central DB + Scanner V4 (indexer/matcher pinned to 1 replica; the legacy scanner is disabled); a SecuredCluster CR secures pro itself. PVCs are 10 GiB (upstream defaults claim 100 GiB). The CRs are checked against the real 4.11.6 CRD schemas by scripts/validate-pro-stackrox-crds.py — see clusters/pro-desktop/validation/crd-schema-check.md.
  • Admin password: operator-generated, never in git. Read back from pro’s cluster Secret and stored in the VM lab store and in Vault (secret/platform/stackrox/central-admin).
  • LDAP SSO (Dex OIDC bridge): Central 4.x has no direct LDAP provider, so sign-in runs over OIDC through Dex (https://id.rhels.com, app tooling-dex on the mini), whose LDAP connector binds to OpenLDAP as the dedicated svc-stackrox account (Vault secret/lab/ldap/stackrox-bind) and emits directory groups as claims. The Central-side provider lives in Central’s database, not git — re-apply or rebuild it with clusters/pro-desktop/stackrox/apply-oidc-provider.py (run on the mini; idempotent; the load-bearing settings — bare-host uiEndpoint, mode: query, extra_scopes: groups — are documented in that script). Rules: directory group acs-admins → Central Admin; everyone else in the directory → no access. Dex’s own secrets sync from Vault with scripts/sync-dex-secrets.sh.
  • Init bundle: Sensor cannot connect until the Central-issued TLS bundle Secrets exist. The bundle is generated once with roxctl, stored in Vault (secret/platform/stackrox/ pro-init-bundle) via scripts/store-pro-init-bundle.sh, and synced into pro by scripts/sync-pro-init-bundle.sh. The generated YAML is deleted after storing — Vault is its only home.
  • UI: Central’s LoadBalancer service is published by Docker Desktop on pro’s localhost. Two doors front it. The primary one is acs.rhels.com: the mini’s ingress-nginx terminates it (Let’s Encrypt via cert-manager) and proxies to a raw-TCP tailscale serve passthrough on pro (tailnet port 8443 → Central’s LoadBalancer), speaking Central’s own TLS on the last hop — GitOps in clusters/docker-desktop/rhels-ingress/ acs-ingress.yaml, passthrough asserted by scripts/bootstrap-pro-registration.sh step 1b. Raw TCP is required because serve’s TLS-terminated handler answers any non-tailnet-name SNI with a TLS alert, so the house domain cannot ride it. The fallback door is the pro host’s tailnet name over HTTPS (tailscale serve --https 443, Central’s self-signed certificate on the backend hop).
  • Known posture debt: no Kyverno and no image-signature verification on pro yet (the request-floor twin is a tracked follow-up); images are pulled by tag from the public registry.

Part B — How to use it

  • Open Central: browse to https://acs.rhels.com from any tailnet device (the pro host’s tailnet name over HTTPS works as a fallback). Sign in with Lab LDAP (Dex OIDC) using your directory (OpenLDAP) username and password — members of the acs-admins group land as Central Admins; other directory users authenticate but get no access. The local admin account (password in Vault, secret/platform/stackrox/central-admin) remains as break-glass only.
  • Is the sensor connected? In Central: Platform Configuration → Clusters — pro should show healthy. From the shell, run scripts/check-pro-stackrox.sh on the mini: it probes the cluster Secret, the Argo apps, token authentication, and the Central/Sensor deployments, stage by stage, and exits non-zero on any FAIL. Sensor CrashLooping on a missing sensor-tls Secret is the named red state for “init bundle not synced” — run scripts/sync-pro-init-bundle.sh on the mini.
  • Re-register pro (token rotation, DD reset): re-run scripts/bootstrap-pro-registration.sh from the Spark VM, then the README’s recover matrix in clusters/pro-desktop/ for the scenario at hand (a pro reset also means a new Central CA: new admin password, new init bundle).
  • Change something: edit the CRs under clusters/pro-desktop/stackrox/ or the Applications under clusters/pro-desktop/argocd/, commit, and let the mini’s Argo sync. Re-run the CRD validation script before any chart-version bump.