RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Tool pages

Vault PKI — Internal CA

STATUS: BUILT (first slice), 2026-10-25. The PKI mount, root CA, and one issuing role are live in the mini’s Vault, and CRC’s cert-manager issues real certificates through a Vault ClusterIssuer. The first consumer is the service-health test suite, whose deliberately short-lived certificate is re-issued continuously. Scope is deliberately narrow: one role, one internal domain (health.lab). The per-plane intermediates from the original design are not built yet.

What exists

  • Mount + root. PKI secrets engine at pki/ in Vault (vault.rhels.com, on the mini). Root: RHELS Lab Root CA (RSA 2048, organization “RHELS Lab”), generated in Vault and never exported — no CA private key exists anywhere else.
  • Issuing role. lab-health-issuer: allowed domain health.lab with subdomains, EC leaf keys required, default TTL 1h, max TTL 30 days.
  • Policy. cert-manager-vault grants only issue/sign on that role plus read of the CA chain — nothing else in Vault.

How it’s set up (behind the scenes)

The consumer path is cert-manager on CRC (Argo app crc-cert-manager, cert-manager chart 1.21.2 — installed for this; the cluster previously had no cert-manager) with a ClusterIssuer named vault-lab (in rhels/local-k8s-gitops, clusters/crc/security/cert-manager/):

  • server: https://vault.rhels.com
  • path: pki/sign/lab-health-issuer — the FULL sign path. cert-manager POSTs the CSR to <server>/v1/<path>; a bare pki posts to the mount root and Vault answers 403.
  • Auth is Vault’s Kubernetes auth method, role crc-cert-manager (bound to the cert-manager service account in the cert-manager namespace, TTL 10 minutes). cert-manager presents a TokenRequest-minted token for that SA.

Two non-obvious halves make that login work at all; both were found live on 2026-10-25 and both are required:

  1. The auth mount carries a reviewer JWT. Without one, Vault validates each presented token by TokenReview using the token itself, which only passes tokens carrying the API-server audience. The reviewer credential is service account vault-reviewer (namespace cert-manager, bound to system:auth-delegator); its legacy token lives in the server-populated Secret vault-reviewer-token, and vault-access/scripts/reconcile.py writes it into the mount config on every reconcile (action set-reviewer re-asserts it alone; recover.sh runs that after each rebuild’s convergence).
  2. The issuer must request the vault audience. The Vault role pins audience vault; the default TokenRequest audiences do not include it, so the ClusterIssuer sets serviceAccountRef.audiences: [vault]. Without it the login fails 403 “invalid audience (aud) claim” even with the reviewer in place.

Unseal step. Vault signs nothing while sealed, so every Vault restart needs the unseal before PKI (and secret delivery) works: the mini’s home holds the unseal key file (mode 600); the unseal runs in-process on the mini — the key is read by the unseal call and never printed, passed as an argument, copied, or logged. recover.sh’s preflight performs it (the older Keychain bootstrap is the fallback). Proven in the 2026-10-25 drill: Vault restart -> sealed -> unseal -> sealed=false -> VaultStaticSecrets resync green.

How users use it

Declare a cert-manager Certificate against the ClusterIssuer — no Vault commands in the issuance path:

  • issuerRef: {name: vault-lab, kind: ClusterIssuer}
  • privateKey: {algorithm: ECDSA} (the role refuses RSA leaves: “role requires keys of type ec”)
  • dnsNames under health.lab
  • Set duration to what you actually want; Vault honors the requested TTL up to the role max, with ~30s of clock skew — the suite’s checker asserts |actual validity - requested| <= 600s.

The worked example is the suite’s own Certificate (suite.health.lab, requested duration 2h) in the lab-service-health chart: first issued 2026-10-25 with measured validity 2h 0m 30s against the 2h request.

What does not change

  • No front-door name moves off Let’s Encrypt; this CA serves lab-internal names only.
  • Vault’s own listener certificate remains the Let’s Encrypt one.
  • The seal model is untouched: issuance is a provisioning-time act, and leaf lifetimes must outlast any realistic sealed window — or, like the suite’s 2h cert, be continuously renewed by cert-manager once Vault is unsealed.