RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Operations

CI/CD

Current IAC flow

Everything in the lab follows infrastructure-as-code reconciled by Argo CD:

  1. An agent drafts a plan on a GitHub issue (rhels/team#NNN).
  2. The peer agent reviews the plan (handoff format: Context / What I did / What I need from you / Acceptance criteria).
  3. The human authorizes execution (“approve and build” or equivalent). A peer’s approval is not authorization — see agent-protocol.md.
  4. The agent implements against the relevant IAC repo (platform repo rhels/local-k8s-gitops or a per-app repo rhels/IAC.<bsn>).
  5. Argo CD syncs the cluster state. The agent verifies health and posts evidence back on the issue.

GitHub Actions workflows that touch the lab run on the self-hosted mini-k8s runners on pro since 2026-11-01 (lab-monorepo#102) (label self-hosted mini-k8s), because GitHub-hosted runners cannot reach the private network.

Signed pipeline (rhels/team#107)

Harbor plus a signed supply chain. Overall status: partial — not a flat “live” (Harbor + Cosign + Trivy live; later stages not). Evidence: rhels/team#107; GitOps Actions run 36876895467 (headSha c5d749f586d43640a61591f0bcdb94ac04d98e0d); GitOps main c5d749f586d43640a61591f0bcdb94ac04d98e0d clusters/crc/security/require-signed-images-policy.yaml (require-signed-images still validationFailureAction: Audit). rhels/team#108 is docs-candidate coordination only, not lab evidence.

Live on that run: git push, GitHub Actions (self-hosted mini-k8s), build + push to Harbor, Cosign sign (keyless, GitHub OIDC), Trivy scan. CI authenticates to Harbor with a GitHub Actions secret; the value was rotated on 2026-10-24 and the previously committed default retired (see vault.md).

Blocked: digest pin into rhels/IAC.<bsn> — run 36876895467 failed at “Pin digest in IAC repo” after build/push, Cosign, and Trivy succeeded (HTTP 403 on the write-back).

Audit, not Enforce: Kyverno ClusterPolicy require-signed-images (verifyImages) in app* namespaces has validationFailureAction: Audit. Unsigned images are reported, not rejected. Do not treat digest pinning or Enforce as live.

  1. Developer pushes to the app repo.
  2. GitHub Actions builds on a self-hosted mini-k8s runner.
  3. Image is pushed to Harbor (enterprise registry on the tooling plane — live).
  4. Cosign signs the image — keyless, via GitHub OIDC.
  5. Trivy scans the image for vulnerabilities.
  6. The image digest is written to the app’s IAC repo (rhels/IAC.<bsn>) — this write-back is the blocked step.
  7. Argo CD syncs the IAC repo to the workload cluster (the signed-digest pin is the blocked predecessor; today this is whatever digest/tag is already in the IAC repo).
  8. Kyverno verifyImages (require-signed-images) is the admission gate in app* namespaces — Audit now; Enforce is the #107 final mile.

Until digest pin and Enforce land, Argo CD still syncs whatever digest/tag is already in the IAC repo, and unsigned images are not blocked at admission.

Kyverno’s registry credential (private Harbor projects)

Verification needs to pull from Harbor before it can check a signature. Public projects (e.g. library, the demo apps) verify anonymously, but an image in a private project failed verification with a Harbor 401 until 2026-10-25: Kyverno’s controllers held no registry credential, so a signed image in a private project FAILed its PolicyReport indistinguishably from an unsigned one. Fixed 2026-10-25 (owner pick: Kyverno credential only; policy scope, attestors, and Audit mode unchanged):

  • Credential: a Harbor system-level robot, robot$kyverno-verifier — pull-only on repositories in all projects (namespace: "*"), never expires. System-level was chosen deliberately: verification must not break again the day a second private project appears. Harbor 2.15 supports this cleanly (proven against the token endpoint for both a private and a public project before rollout).
  • Provisioning: clusters/crc/vault-access/provision-kyverno-verifier.sh in rhels/local-k8s-gitops, run on the mini. It reads the Harbor admin password from the harbor/harbor-admin Kubernetes Secret and the Vault token from ~/.vault-token. The robot, the Vault policy/role, and the KV entry are mini-plane state outside git — git holds only the VSO tenant that delivers the credential and the policy field that consumes it.
  • Recover ensure (the rebuild path): the script’s ensure mode is wired into recover.sh as its own phase. It idempotently writes the Vault policy + role crc-kyverno-vso and converges the credential ONLY if it is missing or broken (KV-named robot gone, KV entry absent, stored credential no longer authenticating); against a healthy state it is a pure no-op, and every failure is loud (ENSURE FAIL [<stage>], non-zero — recover aborts). This closed the hole where the Vault role existed only because a human once ran the script: after a Vault rebuild or a greenfield Harbor, verification silently returned to anonymous 401s. The recover acceptance battery additionally gates on the CRC end: the harbor-verifier VaultStaticSecret synced, plus a live private-image verify (signed PASS + unsigned FAIL) in the PolicyReports.
  • Storage & delivery: Vault KV at secret/platform/kyverno/harbor-verifier (single key .dockerconfigjson, the lab’s established VSO pattern), read by Vault policy/role crc-kyverno-vso. On CRC, the GitOps tenant clusters/crc/vault-access/tenants/kyverno.yaml declares a VaultAuth + VaultStaticSecret that deliver it into the kyverno namespace as the dockerconfigjson Secret harbor-verifier (VSO refresh 30 s). No hand-made cluster objects; the path is recover-safe. The credential value never appears in git.
  • Wiring: the ClusterPolicy’s verifyImages entry names the secret via imageRegistryCredentials.secrets: [harbor-verifier] — the documented verifyImages mechanism, and the ONLY wiring. The controllers briefly also carried an --imagePullSecrets flag (admission, background, reports), but on Kyverno 1.19 that flag alone does not feed the cosign verifier: with the flag live and the secret present, fresh verifications still went out anonymously (401) until the policy field landed. The dead flag was removed 2026-10-25; treat the policy field as the operative wiring.
  • Rotation: re-run the provisioning script (rotate, the default) on the mini. The order is deliberate (create → seed → delete): the NEW robot is created and proven at Harbor’s token endpoint first, Vault is seeded and the stored credential is read back and authenticated, one VSS refresh interval is outwaited — and only then is the old robot deleted. (The original delete-first order left a window where the delivered credential was dead and PolicyReports false-failed with 401s.) Generations are timestamped (robot$kyverno-verifier-<ts>); exactly one exists at steady state, and the active one is whichever the Vault KV entry names. Every rotation ends with the positive/negative proof below — never skip it.
  • Proof (2026-10-25): the signed image in the private lab-health project flips to PolicyReport PASS (“verified image signatures”); the unsigned control still FAILs, now with “no signatures found” instead of the credential 401 — the two failure modes are finally distinguishable. Demo apps in public library still PASS. The service suite’s e2e-kyverno stage reads PASS.

Minimal request floor (minimal-request-floor, both clusters)

Since 2026-10-26, pod resource requests no longer reserve compute on either cluster. A Kyverno mutate ClusterPolicy sets every in-scope container’s requests to a uniform floor — cpu 10m, memory 32Mi — while leaving limits untouched as the safety rails. Scheduling now reflects the floor, and the truthful capacity question becomes real usage (kubelet/Prometheus), which is what the RHACS sizing assessment now runs on.

  • Why a floor instead of deleting requests: Kubernetes defaults a container’s requests from its limits when the requests field is absent, so a delete-only mutation is silently resurrected by the API server wherever limits exist (proven live on CRC: a limits-only pod was admitted Guaranteed with requests == limits). An explicit floor is never defaulted over. A delete-based variant also failed live: a foreach over the absent initContainers list errored and voided the whole mutation.
  • Shape: three rules — containers unconditionally; initContainers and ephemeralContainers gated on the list key existing (preconditions), each using foreach + patchStrategicMerge keyed by container name. Autogen is disabled (pod-policies.kyverno.io/autogen-controllers: "none"): the policy fires on Pods only, so GitOps controller templates stay byte-stable and Argo never fights its own mutation. failurePolicy: Ignore, background: false.
  • Scope: all namespaces except kube-system, openshift-*, and kyverno. Platform namespaces keep their declared requests (they dominate CRC’s request total by design). A container whose memory limit is below the floor is rejected by the API server (“requests must be ≤ limit”) — the fleet inventory found zero such containers on either cluster before rollout.
  • Failure semantics are two layers: the ClusterPolicy’s own failurePolicy: Ignore covers evaluation/mutation errors; admission reachability is the webhook’s. The mini’s webhook runs Ignore (features.forceFailurePolicyIgnore in the chart values) so policy work can never block lab pods; CRC’s webhooks remain Fail — pre-existing posture of the #107 install, unchanged by this policy.
  • Placement: identical policy twins — CRC in clusters/crc/security/ (via crc-root), the tooling plane via Argo apps tooling-kyverno (operator, sync-wave 0) and tooling-kyverno-policy (wave 1) in rhels/local-k8s-gitops. A drift check (scripts/check-request-floor-twins.sh) compares the two specs; reproducible proof pods live in docs/proof/ of that repo.
  • Rollout effect (2026-10-26): existing pods kept their specs until restarted; every in-scope workload was rolling-restarted (Vault excepted — it adopts the floor on its next natural restart). In-scope requests went to exactly floor × containers — mini 1.34 CPU / 3.10 GiB → 0.57 / 1.78; CRC 1.23 / 2.37 → 0.36 / 1.12. Every in-scope pod is now QoS Burstable (was mixed Guaranteed/Burstable): under node memory pressure, eviction is usage-ranked across the whole fleet; the lab’s stance is scrap + recreate with right metrics rather than reservation-based protection.

How to use it: nothing to do — new pods are floored at admission. To see a workload’s real appetite, read usage (not requests): kubectl top-equivalents come from the kubelet stats summary or Prometheus. If a component genuinely needs reservation back (stateful, eviction-sensitive), that is an owner decision to exclude its namespace in the policy on both twins — run the drift check after editing.

Docs pipeline (this repo)

CI on this private source repo (rhels/lab-docs-source) builds and audits an audited dist/ bundle from private source. It does not publish a live public site today — but the docs are live inside the lab. As of 2026-11-01, the automated pipeline (workflow_dispatch with DOCS_PUBLISH_ENABLED) builds/audits and pushes harbor.rhels.com/library/lab-docs:<tag>; the workflow pins that full Harbor reference in GitOps and Argo CD rolls tooling-lab-docs. Verify publish-marker.txt records the intended source_sha. The mini-host lab-docs-image-sync ferry cron is retired; scripts/build-image.sh is a manual fallback only (lab-monorepo#99). The image serves the showcase at / and this book at /agent-docs/ behind the docs front door. The legacy /lab-docs/ path prefix redirects to the root. The public GitHub Pages path below remains gated off.

Since 2026-10-24 the book is part of the same Astro build — an Astro content collection rendered inside the showcase’s own layout — so there is one bundle and one gate set, not two. (The book was a separate MkDocs build before that date; MkDocs is retired. The full story is tools/docs-system.md.)

On pull request (targeting main), push to main, and manual dispatch, build-deploy.yml on mini-k8s runners:

  1. Checkout.
  2. Gitleaks on source.
  3. Regenerate inventory.
  4. Sync inventory into site/.
  5. npm ci && npm run build (showcase + book, one bundle).
  6. Gitleaks on dist/.
  7. audit-bundle.py on dist/ — the whole bundle, book pages included; the audit’s vendor-prefix skips apply only to the theme’s own asset trees, never to authored pages.

The workflow still contains a force-push of dist/ to gh-pages of rhels/lab-docs, but that deploy job is gated off: it runs only on workflow_dispatch when repo variable DOCS_PUBLISH_ENABLED is true. PRs and pushes to main cannot publish. The public GitHub Pages path is unavailable / planned, not live:

  • rhels/lab-docs is private.
  • The rhels org is on the Free plan.
  • GitHub Pages is disabled.
  • https://rhels.github.io/lab-docs/ returns HTTP 404.

Publication is held pending an approved internal-host plan on rhels/team#108. Recommendation recorded there: Mini host Tailscale Serve. The human has not chosen an access model. Do not describe force-push to public gh-pages as the live or current publish path.

Agent-docs (agent-docs/) stay private and are never published.

See the root README for the pipeline description and secrets policy.