RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Tool pages

Argo CD — GitOps on both planes

Argo CD is the only mechanism that changes cluster state in this lab. If it is not in git, it does not run; if Argo did not sync it, it is drift to be explained. There are two independent installs, one per plane, and confusing them is the classic beginner error here.

Part A — How it is set up

Tooling plane (Mac mini)

  • Where: namespace argocd on the tooling cluster; front door argo.rhels.com.
  • Login: Dex fronting OpenLDAP (“Log in via OpenLDAP” — identity.md). The local admin exists as break-glass; its initial secret is recorded in Vault (secret/lab/argocd/mini-initial-admin).
  • Layout: root-app pattern from rhels/local-k8s-gitops (argocd/applications/root.yaml + one Application file per app in argocd/applications/). Most apps sync paths inside local-k8s-gitops itself; the agentops-* apps sync rhels/lab-agentops modules (LDAP, group-sync, observability, MCP server, snow-sim, CMDB).
  • Sync model: automated with prune + self-heal on the platform apps. Repository credentials for the private repos are delivered from Vault (secret/lab/argocd/repositories-* paths).

Workload cluster (CRC / OpenShift GitOps)

  • Where: OpenShift GitOps operator 1.21 (Argo CD 3.4.7) in the openshift-gitops namespace; front door is the OpenShift route under the cluster’s own route domain.
  • Login: “Log in via OpenShift” — OpenShift OAuth with LDAP behind it. Local login is disabled for the platform tenants (OAuth-only since 2026-10-24); recovery is via the cluster, not a password.
  • Layout: bootstrap root application in rhels/local-k8s-gitops (clusters/crc/openshift-gitops/bootstrap/), then per-area apps (observability, logging, vault-access, security, the agentops platform from rhels/lab-agentops gitops-crc/). Apps live in AppProjects (crc-gitops, crc-agentops, …) whose destination whitelists are the real permission boundary — a new namespace or kind needs a commented, deliberate project change.
  • Controller-access shims: apps that create namespaces or cluster-scoped objects carry an argocd-controller-admin binding at sync-wave −1 (the pattern the login-tests app established); without it the controller is forbidden from creating the app’s own service accounts.

Shared rules

  • Sync waves order platform pieces (foundation at −1, tenants after).
  • A failed PostSync hook means the sync did not finish — check hook Jobs before believing “Synced” (the LDAP seed is the standing example, identity.md).

Part B — How to use it

  • Check an app: open the plane’s UI, find the app, read Sync/Health and the last sync operation — or kubectl -n argocd get application <app> / kubectl -n openshift-gitops get application <app> and read .status. Runbook 8 is the checklist.
  • Make a change: edit git, push, watch the app go OutOfSync → Synced/Healthy at the new revision. That revision SHA is the evidence a change landed — quote it, not “it should be fine”.
  • Refresh vs sync: a hard refresh re-reads git (use it when Argo seems to be looking at a stale revision); a sync applies. Automated apps normally need neither — reaching for the button first is a smell; find out why automation did not do it.
  • Roll back: runbook 10 — Argo’s history view redeploys a prior revision, but the durable fix is a git revert; a UI rollback without the revert is undone by the next sync.
  • Add an app: runbook 5 (demo apps) or copy the nearest existing Application file (platform apps). New destination namespace or resource kind ⇒ extend the AppProject whitelist in the same commit, with a comment naming why.
  • Read pod logs through Argo: the resource tree → pod → Logs is the quickest path when you do not have cluster shell access (runbook 3 has the kubectl equivalent).