RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Planes

Workload Cluster

The workload cluster is OpenShift 4.22 (CRC) — the lab’s demo environment. All namespaces are managed by Argo CD via AppProjects and ApplicationSets; nothing is hand-applied.

Part A — How it is set up

Operators

NameVersionDetail
OpenShift GitOps1.21Argo CD 3.4.7. Manages all namespaces via AppProjects/ApplicationSets from rhels/local-k8s-gitops
Vault Secrets Operator—Delivers Vault KV entries into namespaces as Secrets (tenant pattern — see vault.md)
OpenShift Cluster Logging6.6Vector collectors + the ClusterLogForwarder feeding the log pipeline (see observability.md)

Demo apps

BSNLanguageNamespaceIAC repoHealth
app1001Javaapp1001-devrhels/IAC.app1001Healthy
app1002Pythonapp1002-devrhels/IAC.app1002Healthy
app1003Redis-backedapp1003-devrhels/IAC.app1003Healthy

Each app follows the IssueOps build: one thread per app, plan drafted by an agent, peer review, human authorization, then build, then verification evidence posted back on the issue.

Monitoring & logging

  • Prometheus via kube-prometheus-stack (with Alertmanager and kube-state-metrics since 2026-10-24), alongside OpenShift’s built-in monitoring.
  • Grafana for dashboards — OAuth-only sign-in (OpenShift → LDAP) since 2026-10-24 — with Prometheus in-cluster and Loki/Tempo datasources wired to the tooling-plane backends, plus Loki (pro) / Tempo (pro) against the pro fronts (live, rhels/team#107).
  • Log shipping is installed: Cluster Logging collectors forward the log-sim namespace to Logstash and to the Vector aggregator (log-agg), which fans out to the local Loki and Grafana Cloud Loki. A CronJob re-proves delivery every 3 hours. See observability.md.

Platform workloads (the agentops namespace)

Beyond the demo apps, CRC hosts the agent platform the lab is built around (Argo CD apps from rhels/lab-agentops):

  • MCP server — the agent-first front door for namespace provisioning (mcp.rhels.com); agents authenticate with an LDAP bind and work through the platform instead of holding cluster credentials.
  • group-sync — CronJobs projecting LDAP groups into OpenShift groups and Argo CD (see identity.md).
  • CMDB + ServiceNow simulation, and the PDI-backed request apps for the app1001 family: the ServiceNow developer instance (PDI) is the lab’s external backend for the namespace-request flow (registry, catalog, queue). Its admin credential is a Vault tenant (secret/lab/servicenow/pdi-admin).

Conventions

  • BSN format: appDDDD — seven characters, e.g. app1001.
  • One namespace per app-env: app1001-dev, app1002-dev, app1003-dev.
  • One private IAC repo per app: rhels/IAC.app1001, rhels/IAC.app1002, rhels/IAC.app1003.
  • Admin/view RBAC per cluster type on each namespace.
  • Sync waves: lowercased AppProject names; derived destination namespaces; controller-access shims at sync-wave -1 where gated.
  • Admission control: Kyverno require-signed-images (verifyImages) is installed on CRC in Audit for app* namespaces (rhels/team#107; GitOps c5d749f586d43640a61591f0bcdb94ac04d98e0d). Enforce (reject unsigned images at admission) is not live. See cicd.md.

Namespaces not owned by demo apps

The IssueOps build also created ns-bsn1234, ns-bsn1234-blue, and ns-bsn9999 with BSN metadata for blue/green and sandbox patterns. Demo-app namespaces stay appDDDD-dev; those special namespaces follow their own issue plans.

Part B — How to use it

  • Open the console: the console route under the cluster’s own route domain; sign in with OpenShift OAuth (a directory account — identity.md). The kubeadmin break-glass password is retrievable on the CRC host itself (crc console --credentials) and is for recovery, not daily use.
  • Reach the API: agents reach the cluster through the persistent SSH tunnel (networking.md) with the lab kubeconfig from the secure channel. Prove the tunnel end-to-end — the API’s /readyz answering is the test; a bound local port proves nothing (that failure mode happened here).
  • Check workloads: Argo CD on CRC (tools/argocd.md) for sync/health, or oc get pods -n <namespace>; per-app dashboards are in CRC Grafana (tools/grafana.md).
  • Change something: through the app’s IAC repo and Argo CD — never oc apply by hand. The AppProject whitelist decides which namespaces and kinds an app may touch; extending it is a commented, deliberate commit.
  • Cluster lifecycle: crc runs on its host (start/stop/status from the host shell). Starting or stopping the cluster is the owner’s call — CRC does not auto-start, and an intentionally stopped cluster stays stopped until the owner says otherwise. The host sleeps on battery (a multi-day disappearance traced to exactly that, 2026-10-21→24); if the cluster is unreachable, check the host’s power/sleep state before the cluster’s.