RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Tool pages

Harbor — container registry

Harbor is the lab’s enterprise registry: every pipeline image lands here, gets scanned, and is pulled from here. Front door: harbor.rhels.com.

Part A — How it is set up

  • Where: namespace harbor on the tooling cluster, installed by the Argo CD app tooling-harbor from the upstream chart (helm.goharbor.io, chart 1.19.2) with values in rhels/local-k8s-gitops.
  • Components: core, portal, registry, database, redis, jobservice, and the Trivy scanner. Storage is per-component PVCs: the registry volume is disposable (images re-push), the database volume is the precious one — projects, users, robot accounts and the auth configuration live there and are not in git (see disaster-recovery.md).
  • Authentication: LDAP mode against the lab directory (identity.md); the bind credential is in Vault (secret/lab/harbor/ldap-bind). The LDAP auth mode itself is Harbor-internal configuration set through Harbor’s own API/UI — the chart values deliberately do not carry it, which makes it a documented rebuild seam.
  • Admin credential: the built-in admin is break-glass only. Its password was rotated on 2026-10-24 off a default that had been committed to git in the early build; today the value lives in Vault (secret/lab/harbor/admin) and reaches the chart through a pre-created harbor-admin Secret (an existingSecret reference — no password in the values file). CI reads its Harbor password from a GitHub Actions secret that was re-cut from Vault the same day.
  • Pull path quirk: pods on the tooling cluster pull Harbor images through Docker Desktop’s registry mirror from the local image store, with imagePullPolicy: IfNotPresent. A re-used tag can be served stale by the mirror — when an image’s content changes, use a fresh tag.
  • Supply chain: CI pushes, Cosign signs (keyless, GitHub OIDC), Trivy scans — the pipeline state is in cicd.md.

Mirror rollout — 2026-11-01

GitOps #43 remains OPEN: Phases 0–3 are complete; only CRC IDMS/ITMS remains. There are nine proxy-cache projects and Kyverno harbor-image-rewrite mutate twins on mini and pro. The generate policy harbor-image-mirror-trigger creates check-first skopeo copy Jobs into nine private owned-<registry> projects, preserving digests, with per-cluster sweeper CronJobs. Mirror copier robots exist; their credentials are held only in Vault. Sources: PR #122, lab-agentops #77 and #80.

Rebuild checklist — the Harbor-internal state

2026-11-01 note: the project and robot counts below predate #43; they are historical rebuild observations, not current totals. The mirror copier robots and projects above supersede those counts in the #43 context.

Everything below was read from the live Harbor API on 2026-10-24 (v2.15.2); items 4 and 7 were re-verified on 2026-10-28 (see those items). None of it is in git: after the chart installs, re-enter it through the UI/API in this order. The one secret involved — the LDAP bind password — is not here; it lives in Vault (secret/lab/harbor/ldap-bind).

  1. Authentication mode: ldap_auth.
  2. LDAP settings:
    • URL: ldap://openldap.agentops.svc:389 (in-cluster service DNS)
    • Base DN: dc=agentops,dc=local
    • Search DN (bind user): cn=admin,dc=agentops,dc=local (bind password from Vault, above)
    • UID attribute: cn; search scope: subtree (2); filter: empty
    • Verify cert: off; timeout: 5 s
  3. LDAP group settings: group base DN ou=groups,dc=agentops,dc=local; group attribute name cn; search filter objectclass=groupOfNames; membership attribute memberof; scope subtree (2); group admin DN: empty — no LDAP group maps to Harbor administrator.
  4. Projects: two, both owned by the built-in admin (re-verified 2026-10-28; the Harbor API was unreachable that day — core and jobservice pods restarting — so this state was read from Harbor’s own database). library, public, is the pipeline project. Its repositories as of 2026-10-28 — a snapshot, not a fixed list: it grows as pipelines push (re-created by re-pushing): cmdb, cursor-agent-toolchain, demo-java-app, demo-python-app, demo-redis-app, group-sync, lab-docs, log-producer, mcp-server, snow-sim. lab-health (created 2026-10-25) holds the health suite’s images: python, service-health.
  5. Robot accounts: none — global or project-scoped (verified empty). Pipelines authenticate as the built-in admin or a directory user; the CI credential is held in Vault (secret/lab/harbor/ci-password) and mirrored into the workflow’s Actions secret.
  6. Users: no local users besides the built-in admin; everyone else materializes from LDAP on first login.
  7. Settings that differ from a fresh install: the auth/LDAP block above, and the external URL harbor.rhels.com:443, served through the nginx ingress. A NodePort Service (80:30002) also exists in-cluster, but its host forwarding is closed — probes from the mini host on 2026-10-27 and 2026-10-28 both got connection refused — so :30002 is not a working pull path and nothing should be configured or documented as pulling through it. The rest matches upstream defaults (self-registration off, project creation open to everyone, token expiration 30 minutes, robot token duration 30 days, per-project quota enabled).

Part B — How to use it

  • Log in: harbor.rhels.com with a directory account (LDAP). Use the built-in admin only when LDAP is down, and say so on the issue when you do.
  • Pull / push from a shell: docker login harbor.rhels.com (directory credentials), then normal docker pull / docker push against harbor.rhels.com/<project>/<image>:<tag>.
  • Browse: projects group repositories per pipeline/app; a repository’s artifact list shows digests, tags, and the Trivy vulnerability summary per artifact — that summary is the first place to look when a pipeline’s scan step complains.
  • Robot accounts (2026-11-01): #43 mirror copier robots exist, with credentials only in Vault (see the rollout note in Part A). For other consumers, give each a robot account scoped to its project; runbook 4 is the create/rotate procedure, and the robot’s secret belongs in Vault (and, for GitHub Actions consumers, in the repo’s Actions secrets) — never in a workflow file.
  • When a pull fails: check in order — is the tag fresh (mirror staleness, above)? Does the project/robot permission cover this repository? Is the artifact’s digest the one the IAC repo pins? ImagePullBackOff triage is runbook 7.