RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Operations

Runbooks

Each runbook assumes you hold a valid kubeconfig (from the secure channel — never from this repo) and that connectivity checks stay read-only. Substitute <bsn> (e.g. app1001) and <env> (e.g. dev) as needed.

1. Check app health

  1. List pods in the app’s namespace: kubectl -n <bsn>-<env> get pods.
  2. All pods Running and READY 1/1 → healthy.
  3. Cross-check Argo CD: the Application for the app shows Healthy and Synced (see runbook 8).
  4. If anything is not Running, jump to runbook 7.

2. Re-sync an Argo CD Application

  1. Identify the Application name (convention: derived from BSN; check inventory.json → workload_cluster.apps).
  2. Hard refresh to detect drift: argocd app get <app> --hard-refresh (or trigger refresh via the API/CLI you have).
  3. Sync: argocd app sync <app>.
  4. Watch until Health: Healthy, Sync: Synced.
  5. If sync fails, do not retry blindly — read the sync result message, then see runbook 10 (roll back) if the new state is bad.

3. Read app logs

  1. kubectl -n <bsn>-<env> logs deploy/<name> --tail=200 for the current logs.
  2. For a crashed pod: kubectl -n <bsn>-<env> logs <pod> --previous.
  3. For the log-sim namespace, logs are also forwarded: query Loki from Grafana Explore (local Loki for the tooling plane + forwarded streams, Grafana Cloud Loki for the aggregator’s output) with LogQL such as {log_type="application"}. Remember the local Loki holds aggregator-path events twice by design — count distinct events, not raw lines (see observability.md).
  4. Other CRC namespaces are not forwarded yet; their logs are per-pod and ephemeral. Capture anything you need into the issue thread before the pod is recycled.

4. Rotate a Harbor robot account

Harbor is live on the tooling plane (rhels/team#107). Creating or rotating robot-account credentials remains a human-only secrets step — see agent-protocol.md.

  1. In Harbor, create a new robot account with the minimum scopes the CI needs (push/pull on the app’s project).
  2. Update the CI credential in the secure store (human step — see agent-protocol.md: human-only secrets steps).
  3. Trigger a test build; confirm push succeeds.
  4. Delete the old robot account only after the new credential is proven in CI.
  5. Record the rotation date on the tracking issue; never record the credential.

5. Add a new app appDDDD

  1. Pick the next free BSN (appDDDD, 7 chars, zero-padded).
  2. Create the private IAC repo rhels/IAC.<bsn> (human or org admin step if you lack permissions).
  3. Add the namespace manifest <bsn>-dev, RBAC (admin/view per cluster type), and the ApplicationSet entry in rhels/local-k8s-gitops following the existing apps’ shape (lowercased AppProject, derived destination namespace, controller-access shim at sync-wave -1 where gated).
  4. Open a GitHub issue with the plan; get peer review; get human authorization.
  5. Build, let Argo CD sync, verify Healthy/Synced, post evidence on the issue.
  6. Regenerate inventory.json so the new app appears.

6. Rotate a secret

  1. Identify every consumer of the secret (CI workflows, Argo CD, app pods) and its Vault path (see vault.md for the layout).
  2. Create the replacement value via the owning system (human step for credentials — agents never invent or display secret values).
  3. Update the KV entry in Vault (agent write gate: the owner-placed token file + the in-pod CLI — see vault.md). Consumers wired as Secrets Operator tenants pick the new value up on the secret’s refresh interval; roll any other consuming pods (kubectl -n <ns> rollout restart).
  4. Verify consumers work with the new value (test build / app health / the health suite’s login checks where the secret is a login).
  5. Revoke the old value at the issuer.
  6. Never write the value into git, chat logs kept in the repo, or these docs.

7. Diagnose ImagePullBackOff

  1. kubectl -n <bsn>-<env> describe pod <pod> — read the Failed to pull image event for the exact error.
  2. Common causes, in order:
    • Wrong image reference or tag in the IAC repo → fix the repo, let Argo CD sync.
    • Registry unreachable from the cluster → check networking (runbook: private tailnet path); report if the tunnel is down.
    • Missing imagePullSecret → add it in the IAC repo.
    • Kyverno require-signed-images is Audit today (rhels/team#107; GitOps c5d749f586d43640a61591f0bcdb94ac04d98e0d); it reports unsigned images in PolicyReports and does not block pulls. After the Enforce flip, an unsigned image rejected at admission means the pipeline must sign the image — do not bypass the policy, fix the signing step. Do not treat Enforce as live.
  3. Fix forward in git; never kubectl set image by hand — the change must live in the IAC repo or Argo CD will revert it.

8. Check Argo CD sync status

  1. List Applications: argocd app list (or the CRDs: kubectl get applications -n <argocd-namespace>).
  2. For each app: Sync: Synced and Health: Healthy is the steady state.
  3. OutOfSync → inspect the diff (argocd app diff <app>); if the drift is unexpected, re-sync (runbook 2) or fix the repo.
  4. Degraded/Unknown health → treat as an incident: check pods (runbook 1), logs (runbook 3).

9. Verify inventory freshness

  1. Read inventory.json: check generated_at and source.
  2. source: "live" and recent → trust it.
  3. source: "recorded" or stale → re-run python3 inventory/generate-inventory.py from the repo root; it will attempt a live read and fall back to recorded.
  4. After regenerating, diff against the previous file and note what changed on the relevant issue or in chat. Copy the result to agent-docs/docs/inventory.json.

10. Roll back a bad sync

  1. Confirm the app is unhealthy after a sync (runbook 1) and the cause is the new state, not the platform.
  2. In the app’s IAC repo, revert the offending commit (git revert), push.
  3. Argo CD syncs the revert automatically; or force with argocd app sync <app>.
  4. Verify Healthy/Synced; post the revert and the evidence on the issue.
  5. Open a follow-up issue for the root cause — do not re-land the change without a reviewed plan.

11. Unseal Vault after a tooling-host reboot

Vault seals at every tooling-host boot; nothing is broken — this is the sequence (background: vault.md).

  1. Owner step (human only): on the tooling host, run the Vault bootstrap, which takes the unseal key from the owner’s Keychain. Agents never hold the unseal key and cannot do this step.
  2. Agent verification: check Vault status from inside the pod — the CLI runs via the tooling cluster (kubectl -n vault exec); sealed must read false.
  3. Check the Secrets Operator tenants recover: on CRC, the production VaultStaticSecrets report SYNCED=True again (the two proof tenants’ SYNCED=False is a known, separate quirk).
  4. If an agent needs the write gate, confirm the owner has re-placed the token file; do not ask for the token in chat.
  5. Re-run the health suite: Vault-dependent logins and the delivery check should return to green.

12. Verify log delivery end to end

The standing proof that the pipeline delivers to both Lokis (background: observability.md).

  1. On-demand: from the lab-agentops checkout, run the suite’s single check: run_suite.py --check grafana-cloud-log-delivery (same --env the hourly run uses).
  2. PASS means: 10 marker events emitted, and each Loki — local and Grafana Cloud — holds all 10 by distinct sequence count.
  3. Raw local counts near double are expected (dual local paths), not a failure signal.
  4. Scheduled proof: the log-delivery-test CronJob in log-sim runs the same check from inside CRC every 3 hours; read its latest job’s logs before digging by hand.
  5. If it fails, reconcile per-hop counters (collector → aggregator sinks → Loki) per the trial recipe in observability.md — the hop where the distinct count shrinks is the loser.

13. A lab CronJob check failed — what now

  1. Identify the job run: kubectl -n log-sim get jobs for the delivery test; the login-test CronJob reports the same way in its namespace.
  2. Read the failed job’s logs — the check prints per-target results; separate “the target failed” from “the test path failed” (tunnel, forwards, image) before concluding anything about the target.
  3. One failure is signal, not incident: the alerting rule (landing — see observability.md) fires only on the second consecutive failure.
  4. Cross-check with the hourly health suite’s latest report and fresh direct probes; the suite and the CronJobs measure overlapping paths and disagreeing results are themselves diagnostic.
  5. File/fix in git like any other lab change; never patch the CronJob in place.