RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Tool pages

Prometheus & Alertmanager — metrics and alerts

Metrics exist on three tiers, batch results arrive through Pushgateways, and alerts deliberately page nobody yet — they surface in the Alertmanager and Grafana UIs, which is the owner’s standing choice (2026-10-24), not an oversight.

Part A — How it is set up

The three tiers

  • Tooling plane — a hand-rolled stack in the mini’s agentops namespace (GitOps from rhels/lab-agentops, gitops/modules/observability): Prometheus v2.55.1, Alertmanager v0.27.0, Grafana 11.3.1. Stateless by construction (no volumes — config in git, state in-pod). Front doors prom.rhels.com and alerts.rhels.com (no login; tailnet-only).
  • Workload cluster — kube-prometheus-stack on CRC (namespace monitoring), alongside OpenShift’s built-in monitoring. The scrape contract: the app exposes /metrics, a ServiceMonitor selects it, Prometheus scrapes it. Alertmanager and kube-state-metrics were enabled on 2026-10-24 — before that this stack had no CronJob metrics at all and its alerts went nowhere.
  • The Macs — node_exporter on each host, scraped by Grafana Alloy, remote-written to Grafana Cloud. This tier is the host ground truth (including the SSH-session counts that watch the macOS session cap); when a host “seems down”, these series — not a stale report — decide. See tools/grafana.md.

Pushgateways (for things that cannot be scraped)

CronJobs and one-shot jobs exit before a scrape would find them, so batch producers push:

  • Tooling plane: prom/pushgateway:v1.9.0 in the agentops stack. Its tenant is group-sync, which pushes per-group check results every 2 minutes (identity.md).
  • CRC: a Pushgateway in the monitoring namespace (Argo CD app from clusters/crc/observability/pushgateway/), scraped by the CRC Prometheus through a ServiceMonitor. Its tenants are the test producers — the log-login-tests and log-delivery-test CronJobs and the agent-side health suite — which PUT a per-run result set (per-check status, run counts, duration) at the end of each run. The exact metric model is documented in that app’s README.md in the GitOps repo, and rendered by the Lab Test Results Grafana dashboard.
  • Retention truth: the CRC Prometheus retention was raised from 24 h to 15 days for those trends, and since 2026-10-24 its TSDB sits on a 10 Gi PVC (a volumeClaimTemplate in the stack’s GitOps values) instead of the original emptyDir. History now survives pod restarts — proven the same day by deleting the pod and finding the pre-restart samples identical after — and is lost only with the CRC VM itself.

Alerting

  • CRC: PrometheusRule crc-cronjob-consecutive-failures (labelled so the stack’s Prometheus picks it up) carries one alert per check CronJob — LogDeliveryTestConsecutiveFailures and LoginTestsConsecutiveFailures. Each fires only after two consecutive failed runs; at the 3-hour cadence that means ~6 hours of real breakage and no single-flake noise. Deliberate dogfooding: the login tests verify the alerting stack’s own logins, and the alerting stack watches the tests.
  • Receivers: the default route lands in the CRC Alertmanager and is read in its UI and in Grafana (datasource wired 2026-10-24). No email/webhook receivers — owner’s pick; do not invent one.
  • Tooling plane: the agentops Alertmanager routes the group-sync signal; its escalation design keys off the failing_since timestamp group-sync records, so only long-lived group failures escalate.
  • Adjacent signal: Kyverno PolicyReports on CRC are read like any other cluster signal (see observability.md).

Part B — How to use it

  • Query metrics: the Prometheus UI (prom.rhels.com for the tooling plane; the CRC stack through Grafana Explore or its own route) — start from the metric, not the dashboard: check the series exists before wondering why a panel is empty.
  • See what is firing: Alertmanager UI (alerts.rhels.com, or the CRC Alertmanager via Grafana’s Alerting view). An alert listed there has already passed its consecutive-failure bar — treat it as real until a direct probe says otherwise.
  • Silence an alert: create the silence in the Alertmanager UI with an expiry and a comment naming the issue. Silences are the one accepted UI-only action here; an unexplained silence is a finding, not a fix.
  • Read test trends: the Lab Test Results dashboard in CRC Grafana (tools/grafana.md) — runs, passed, failed, per-check history. To verify a producer pushed, query the Pushgateway’s own /metrics or the CRC Prometheus for the run’s series before blaming the dashboard.
  • Add a scrape target: give the workload a /metrics endpoint and a ServiceMonitor in git (CRC) — never a hand-edited scrape config on the Prometheus pod.
  • Add an alert: a PrometheusRule in the GitOps repo carrying the stack’s release label, an expression that requires a sustained condition (the consecutive-failure pattern above is the house style), and a summary that says what is broken and where to look. Alert rules added in a UI do not exist, as far as this lab is concerned.