RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Tool pages

Grafana — dashboards & Explore

Grafana is where the lab looks at itself. There are three surfaces, and knowing which one holds what you need is most of the skill.

Part A — How it is set up

CRC Grafana (the platform surface)

  • Where: namespace grafana on CRC, GitOps from rhels/local-k8s-gitops (clusters/crc/observability/grafana/); front door is the cluster route under the CRC route domain.
  • Auth: OAuth-only since 2026-10-24 — “Sign in with OpenShift” (LDAP behind it, identity.md); the local login form and local admin are disabled. Break-glass is grafana-cli inside the pod. Its admin and OAuth credentials are delivered from Vault by a Secrets Operator tenant (secret/lab/grafana/crc-admin, secret/lab/grafana/crc-oauth — vault.md).
  • Provisioned, not clicked: datasources and dashboards come from git (a dashboards provider + JSON files). Datasources: Prometheus (default, in-cluster), Loki and Tempo (tooling plane), Loki (pro) and Tempo (pro) (the pro observability fronts, dual-forward), and the CRC Alertmanager (wired 2026-10-24). Grafana applies datasource provisioning only at process startup, so the tree builds that ConfigMap with a kustomize configMapGenerator: any content change renames it and rolls the pod (learned the hard way in rhels/local-k8s-gitops#115 — a plain ConfigMap update silently never loaded). Dashboards include the three demo apps and Lab Test Results (uid lab-test-results), which renders the test producers’ Pushgateway metrics (see tools/prometheus-alertmanager.md).
  • Its database volume holds UI state only; everything that matters re-provisions from git.

Tooling-plane Grafana (the mini surface)

  • Where: inside the agentops observability stack on the mini (Grafana 11.3.1, hand-rolled manifests in rhels/lab-agentops, gitops/modules/observability); front door grafana.rhels.com.
  • Auth: anonymous Viewer is on and a local admin is retained as break-glass. Two honest gaps: the lockdown decision is still open, and this Grafana’s admin credential exists only inside Grafana itself — it is not yet recorded in Vault.

Grafana Cloud (stack ai0)

  • Hosted metrics for the two Macs (node_exporter → Alloy → remote-write) and the log pipeline’s second Loki. The Mac Health page (latest values plus CPU / memory / load / disk / SSH-session charts for both Macs) is the lab’s host ground truth.
  • Access is by scoped tokens only, one per job (logs-write for the sink, logs-read for verification, metrics-read for checks); values live in Vault (secret/lab/grafana-cloud/…).

Part B — How to use it

  • Sign in: CRC Grafana — “Sign in with OpenShift” with a directory account. Mini Grafana — grafana.rhels.com (Viewer is open; admin only for break-glass). Cloud — the owner’s Grafana account; agents use the scoped read tokens instead.
  • Find a dashboard: the Dashboards browser groups by folder; the ones you will actually open are the demo-app dashboards (request rate, error rate, latency percentiles), Lab Test Results (runs / passed / failed trends for the lab’s test CronJobs and the health suite), and, in Cloud, the Mac Health page.
  • Query logs: Explore → Loki datasource → LogQL, e.g. {log_type="application"}. Remember the local Loki holds the aggregator’s events twice by design (direct sink + Logstash fallback) — for counts that matter, count distinct producer sequences or use the delivery check (runbook 12), not raw lines. Pipeline detail: tools/logging-pipeline.md.
  • Check whether an alert is firing: the Alerting section (CRC Grafana reads the CRC Alertmanager through its datasource) or the Alertmanager UI itself — the owner chose UI-only notification on 2026-10-24, so looking is the notification path today.
  • Add or change a dashboard: edit the JSON in the GitOps repo and let provisioning apply it. A dashboard built in the UI and not committed is a draft, not a change — it will not survive a rebuild and nobody else can review it.

The Service Status board (pro Grafana)

  • What: the Service Status dashboard (grafana.rhels.com/d/service-status) — one tile per lab service: the 14 Service Directory entries plus the platform services the directory does not list yet (Gitea, Loki, Tempo, Logstash, Kyverno, the front door, CI runners, agent-memory, the E2E health service). Built for lab-monorepo#78’s monitoring standard; the JSON lives in rhels/lab-agentops, gitops-pro/modules/observability/.
  • The three feeds (a tile is only as true as its feed): live — pro Prometheus scrapes the service’s own /metrics (Grafana, Prometheus, AlertManager, the MCP server, Loki, Tempo, Kyverno); grade — the health and QA suites’ latest committed verdicts for that service; none — the tile honestly reads NOT INSTRUMENTED instead of disappearing. Memory MCP metrics are now available (2026-11-01, lab-monorepo#101); the Service Status tile feed classification is not established by that evidence. Harbor and Gitea are grade-fed on purpose: Harbor’s chart metrics are not enabled, and Gitea’s /metrics sits behind sign-in view.
  • Grades path: suite runs commit their result JSON to rhels/lab-agentops (health-suite/reports/); the publish-grades workflow on the self-hosted runners parses the newest run per suite and pushes the checks to the pro Pushgateway as service_check (0 fail / 1 pass / 2 pending) with the run’s own timestamp. A re-push therefore can never make an old grade look fresh: past its window (2 h for the hourly suites, 36 h for the daily QA suites) a tile flips to STALE, never stays green.
  • Reading it: a tile shows the grade state, the live state where a scrape exists, and the grade’s age in minutes; the Host variable (mini / pro / crc) follows the host-label standard, and a service that does not belong to the selected host shows No data rather than STALE.

Memory MCP — Usage (2026-11-01)

The Memory MCP — Usage dashboard (uid memory-mcp-usage) has panels 9–14 covering reads versus writes, entries by status, zero-result rate, curation funnel and store health. The four loaded alerts are MemoryNoWrites24h, MemoryAuthFailures, MemoryDropsBacklog and MemoryExportStale. lab-monorepo#101 is closed; this evidence does not establish a change to the Service Status tile feed.