Observability

Now (verified 2026-10-24)

Metrics — two stacks plus the cloud

Tooling plane — a hand-rolled stack in the agentops namespace (GitOps from rhels/lab-agentops): Prometheus, Grafana, Alertmanager. Front doors prom.rhels.com, grafana.rhels.com, alerts.rhels.com (see certificates.md).
Workload cluster — kube-prometheus-stack on CRC (monitoring namespace), alongside OpenShift’s built-in monitoring. Service monitoring contract: the app exposes /metrics, a ServiceMonitor selects it, Prometheus scrapes it. Alertmanager and kube-state-metrics were enabled on 2026-10-24 — before that, this stack had no job/CronJob metrics at all and its alerts went nowhere.
The Macs themselves — node_exporter on each Mac, scraped by Grafana Alloy, remote-written to Grafana Cloud. This is the lab’s host ground truth (including the SSH-session counts that watch the macOS session cap); when a host seems down, its series here — not a stale report — decide.

Grafana — three surfaces

CRC Grafana (grafana namespace, GitOps at clusters/crc/observability/grafana/). OAuth-only since 2026-10-24: sign-in is “Sign in with OpenShift” (LDAP behind it — identity.md); the local login form and local admin are disabled, and break-glass is grafana-cli in the pod. Admin and OAuth credentials reach it from Vault via a Secrets Operator tenant (see vault.md). Provisioned datasources: Prometheus (default, in-cluster), Loki and Tempo (tooling plane), Loki (pro) and Tempo (pro) (pro fronts, dual-forward), and the CRC Alertmanager (wired 2026-10-24). Datasource provisioning applies only at Grafana startup; the gitops tree builds that ConfigMap with a configMapGenerator so changes roll the pod (full detail: tools/grafana.md). Provisioned dashboards: Demo Java App (request rate, error rate, latency p50/p95, JVM heap), Demo Python App (request rate, error rate, latency p50/p95), Demo Redis App (app request rate, Redis commands/s, latency p95, connected clients/memory), and Lab Test Results (uid lab-test-results — runs/passed/failed trends for the test producers below, fed by a Pushgateway in the monitoring namespace; the CRC Prometheus retention is 15 days and its TSDB sits on a 10 Gi PVC (since 2026-10-24), so the trend horizon is a rolling 15 days that survives pod restarts). The metrics/alerting reference is tools/prometheus-alertmanager.md.
Tooling-plane Grafana (in the agentops stack above): dashboards for the tooling plane itself. Local admin is retained as break-glass and anonymous Viewer is on — a lockdown decision is still open, and its admin credential is not yet recorded in Vault.
Grafana Cloud (stack ai0) — hosted metrics for the Macs above, hosted Loki (the log pipeline’s second sink, below), and the Mac Health page (latest values plus CPU/memory/load/disk/SSH-session charts for both Macs). Access is by scoped tokens only, one per job: a logs-write token for the sink, a logs-read token for delivery verification, a metrics-read token for the health checks. Token names are fine to write down; values live in Vault (secret/lab/grafana-cloud/…).

Logging — the pipeline as built

The 2026-10-11 trial (lessons below) graduated into a standing pipeline:

Collection — OpenShift Cluster Logging (Vector collectors) with a ClusterLogForwarder whose application input covers the log-sim namespace. That scope is deliberate and still the whole scope: cluster-wide forwarding is a later step, not a gap in disguise.
Two paths from the forwarder:
- Logstash (tooling plane, observability namespace) → the local Loki, and a plain-text file archive (cluster/namespace/pod/container/log.log on a PVC), mirroring the office pattern.
- The Vector aggregator — a StatefulSet in CRC’s log-agg namespace whose config is one file per concern and one file per sink, so a sink is added or removed as exactly one file in git. End-to-end acknowledgements are on; every sink has a disk buffer that blocks rather than drops when full.
The aggregator’s sinks:
- 00-metrics — its own Prometheus exporter;
- 10-loki — the local Loki on the tooling plane;
- 15-logstash-fallback — Logstash over a dedicated bridge, so the archive path survives even when reached indirectly;
- 20-grafana-cloud-loki — Grafana Cloud hosted Loki, authenticated with the write-only token delivered from Vault.
Labels are exactly log_type, log_source, hostname, level — the allowlist discipline from the trial lessons below is load-bearing, not decoration.

One consequence to internalize: aggregator-path events land in the local Loki twice by design (direct via sink 10, and again via the Logstash fallback). Raw line counts lie; distinct producer-sequence counts are the truth — the verification below counts that way.

Loki (tooling plane): upstream chart, SingleBinary, filesystem storage (10 Gi PVC); max_global_streams_per_user 50000; retention 72 h (compactor). Query it from Grafana Explore with LogQL, e.g. {log_type="application"}. Tempo (tooling plane): tracing backend live, Grafana datasource wired; demo apps are not instrumented yet, so there are no real spans to query.

Delivery verification — the pipeline proves itself

Sink proof (2026-10-24): a 27-event marker probe reconciled exactly — 27 to the local Loki, 27 to Grafana Cloud, 0 discarded — and, once a logs-read token existed, the cloud side was read back: exactly 27 entries. Send-acked and receive-verified.
Health-suite check grafana-cloud-log-delivery: emits 10 marker events and polls both Lokis until each holds all 10 (distinct-seq). Runs with the hourly suite; also triggerable on its own (runbook 12).
In-cluster CronJob log-delivery-test (namespace log-sim, every 3 hours at :17): the same proof with no agent VM in the loop — the job emits and queries from inside CRC, its read token delivered by the Secrets Operator. Live and synced; manual runs from the CronJob template pass end to end (10/10 distinct sequences in both backends), and it also pushes its result to the Pushgateway for the Lab Test Results dashboard.

The health suite

An hourly suite (source: rhels/lab-agentops, health-suite/) drives real browser logins through the tooling stack — it is the standing proof for identity.md’s consumer table and for delivery above. Nine checks: console LDAP login, console wrong-password rejection, Argo CD via OpenShift, Argo CD OAuth-only lockdown, Grafana via OpenShift, Grafana OAuth-only lockdown, tooling-plane Argo CD via Dex/LDAP, Harbor LDAP auth (API), and the log-delivery check.

The eight login checks also exist as an in-cluster CronJob in a dedicated login-tests namespace (every 3 hours at :47, a headless Chromium in the official Playwright image, credentials delivered by a Secrets Operator tenant): the same flows with no agent VM or tunnel in the path. The Argo CD app is Synced/Healthy and a manual run from the CronJob template passed 8/8 in ~72 seconds on 2026-10-24 — the in-cluster path is the fast, direct one; the hourly suite remains the through-the-tunnel cross-check, and both push their results to the Pushgateway (the suite’s push is best-effort).

Two operating lessons are baked into how results are read:

The suite measures through the tunnel, so a bad run can be a bad measurement path, not a bad app. On 2026-10-24 the morning run was 9/9 while afternoon runs failed on navigation timeouts — with every app simultaneously verified healthy by direct probes. Read failures against fresh direct evidence before declaring an app down.
Its tooling-plane probes ride port-forwards that can go stale after a host reboot; stale probes report infra-down, which is a suite problem to fix, not a lab outage.

Alerting

Tooling plane: Alertmanager (in the agentops stack) is the router; its front door is alerts.rhels.com.
CRC: Alertmanager was enabled on 2026-10-24, and the owner’s receiver pick landed the same day: Alertmanager + Grafana UI only — no email or webhook receivers, and none should be invented.
Landed 2026-10-24: PrometheusRule crc-cronjob-consecutive-failures fires when either GitOps CronJob check (log delivery, login tests) fails twice in a row — at the 3-hour cadence that means ~6 hours of breakage, no single-flake noise. Deliberate dogfooding: the login tests verify the alerting stack’s own logins, and the alerting stack watches the tests. The reference page is tools/prometheus-alertmanager.md.
Kyverno — on CRC (kyverno namespace, chart 3.9.1, Argo CD-managed). require-signed-images runs in Audit mode; decisions surface as PolicyReports — read them like any other cluster signal.

Tempo — memory sizing (verified 2026-10-31)

Rule: size memory limits for the flush-peak cgroup total (anon + dirty page cache), not quiet-time RSS. Ballast must be 0; an idle RSS sample leaves a latent OOM.
Observed (lab-monorepo#92): on pro, the chart default memBallastSizeMbs: 1024 rendered -mem-ballast-size-mbs=1024 against a 512Mi limit. WAL block f107a83b had never completed; traces/ on the PVC had been empty since Oct 28. Completion retried after each start. On slow local-path storage, the cgroup hit the 512Mi ceiling: OOMKilled (exit 137), or reclaim stalls froze /ready and triggered liveness-probe kills. The ballast-only fix left quiet RSS at 131 MiB, but live restarts rose 0->6+: ballast alone did not fix sizing.
Landed 2026-10-31: rhels/local-k8s-gitops#107 (merged 5a63d298) set tempo.memBallastSizeMbs: 0 and raised limits.memory from 512Mi to 2Gi. Block f107a83b completed and flushed, the first block on that PVC; WAL fell 5.0M -> 860K. The pod stayed Ready with restarts=0 through a 30m23s soak; /ready returned 200 x6, and a fresh OTLP span was found by trace ID.
Short version: see the header of clusters/pro-desktop/argocd/applications/pro-tempo.yaml.

Logging: measured results and lessons (2026-10-11, `rhels/team#115`)

Ceiling — the direct pipeline is lossless to 5,000 lines/s of 1 KB lines (≈5 MB/s, ≈430 GB/day). Above that, loss sits at the Vector collection hop on CRC (per-line tail/parse CPU saturation), not at Logstash or Loki. Pushing past ~5–6k lines/s is collector-side work (CPU, disk-backed sink buffer, batching, sharded collection) — not more sink capacity. Expect ~+2 duplicate entries per run in Loki (Vector at-least-once delivery); distinct-sequence counting is the truth.
Label discipline is load-bearing — logstash-output-loki with an empty include_fields turns every event field into a Loki label; per-line timestamps created ~10.7k junk streams, hit the stream cap, and Loki 429s were dropped silently. Keep the include_fields allowlist (log_type, log_source, hostname, level) paired with any stream-limit raise.
Three buffers, one chain — every log pipeline here is a chain of buffers: (1) the container log files on the node, (2) the collector’s sink buffer, (3) any aggregator’s / the sink-side buffer. An event sits in one buffer at a time; each buffer guards a different hop, and the chain loses at whichever buffer sits in front of the real bottleneck. A downstream buffer cannot rescue a slow reader.
Aggregator tier = insurance, not a leak (A/B, lab) — replicating the office topology both ways: a Vector StatefulSet ×3 with a disk buffer between collector and Logstash discarded zero events at 5k/7.5k/10k lines/s and raised delivery vs direct (100/100/88.1% vs 88.5/74.2/61.3%). Its costs are buffer-drain latency under overload and connection-level (not event-level) load pinning across replicas. The lab aggregator was torn down after the test; manifests live in the trial kit. (The standing aggregator described above is the graduated, single-StatefulSet form of this experiment.)
Kubelet rotation trap — CRI container logs are rotated by kubelet: 0.log (live) plus timestamped files; older rotated files are gzip-compressed, only the latest rotated file stays plain — a one-file grace period for the collector. Vector’s file reading cannot decompress .gz. With containerLogMaxSize 50 MB × containerLogMaxFiles 5, a hot container’s window is ~250 MB — seconds at MB/s write rates — and a collector that falls two rotations behind loses whole files silently (no error, no counter). Size the rotation window to the collector’s worst-case read lag, and check whether CRI-O’s own log_size_max is set (it can override kubelet’s settings).
Finding a losing hop — reconcile per-hop counters over the same window (source received → sink sent → next tier received → … → distinct-count at the final sink). The hop where the count shrinks, with its component_discarded_events_total climbing, is the loser. Recipe: trial kit RUNBOOK in rhels/team.

Planned / proposed

Collector tuning only if more than ~5 MB/s is ever needed (see ceiling).
TLS/auth on the Logstash HTTP input before any exposure wider than tailnet-only; today it is unauthenticated and acceptable only because access is tailnet-only.
Producer image into Harbor if the trial pattern graduates (today it is a local, no-pull image on CRC).
Trace instrumentation for the demo apps (Tempo is ready to receive).
Kyverno require-signed-images Audit → Enforce (rhels/team#107 final mile).
A third log sink (Datadog/Splunk) exists only as inert example files in the pipeline repo — no decision taken; Grafana Cloud is the second sink.

What an agent should check

Grafana dashboards for the app’s namespace (request/error rate, latency, pod restarts, JVM/Redis panels where applicable).
Loki via Grafana Explore LogQL — the local Loki holds tooling-plane logs and the forwarded log-sim streams; Grafana Cloud Loki holds the aggregator’s output. For pipeline health, reconcile counters per the trial RUNBOOK recipe, and prefer the delivery check (runbook 12) over eyeballing.
Tempo via Grafana Explore once apps are instrumented.
Kyverno PolicyReports in app namespaces for admission decisions.
Prometheus alerts on the workload cluster — a red app is either a bad sync, a bad image, or a resource problem; see the runbooks for each.
The health suite’s latest report before believing any single probe — and fresh direct evidence before believing the suite (see the suite notes above).

Gaps (known)

Forwarding scope: only the log-sim namespace is forwarded from CRC; cluster-wide CRC log forwarding is not enabled.
Logstash HTTP input is unauthenticated (tailnet-only access mitigates).
Collector ceiling ≈5,000 lines/s as configured (see lessons).
No trace instrumentation in demo apps yet.
Alerting routes exist but page nobody: by the owner’s 2026-10-24 choice, alerts surface in the Alertmanager and Grafana UIs only — an alert nobody looks at is a tree falling, so the UIs are part of the daily glance until the owner picks a louder channel.