QA Use Cases — Logging (Loki + Tempo + Logstash)

Tool page: tools/logging-pipeline.md; deep dive: observability.md. State after the 2026-10-28/29 migration (lab-monorepo#41/#42): Loki, Logstash, and Tempo run on pro behind dedicated tailnet fronts, proven standalone by scripts/check-pro-logging.sh. CRC’s Vector aggregator dual-forwards to both the mini and pro stacks (git-merged, PR #61) — the dual-delivery proof and the mini stack’s retirement happen at CRC’s next start, per the CRC-start checklist. Until then the mini stack still runs: pro is the target stack, the mini stack the not-yet-retired copy.

Admin stories

LOG-A1 — Pro stack acceptance probe [CLI]

• Persona: Platform admin. Goal: the whole pro logging stack passes its staged acceptance in one run.
• Steps: run scripts/check-pro-logging.sh from the mini.
• Expected: all stages OK — node ready, apps Synced/Healthy, Logstash ready, Loki ready, Tempo ready, and the push→query roundtrip returns the probe’s unique marker (source: pro logging standup record, PR #60).
• Assert: script exit code 0; the marker string appears in the Loki query result inside the probe window.

LOG-A2 — Retention and storage shape [config/API]

• Persona: Platform admin.
• Steps: read the live Loki config from the pro app (chart values in git).
• Expected: 72 h retention (compactor), SingleBinary, filesystem storage on a 10 GiB PVC — the documented local-Loki contract carried to pro (source: logging-pipeline.md; pro standup record: chart 7.3.0, fresh 10 GiB PVC, no data migration).
• Assert: rendered values: limits_config.retention_period == "72h"; topology == SingleBinary; PVC size == 10Gi.

LOG-A3 — Label allowlist intact [config/API]

• Persona: Platform admin. Goal: the load-bearing label discipline has not regressed.
• Steps: query Loki’s label set on pro for a recent window.
• Expected: stream labels are exactly the allowlist log_type, log_source, hostname, level. Extra labels are the documented failure mode (~10.7k junk streams, silent drops) (source: logging-pipeline.md — “Labels are exactly…”).
• Assert: GET /loki/api/v1/labels set == {log_type, log_source, hostname, level} (plus Loki’s own internal labels, enumerated in the check manifest).

LOG-A4 — Dual-forward configuration staged in git [config/API]

• Persona: Platform admin. Goal: when CRC starts, both delivery paths exist without further change.
• Steps: read the aggregator config sources in git: sinks 11 (Loki → pro) and 16 (Logstash → pro) beside the mini sinks 10/15; the ClusterLogForwarder’s logstash-pro output; CRC Grafana’s Loki (pro) / Tempo (pro) datasources; aggregator PVC template at 3 Gi.
• Expected: all present, additive only, Grafana Cloud sink untouched. The live claim expansion and dual-delivery proof are CRC-start items — explicitly pending, not failures (source: CRC-start checklist, [cutover] items; PR #61).
• Assert: file/content checks against the git tree: sink files 11-* and 16-* exist with pro front targets; CLF outputs contain logstash-pro; the PVC template requests 3Gi. (This story’s live half joins LOG-U3 at CRC start.)

LOG-A5 — Fronts reachable from the mini [CLI]

• Persona: Platform admin.
• Steps: from the mini, reach pro’s Logstash, Loki gateway, and Tempo fronts.
• Expected: all three answer; each is the Tailscale Serve → LoadBalancer pattern (the ACS front pattern), dedicated port per service (source: #41 standup record).
• Assert: Loki GET /ready → 200; Tempo GET /ready → 200; Logstash front accepts a TCP connection and a test event appears in Loki within the window (the LOG-A1 probe covers this end-to-end; this story is its per-front decomposition for failure attribution).

User stories

LOG-U1 — Find a known marker line [browser]

• Persona: dev1. Preconditions: a unique marker was pushed within the retention window (LOG-A1’s probe marker qualifies).
• Steps: in Grafana Explore against the pro Loki datasource, query for the marker string.
• Expected: the marker line returns with allowlisted labels; counts follow the book — distinct producer sequences are the truth, raw line counts are not (source: logging-pipeline.md Parts A–B).
• Assert: LogQL {log_type="application"} |= "<marker>" → >= 1 line whose stream labels ⊆ the LOG-A3 allowlist.

LOG-U2 — Trace lookup in Tempo [browser]

• Persona: dev1.
• Steps: open the Tempo datasource in Explore; run a search over the retention window.
• Expected: Tempo responds and search executes. Trace content is not asserted: the book records Tempo as “live, datasource wired, waiting for instrumented apps” (source: logging-pipeline.md).
• Assert: GET /api/search → 200 with a well-formed response (trace list may be empty). A content assertion is TBD until an app is instrumented.

LOG-U3 — End-to-end delivery, when CRC runs [config/API]

• Persona: Platform admin as the user’s proxy (the health suite runs this persona hourly).
• Preconditions: CRC running.
• Steps: emit exactly 10 marker events from the log-sim producer; poll local Loki and Grafana Cloud Loki for the distinct sequence numbers.
• Expected: all 10 distinct events in both backends. Proven hourly by grafana-cloud-log-delivery (source: health-suite README; logging-pipeline.md, “Prove delivery”). After the cutover proof at CRC’s next start, the pro-Loki leg from the checklist joins this story.
• Assert: distinct seq count == 10 in each backend within the poll window.

Gaps

• Doc drift: the logging tool page describes the pipeline as tooling-plane (mini) only; the pro stack, fronts, and dual-forward live in PRs #60/#61, issue #41, and the CRC-start checklist. The page needs its post-migration pass.
• TBD: no documented expectation covers a file-archive assertion on pro (the mini Logstash pattern mirrors a plain-text archive on a PVC; pro carried the pipeline config and allowlist, but the archive was not separately asserted).
• Manual-only: none.