RHELS LAB
Contents · the agent book

Start here

  1. RHELS Lab — Agent Docs
  2. Service Directory
  3. Lab Overview

Planes

  1. Tooling Plane — the Mac mini cluster
  2. Workload Cluster — CRC / OpenShift
  3. Networking

Tool pages

  1. OpenLDAP — Identity & Access
  2. Vault & Secrets Delivery
  3. Certificates — cert-manager & Let's Encrypt
  4. Tailscale & Cloudflare — the name layer
  5. Agent onboarding — how an agent joins the team
  6. Argo CD — GitOps on both planes
  7. Harbor — container registry
  8. Grafana — dashboards & Explore
  9. Logging pipeline — Vector, Loki, Logstash
  10. Prometheus & Alertmanager — metrics and alerts
  11. GitHub Actions — self-hosted runners
  12. ServiceNow PDI & the MCP servers
  13. StackRox — security platform on pro
  14. This documentation system
  15. Namespace as a Service — the MCP front door
  16. Vault PKI

Operations

  1. CI/CD
  2. Observability — deep dive
  3. Disaster Recovery & Rebuild
  4. SRE: IaC, DR, Upgrades, SLO/SLA
  5. Runbooks
  6. The Agent-Run Lab — A Working Showcase

QA Use Cases

  1. QA Use Cases — how this section works
  2. QA — Vault
  3. QA — Argo CD
  4. QA — Identity (OpenLDAP + Dex)
  5. QA — Grafana, Prometheus, Alertmanager
  6. QA — Logging (Loki, Tempo, Logstash)
  7. QA — ACS (StackRox)
  8. QA — Kyverno
  9. QA — MCP server (Namespace as a Service)
  10. QA — CI runners (ARC)
  11. QA — Front door (ingress, cert-manager, Tailscale operator)
  12. QA — Alert bridge
  13. QA — ServiceNow PDI, sims, group-sync
  14. QA — Lab docs
  15. QA — CRC / OpenShift

Reference

  1. Glossary
  2. Agent Protocol

Agent book · Tool pages

Logging pipeline — Vector, Loki, Logstash

The pipeline moves CRC application logs to two Loki backends with a standing, self-checking proof that they arrived. This page is the setup + usage reference; the measured results and hard-won lessons (5k lines/s ceiling, label discipline, the kubelet rotation trap) live in observability.md and are not repeated here.

Part A — How it is set up

  1. Collection — OpenShift Cluster Logging (Vector collectors) with a ClusterLogForwarder whose application input covers the log-sim namespace only. That scope is deliberate: cluster-wide forwarding is a later step, not a gap in disguise.
  2. Two paths from the forwarder:
    • Logstash (tooling plane, observability namespace) → the local Loki, plus a plain-text file archive (cluster/namespace/pod/container/log.log on a PVC) mirroring the office pattern.
    • The Vector aggregator — a StatefulSet in CRC’s log-agg namespace. Its config is one file per concern and one file per sink, so a sink is added or removed as exactly one file in git. End-to-end acknowledgements are on; every sink has a disk buffer (256 MiB) that blocks rather than drops when full.
  3. The aggregator’s sinks (files in clusters/crc/logging/vector-aggregator/config/sinks/):
    • 00-metrics — its own Prometheus exporter;
    • 10-loki — the local Loki on the tooling plane;
    • 15-logstash-fallback — Logstash over a dedicated bridge, so the archive path survives even when reached indirectly;
    • 20-grafana-cloud-loki — Grafana Cloud hosted Loki, authenticated with a write-only token delivered from Vault by a Secrets Operator tenant (secret/lab/grafana-cloud/loki-token).
  4. Labels are exactly log_type, log_source, hostname, level — the allowlist is load-bearing (an unfiltered field-to-label mapping once created ~10.7k junk streams and silent drops).
  5. Local Loki (tooling plane): upstream chart, SingleBinary, filesystem storage on a 10 GiB PVC, 72 h retention (compactor), max_global_streams_per_user 50000. Tempo sits next to it as the tracing backend — live, datasource wired, waiting for instrumented apps.
  6. Double storage is by design: aggregator-path events land in the local Loki twice (sink 10 direct, sink 15 via Logstash). Raw line counts lie; distinct producer-sequence counts are the truth, and every verifier below counts that way.

Part B — How to use it

  • Read logs: Grafana Explore against the Loki datasource (tools/grafana.md) with LogQL — start from {log_type="application"} and narrow by hostname / level. The same streams exist in Grafana Cloud’s hosted Loki (sink 20) if the tooling plane is the thing that is down.
  • Prove delivery (do this instead of eyeballing): the log-delivery-test CronJob in log-sim re-proves both backends every 3 hours; runbook 12 is the manual version (emit a marker, count distinct sequences in both Lokis). The health suite’s grafana-cloud-log-delivery check runs the same proof hourly from the agent side.
  • Trigger the CronJob now: kubectl create job --from=cronjob/log-delivery-test <name> -n log-sim on CRC, then read the job’s logs — it prints both counts and exits non-zero on a miss. A failed CronJob twice in a row raises the LogDeliveryTestConsecutiveFailures alert (see tools/prometheus-alertmanager.md).
  • Add a sink: one new file in the sinks directory following the existing files’ shape (buffer block included), plus one line in the aggregator’s generator list — nothing else. If the sink needs a credential, it arrives as a Vault Secrets Operator tenant (vault.md), never as a literal in the config.
  • Check pipeline health: the aggregator’s own metrics (sink 00) expose per-sink sent/discarded counters; the hop where the count shrinks while its discarded counter climbs is the losing hop.
  • The file archive: Logstash’s PVC holds the plain-text mirror under cluster/namespace/pod/container/log.log paths — the fallback of last resort when both Lokis are suspect.