RHELS LAB

See everything

Observability

Metrics, logs, and traces — the watchtower is lit. Prometheus and Grafana watch the workload cluster; Loki and Tempo hold logs and traces on the tooling plane.

Metrics · live

kube-prometheus-stack runs on the workload cluster. Service monitoring is one contract: expose /metrics, add a ServiceMonitor, get scraped. 

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app1001
  namespace: app1001-dev
spec:
  selector:
    matchLabels:
      app: app1001
  endpoints:
    - port: http
      path: /metrics
  • ▸ Prometheus (kube-prometheus-stack)
  • ▸ Grafana

Logs · live

Loki aggregates logs on the tooling plane, queryable from Grafana Explore with LogQL — no more per-pod kubectl logs spelunking. 

{namespace="app1001-dev"} |= "ERROR"

The Loki store and the Grafana datasource are live — and workload-cluster logs now ship into it: collectors feed a Vector aggregator that fans out to this Loki and to a hosted Loki in Grafana Cloud, with a scheduled job re-proving delivery every few hours.

Traces · live

Tempo stores distributed traces on the tooling plane, wired into Grafana as a datasource.

In Grafana Explore, pick the Tempo datasource and search by service name to follow a request across services — once the demo apps are instrumented, that is the next step.

Metrics tell you what happened, logs tell you why, traces tell you where. The lab now keeps all three.

Grafana · the single pane

Grafana runs on the workload cluster, provisioned by GitOps with three datasources: Prometheus (default, in-cluster), Loki and Tempo (tooling plane). Provisioned dashboards ship per app:

  • Demo Java App

    Request rate, error rate, latency p50/p95, JVM heap used.

  • Demo Python App

    Request rate, error rate, latency p50/p95.

  • Demo Redis App

    App request rate, Redis commands/s, latency p95, connected clients & memory.

Policy as a signal · Kyverno

Kyverno admission control runs on the workload cluster. The require-signed-images policy currently audits — every allow/deny decision lands in a PolicyReport you can read like any other cluster signal. Flipping it to Enforce is the last mile of rhels/team#107.

What's next

  • ▸ Wider log forwarding — the Vector pipeline is deployed for the trial namespace; extending collection cluster-wide is the next step.
  • ▸ Trace instrumentation — teach the demo apps to emit spans so Tempo has real traces to show.

Next chapter

5. Add an app

You now know the whole machine — workshop, factory floor, delivery line, watchtower. Next, the payoff: shipping your own app.