@@ -4,11 +4,10 @@ Date: 2026-07-03. Status: accepted (records a decision already in production).
44
55## Context
66
7- CML runs several research platforms (RELab and others) that need logs,
8- traces, and metrics in one place. Telemetry volume is modest — a handful of
9- services, single-digit requests per second — but the projects are
10- long-lived, maintained by a very small team, and must stay cheap and
11- auditable.
7+ CML runs several long-lived research platforms (RELab and others) that need
8+ their logs, traces, and metrics in one place. The volume is modest — a handful
9+ of services at single-digit requests per second — but one small team maintains
10+ all of it, so whatever we run must stay cheap and auditable.
1211
1312## Decision
1413
@@ -17,34 +16,33 @@ the only ingestion endpoint, fanning out to Loki (logs), Tempo (traces), and
1716Prometheus (metrics), with Grafana on top. Expose Grafana and the OTLP
1817endpoints via Cloudflare Tunnel; bind everything else to ` 127.0.0.1 ` .
1918
20- - ** OTLP-native:** projects configure one endpoint and one protocol.
21- Backends can be swapped (or moved to hosted equivalents) without touching
22- any application.
23- - ** Single-host:** one compose file is something one person can fully audit
24- and rebuild. Distributed ingest (Kubernetes, Mimir, multi-tenant Loki)
25- buys nothing at this volume and costs ongoing operational attention.
19+ - ** OTLP-native:** one endpoint, one protocol. Swap a backend or move it to a
20+ hosted equivalent without touching any application.
21+ - ** Single-host:** one compose file one person can audit and rebuild.
22+ Distributed ingest (Kubernetes, Mimir, multi-tenant Loki) would add operational
23+ overhead for no benefit at this volume.
2624- ** Traces as the primary signal:** Tempo's metrics generator derives RED
27- metrics and a service graph from spans, so a service that only sends
28- traces still gets dashboards and error-rate alerting.
25+ metrics and a service graph from spans, so a service that only sends traces
26+ still gets dashboards and error-rate alerting.
2927
3028## Alternatives considered
3129
32- - ** Grafana Cloud / hosted SaaS:** less to operate, but recurring cost ,
33- data residency questions for research data, and less useful as
34- institutional infrastructure knowledge.
35- - ** Per-project stacks:** no shared endpoint to maintain, but duplicates
36- storage and Grafana per project and makes cross-project correlation
37- impossible.
38- - ** ELK / OpenSearch:** heavier to run, log-centric; weaker native OTLP and
39- trace correlation story than the Grafana stack.
30+ - ** Grafana Cloud / hosted SaaS:** less to operate, but a recurring bill ,
31+ data- residency questions for research data, and little institutional
32+ infrastructure knowledge to show for it .
33+ - ** Per-project stacks:** no shared endpoint to maintain, but each project
34+ duplicates its own storage and Grafana, and cross-project correlation
35+ becomes impossible.
36+ - ** ELK / OpenSearch:** heavier to run and log-centric, with a weaker native
37+ OTLP and trace- correlation story than the Grafana stack.
4038
4139## Consequences
4240
43- - The host is a single point of failure; acceptable because the monitored
44- platforms degrade gracefully without telemetry (OTLP export is
41+ - The host is a single point of failure — acceptable, because the monitored
42+ platforms degrade gracefully when telemetry stops (OTLP export is
4543 fire-and-forget) and the stack rebuilds from this repo in minutes.
46- - Local disk bounds retention (30d logs/metrics, 7d traces). The documented
47- escape hatch is S3-compatible storage for Loki/Tempo — see
48- ` compose.storage.s3.yml ` — before any move to distributed ingest .
49- - Everything is pinned and validated by ` just check ` in CI, so the stack
44+ - Local disk bounds retention (30d logs/metrics, 7d traces). The escape hatch,
45+ reached before any move to distributed ingest, is S3-compatible storage for
46+ Loki and Tempo — see ` compose.storage.s3.yml ` .
47+ - Every image is pinned and validated by ` just check ` in CI, so the stack
5048 stays reproducible.
0 commit comments