Skip to content

Commit d078a75

Browse files
docs: Improve wording
1 parent be6d97a commit d078a75

2 files changed

Lines changed: 42 additions & 41 deletions

File tree

README.md

Lines changed: 17 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
11
# Observability stack
22

3-
Central Grafana + Loki + Tempo + Prometheus + OTel Collector, wired together
4-
with logs ↔ traces metrics correlation. Meant to run on one host and
5-
receive telemetry from multiple projects over OTLP.
3+
One host running Grafana, Loki, Tempo, Prometheus, and an OpenTelemetry
4+
Collector, wired so logs, traces, and metrics cross-link. Point any number of
5+
projects at it over OTLP and their telemetry lands in one place.
66

77
```mermaid
88
flowchart LR
@@ -42,7 +42,7 @@ Tear it down with `just demo-down`.
4242

4343
## Layout
4444

45-
```
45+
```sh
4646
compose.yml # core services
4747
compose.tunnel.yml # production overlay: Cloudflare Tunnel
4848
compose.demo.yml # demo overlay: sample telemetry source
@@ -130,13 +130,16 @@ Everything persists to local Docker volumes (`loki_data`, `tempo_data`,
130130

131131
## Design decisions
132132

133-
Everything enters through **one OTLP gateway** (the collector), so projects
134-
configure a single endpoint and the backends can be swapped without touching
135-
any app. The stack is **single-host by design** — CML's telemetry volume
136-
doesn't justify distributed ingest, and one compose file is auditable — with
137-
a documented scale-out path (S3-backed Loki/Tempo above) when it stops
138-
fitting. Logs keep **label cardinality low** (`service`, `env`, `host` only;
139-
everything else is query-time filtering) because high-cardinality labels are
140-
how Loki installs fall over. RED metrics are **derived from traces** by
141-
Tempo's metrics generator, so any service that sends traces gets the
142-
Service Health dashboard for free, instrumented or not.
133+
Everything enters through one gateway — the collector — so a project
134+
configures a single OTLP endpoint, and a backend can be swapped without
135+
touching any app. It runs on one host on purpose: CML's telemetry is a handful
136+
of services at single-digit requests per second, so distributed ingest would
137+
add operational weight for no gain, while one compose file stays auditable by a
138+
single person. When local disk runs short, S3-backed Loki and Tempo (above)
139+
are the documented way out.
140+
141+
Logs carry only low-cardinality labels — `service`, `env`, `host` — and
142+
everything else is a query-time filter; high-cardinality labels are the usual
143+
way a Loki install falls over. And because Tempo's metrics generator derives
144+
RED metrics from spans, any service that sends traces gets the Service Health
145+
dashboard whether or not it emits metrics of its own.

docs/adr/0001-observability-stack.md

Lines changed: 25 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -4,11 +4,10 @@ Date: 2026-07-03. Status: accepted (records a decision already in production).
44

55
## Context
66

7-
CML runs several research platforms (RELab and others) that need logs,
8-
traces, and metrics in one place. Telemetry volume is modest — a handful of
9-
services, single-digit requests per second — but the projects are
10-
long-lived, maintained by a very small team, and must stay cheap and
11-
auditable.
7+
CML runs several long-lived research platforms (RELab and others) that need
8+
their logs, traces, and metrics in one place. The volume is modest — a handful
9+
of services at single-digit requests per second — but one small team maintains
10+
all of it, so whatever we run must stay cheap and auditable.
1211

1312
## Decision
1413

@@ -17,34 +16,33 @@ the only ingestion endpoint, fanning out to Loki (logs), Tempo (traces), and
1716
Prometheus (metrics), with Grafana on top. Expose Grafana and the OTLP
1817
endpoints via Cloudflare Tunnel; bind everything else to `127.0.0.1`.
1918

20-
- **OTLP-native:** projects configure one endpoint and one protocol.
21-
Backends can be swapped (or moved to hosted equivalents) without touching
22-
any application.
23-
- **Single-host:** one compose file is something one person can fully audit
24-
and rebuild. Distributed ingest (Kubernetes, Mimir, multi-tenant Loki)
25-
buys nothing at this volume and costs ongoing operational attention.
19+
- **OTLP-native:** one endpoint, one protocol. Swap a backend or move it to a
20+
hosted equivalent without touching any application.
21+
- **Single-host:** one compose file one person can audit and rebuild.
22+
Distributed ingest (Kubernetes, Mimir, multi-tenant Loki) would add operational
23+
overhead for no benefit at this volume.
2624
- **Traces as the primary signal:** Tempo's metrics generator derives RED
27-
metrics and a service graph from spans, so a service that only sends
28-
traces still gets dashboards and error-rate alerting.
25+
metrics and a service graph from spans, so a service that only sends traces
26+
still gets dashboards and error-rate alerting.
2927

3028
## Alternatives considered
3129

32-
- **Grafana Cloud / hosted SaaS:** less to operate, but recurring cost,
33-
data residency questions for research data, and less useful as
34-
institutional infrastructure knowledge.
35-
- **Per-project stacks:** no shared endpoint to maintain, but duplicates
36-
storage and Grafana per project and makes cross-project correlation
37-
impossible.
38-
- **ELK / OpenSearch:** heavier to run, log-centric; weaker native OTLP and
39-
trace correlation story than the Grafana stack.
30+
- **Grafana Cloud / hosted SaaS:** less to operate, but a recurring bill,
31+
data-residency questions for research data, and little institutional
32+
infrastructure knowledge to show for it.
33+
- **Per-project stacks:** no shared endpoint to maintain, but each project
34+
duplicates its own storage and Grafana, and cross-project correlation
35+
becomes impossible.
36+
- **ELK / OpenSearch:** heavier to run and log-centric, with a weaker native
37+
OTLP and trace-correlation story than the Grafana stack.
4038

4139
## Consequences
4240

43-
- The host is a single point of failure; acceptable because the monitored
44-
platforms degrade gracefully without telemetry (OTLP export is
41+
- The host is a single point of failureacceptable, because the monitored
42+
platforms degrade gracefully when telemetry stops (OTLP export is
4543
fire-and-forget) and the stack rebuilds from this repo in minutes.
46-
- Local disk bounds retention (30d logs/metrics, 7d traces). The documented
47-
escape hatch is S3-compatible storage for Loki/Tempo — see
48-
`compose.storage.s3.yml` — before any move to distributed ingest.
49-
- Everything is pinned and validated by `just check` in CI, so the stack
44+
- Local disk bounds retention (30d logs/metrics, 7d traces). The escape hatch,
45+
reached before any move to distributed ingest, is S3-compatible storage for
46+
Loki and Tempo — see `compose.storage.s3.yml`.
47+
- Every image is pinned and validated by `just check` in CI, so the stack
5048
stays reproducible.

0 commit comments

Comments
 (0)