Skip to content

Commit 271db37

Browse files
simonvanlierdesimonvanlierde
committed
docs: update README
1 parent da9767b commit 271db37

1 file changed

Lines changed: 107 additions & 95 deletions

File tree

‎README.md‎

Lines changed: 107 additions & 95 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,50 @@
11
# Observability stack
22

3-
Central monitoring for CML's research software. One host runs Grafana,
4-
Loki, Tempo, Prometheus, and an OpenTelemetry Collector, wired together so
5-
logs, traces, and metrics cross-reference each other. Projects send their
6-
telemetry here over OTLP (the OpenTelemetry protocol) and it lands in one
7-
place, queryable side by side.
3+
[![CI](https://github.com/CMLPlatform/monitoring/actions/workflows/ci.yml/badge.svg)](https://github.com/CMLPlatform/monitoring/actions/workflows/ci.yml)
4+
[![OpenTelemetry](https://img.shields.io/badge/OpenTelemetry-OTLP-425cc7?logo=opentelemetry&logoColor=white)](config/otel-collector.yaml)
5+
[![Grafana LGTM](https://img.shields.io/badge/Grafana-Loki%20%7C%20Tempo%20%7C%20Prometheus-f46800?logo=grafana&logoColor=white)](compose.yml)
6+
[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
87

9-
## Architecture
8+
Central monitoring for CML's research software. One host runs Grafana, Loki,
9+
Tempo, Prometheus, and an OpenTelemetry Collector, wired together so logs,
10+
traces, and metrics cross-reference each other. Projects send their telemetry
11+
here over OTLP (the OpenTelemetry protocol) and it lands in one place,
12+
queryable side by side.
1013

11-
Everything enters through a single gateway, the OpenTelemetry Collector. A
12-
project only ever configures one endpoint, and we can swap a storage
13-
backend later without touching any application. Tempo also derives
14-
request-rate, error-rate, and duration ("RED") metrics from the traces it
15-
receives, so a service that sends nothing but traces still gets a working
16-
dashboard and error alerting.
14+
## Try it in one command
1715

18-
The stack deliberately runs on a single host. At CML's telemetry volume,
19-
distributed ingestion would add operational weight for no gain. The
20-
reasoning, and the alternatives we considered, are recorded in
21-
[ADR 0001](docs/adr/0001-observability-stack.md).
16+
```sh
17+
cp .env.example .env
18+
just demo
19+
```
20+
21+
This starts the full stack plus a small FastAPI service under constant
22+
artificial load (`compose.demo.yml`). The service is instrumented with
23+
OpenTelemetry auto-instrumentation and fails about one request in ten, on
24+
purpose. Give it a minute, then open Grafana at <http://localhost:3000>
25+
(admin / change-me) and look at:
26+
27+
- **Dashboards → Service Health (RED)** — request rate, error rate, and
28+
latency. The dots on the latency panel are exemplars: click one and Grafana
29+
opens the exact trace behind that measurement.
30+
- **Dashboards → Logs Overview** — log volume by service and level, an error
31+
feed, and a live tail of everything arriving over OTLP.
32+
- **Alerting → Alert rules** — the stack-health and error-rate rules Prometheus
33+
is evaluating.
34+
35+
![Service Health (RED) dashboard](docs/img/service-health.png)
36+
37+
`just demo-down` removes the demo services again; the rest of the stack keeps
38+
running.
39+
40+
## How it works
41+
42+
Everything enters through a single gateway, the OpenTelemetry Collector. A
43+
project only ever configures one endpoint, and we can swap a storage backend
44+
later without touching any application. Tempo also derives request-rate,
45+
error-rate, and duration ("RED") metrics from the traces it receives, so a
46+
service that sends nothing but traces still gets a working dashboard and error
47+
alerting.
2248

2349
```mermaid
2450
flowchart LR
@@ -37,36 +63,71 @@ flowchart LR
3763
end
3864
```
3965

40-
Solid arrows show telemetry being written; dotted arrows show Grafana
41-
reading at query time. Locally (`just up` or `just demo`) there is no
42-
tunnel involved: everything talks over the compose network and Grafana is
43-
at `localhost:3000`.
66+
Solid arrows show telemetry being written; dotted arrows show Grafana reading
67+
at query time. Locally (`just up` or `just demo`) there is no tunnel involved:
68+
everything talks over the compose network and Grafana is at `localhost:3000`.
4469

45-
## Demo: see it work in one command
70+
The stack deliberately runs on a single host. At CML's telemetry volume,
71+
distributed ingestion would add operational weight for no gain. The reasoning,
72+
and the alternatives we considered, are recorded in
73+
[ADR 0001](docs/adr/0001-observability-stack.md).
74+
75+
## Run it for real
4676

4777
```sh
48-
cp .env.example .env
49-
just demo
78+
cp .env.example .env # set GRAFANA_ADMIN_PASSWORD
79+
just up # core stack, local only
80+
just up-tunnel # production: core stack + Cloudflare Tunnel
81+
just check # validate every config in the repo
5082
```
5183

52-
This starts the full stack plus a small FastAPI service under constant
53-
artificial load (`compose.demo.yml`). The service is instrumented with
54-
OpenTelemetry auto-instrumentation and fails about one request in ten, on
55-
purpose. Give it a minute, then open Grafana at <http://localhost:3000>
56-
(admin / change-me) and look at:
84+
Grafana: <http://localhost:3000> (admin / whatever you set).
5785

58-
- **Dashboards → Service Health (RED)** — request rate, error rate, and
59-
latency. The dots on the latency panel are exemplars: click one and
60-
Grafana opens the exact trace behind that measurement.
61-
- **Dashboards → Logs Overview** — log volume by service and level, an
62-
error feed, and a live tail of everything arriving over OTLP.
63-
- **Alerting → Alert rules** — the stack-health and error-rate rules
64-
Prometheus is evaluating.
86+
In production the stack sits behind a Cloudflare Tunnel, and that edge is code
87+
too. The tunnel, its hostnames, DNS, and the Cloudflare Access rule that puts
88+
an email one-time-PIN in front of Grafana all live in `infra/` as a small
89+
OpenTofu configuration. Applying it produces the `CLOUDFLARE_TUNNEL_TOKEN` that
90+
`just up-tunnel` needs; bootstrap steps are at the top of
91+
[infra/main.tf](infra/main.tf).
6592

66-
![Service Health (RED) dashboard](docs/img/service-health.png)
93+
`just check` validates compose files, Prometheus config and alert rules, the
94+
collector config, YAML, workflows, and dashboard JSON. Every validator runs in
95+
a pinned container, so nothing needs to be installed on the host. CI runs the
96+
same command on every push and pull request, plus a smoke test that boots the
97+
stack and waits for Grafana to come up healthy.
98+
99+
## Sending telemetry from a project
100+
101+
You need three things: the OTLP endpoint, the bearer token (`OTLP_AUTH_TOKEN`),
102+
and a few naming conventions. Copy-paste templates for each route — zero-code
103+
Python/FastAPI, plain OTLP environment variables, the Loki Docker driver, and
104+
Grafana Alloy for log files — are in
105+
**[docs/ONBOARDING.md](docs/ONBOARDING.md)**.
106+
107+
> [!WARNING]
108+
> Never publish ports 4317/4318 to the internet. The compose file binds them to
109+
> `127.0.0.1`; the tunnel is the way in.
110+
111+
## Alerting
112+
113+
Prometheus evaluates the rules in `config/alerts/`: scrape target down, OTel
114+
export failures, error rate above 5%, disk above 80%. Alertmanager delivers
115+
them to whatever webhook you set in `ALERT_WEBHOOK_URL` (ntfy, Slack, and so
116+
on).
67117

68-
`just demo-down` removes the demo services again; the rest of the stack
69-
keeps running.
118+
One rule, `Watchdog`, fires permanently by design and posts to `HEARTBEAT_URL`
119+
every five minutes. Point that at a dead man's switch such as healthchecks.io —
120+
a service that alerts when the pings *stop* — and you will also hear about the
121+
one failure the host cannot report itself: its own death. Both variables are
122+
optional; leave them unset and alerts are simply visible in Grafana.
123+
124+
## Storage
125+
126+
Everything persists to local Docker volumes (`loki_data`, `tempo_data`,
127+
`prometheus_data`, `grafana_data`). When local disk stops fitting, Loki and
128+
Tempo can move to any S3-compatible object store (Backblaze B2, Cloudflare R2,
129+
Hetzner, MinIO); `compose.storage-s3.yml` documents the concrete shape of that
130+
change.
70131

71132
## Layout
72133

@@ -88,62 +149,13 @@ docs/ # runbook, onboarding templates, ADRs, screenshots
88149
infra/ # OpenTofu: Cloudflare tunnel, ingress routes, DNS
89150
```
90151

91-
## Run
92-
93-
```sh
94-
cp .env.example .env # set GRAFANA_ADMIN_PASSWORD
95-
just up # core stack, local only
96-
just up-tunnel # production: core stack + Cloudflare Tunnel
97-
```
98-
99-
Grafana: <http://localhost:3000> (admin / whatever you set).
100-
101-
In production the stack sits behind a Cloudflare Tunnel, and that edge is
102-
code too. The tunnel, its hostnames, DNS, and the Cloudflare Access rule
103-
that puts an email one-time-PIN in front of Grafana all live in `infra/`
104-
as a small OpenTofu configuration. Applying it produces the
105-
`CLOUDFLARE_TUNNEL_TOKEN` that `just up-tunnel` needs; bootstrap steps are
106-
at the top of [infra/main.tf](infra/main.tf).
107-
108-
`just check` validates the whole repository: compose files, Prometheus
109-
config and alert rules, collector config, YAML, workflows, and dashboard
110-
JSON. Every validator runs in a pinned container, so nothing needs to be
111-
installed on the host. CI runs the same command on every push and pull
112-
request, plus a smoke test that boots the stack and waits for Grafana to
113-
come up healthy.
114-
115-
## Sending telemetry from a project
116-
117-
You need three things: the OTLP endpoint, the bearer token
118-
(`OTLP_AUTH_TOKEN`), and a few naming conventions. Copy-paste templates
119-
for each route — zero-code Python/FastAPI, plain OTLP environment
120-
variables, the Loki Docker driver, and Grafana Alloy for log files — are
121-
in **[docs/ONBOARDING.md](docs/ONBOARDING.md)**.
152+
## Documentation
122153

123-
Never publish ports 4317/4318 to the internet. The compose file binds them
124-
to `127.0.0.1`; the tunnel is the way in.
154+
| Document | What it covers |
155+
| --- | --- |
156+
| [docs/ONBOARDING.md](docs/ONBOARDING.md) | Connecting a project: endpoint, token, copy-paste templates |
157+
| [docs/RUNBOOK.md](docs/RUNBOOK.md) | Day-to-day ops: rotating tokens, disk pressure, backup and restore |
158+
| [docs/adr/](docs/adr/) | Why the stack looks like this |
159+
| [CHANGELOG.md](CHANGELOG.md) | Release history |
125160

126-
## Alerting
127-
128-
Prometheus evaluates the rules in `config/alerts/`: scrape target down,
129-
OTel export failures, error rate above 5%, disk above 80%. Alertmanager
130-
delivers them to whatever webhook you set in `ALERT_WEBHOOK_URL` (ntfy,
131-
Slack, and so on).
132-
133-
One rule, `Watchdog`, fires permanently by design and posts to
134-
`HEARTBEAT_URL` every five minutes. Point that at a dead man's switch such
135-
as healthchecks.io — a service that alerts when the pings *stop* — and you
136-
will also hear about the one failure the host cannot report itself: its own
137-
death. Both variables are optional; leave them unset and alerts are simply
138-
visible in Grafana.
139-
140-
Day-to-day operations (rotating tokens, disk pressure, backup and restore)
141-
are covered in **[docs/RUNBOOK.md](docs/RUNBOOK.md)**.
142-
143-
## Storage
144-
145-
Everything persists to local Docker volumes (`loki_data`, `tempo_data`,
146-
`prometheus_data`, `grafana_data`). When local disk stops fitting, Loki
147-
and Tempo can move to any S3-compatible object store (Backblaze B2,
148-
Cloudflare R2, Hetzner, MinIO); `compose.storage-s3.yml` documents the
149-
concrete shape of that change.
161+
Licensed under the [MIT License](LICENSE).

0 commit comments

Comments
 (0)