Skip to content

Commit 99420a5

Browse files
docs: add architecture diagram, demo quickstart, screenshot, and changelog
README gains a Mermaid diagram, a one-command demo section up top, a genuine screenshot of the Service Health dashboard rendered from the running demo, run/check/alerting docs, and a design-decisions section. CHANGELOG.md starts the release history at v0.1.0.
1 parent 8eeb059 commit 99420a5

3 files changed

Lines changed: 110 additions & 2 deletions

File tree

CHANGELOG.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Changelog
2+
3+
Notable changes to this stack. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
4+
versions follow [SemVer](https://semver.org/).
5+
6+
## [0.1.0] - 2026-07-03
7+
8+
First tagged release: the stack is runnable, demoable, and validated in CI.
9+
10+
### Added
11+
12+
- `just demo`: one-command demo — an auto-instrumented FastAPI service under
13+
constant load populates Grafana with correlated traces, metrics, and logs.
14+
- `just check`: validation gate (compose syntax, Prometheus config + alert
15+
rules, collector config, YAML, workflows, dashboard JSON), all in pinned
16+
containers. Run by GitHub Actions on every push/PR, plus a `just smoke`
17+
boot test.
18+
- Provisioned **Service Health (RED)** dashboard driven by Tempo span
19+
metrics — works for any service that sends traces.
20+
- Prometheus alert rules: target down, OTel export failures, high span
21+
error rate.
22+
- README: architecture diagram, demo quickstart, screenshot, design
23+
decisions.
24+
25+
### Changed
26+
27+
- `cloudflared` moved to an optional `compose.tunnel.yml` overlay and pinned
28+
(was `:latest`); the core stack no longer needs a tunnel token to run.
29+
30+
### Fixed
31+
32+
- OTLP metrics ingestion: Prometheus 3.x renamed the OTLP receiver flag,
33+
so the collector's metrics export 404ed and OTLP metrics were dropped.
34+
35+
## [0.0.1] - 2026-06-30
36+
37+
Initial stack: OTel Collector → Loki / Tempo / Prometheus → Grafana wired
38+
for cross-signal correlation, exposed via Cloudflare Tunnel, with RELab
39+
dashboards.

README.md

Lines changed: 71 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,55 @@ Central Grafana + Loki + Tempo + Prometheus + OTel Collector, wired together
44
with logs ↔ traces ↔ metrics correlation. Meant to run on one host and
55
receive telemetry from multiple projects over OTLP.
66

7+
```mermaid
8+
flowchart LR
9+
apps["Project apps<br/>(OTLP gRPC 4317 / HTTP 4318)"] --> otel["OTel Collector<br/>(ingestion gateway)"]
10+
otel -- logs --> loki[Loki]
11+
otel -- traces --> tempo[Tempo]
12+
otel -- metrics --> prom[Prometheus]
13+
tempo -- "span metrics (RED)" --> prom
14+
loki --> grafana[Grafana]
15+
tempo --> grafana
16+
prom --> grafana
17+
grafana --- cf["Cloudflare Tunnel<br/>(optional overlay)"]
18+
```
19+
20+
## Demo: see it work in one command
21+
22+
```sh
23+
cp .env.example .env
24+
just demo
25+
```
26+
27+
This starts the full stack plus a small auto-instrumented FastAPI service
28+
under constant load (`compose.demo.yml`). Give it a minute, then open
29+
Grafana at <http://localhost:3000> (admin / change-me):
30+
31+
- **Dashboards → Service Health (RED)** — request rate, error rate, and
32+
latency for the demo service. Latency dots are exemplars: click one to
33+
open that exact trace in Tempo.
34+
- **Explore → Loki**`{service_name="demo-api"}`; expand an error line
35+
and follow its `trace_id` to the trace.
36+
- **Alerting → Alert rules** — stack-health and error-rate rules,
37+
evaluated by Prometheus.
38+
39+
![Service Health (RED) dashboard](docs/img/service-health.png)
40+
41+
Tear it down with `just demo-down`.
42+
743
## Layout
844

945
```
10-
compose.yml # all services
46+
compose.yml # core services
47+
compose.tunnel.yml # production overlay: Cloudflare Tunnel
48+
compose.demo.yml # demo overlay: sample telemetry source
49+
demo/ # the demo FastAPI service
1150
config/
1251
otel-collector.yaml # ingestion gateway (OTLP in → Loki/Tempo/Prometheus out)
1352
loki.yaml # logs
1453
tempo.yaml # traces
1554
prometheus.yaml # metrics
55+
alerts/ # Prometheus alert rules
1656
grafana/ # provisioned datasources + dashboard loader
1757
dashboards/ # drop JSON dashboards here; Grafana auto-loads them
1858
```
@@ -21,11 +61,18 @@ dashboards/ # drop JSON dashboards here; Grafana auto-loads them
2161

2262
```sh
2363
cp .env.example .env # set GRAFANA_ADMIN_PASSWORD
24-
just up
64+
just up # core stack, local only
65+
just up-tunnel # core stack + Cloudflare Tunnel (needs CLOUDFLARE_TUNNEL_TOKEN)
2566
```
2667

2768
Grafana: <http://localhost:3000> (admin / whatever you set).
2869

70+
`just check` validates everything (compose files, Prometheus config and
71+
alert rules, collector config, YAML, workflows, dashboard JSON) in pinned
72+
containers — no host installs. CI runs the same command on every push and
73+
PR, plus a smoke test that boots the stack and waits for Grafana to report
74+
healthy.
75+
2976
## Sending telemetry from a project
3077

3178
### Traces + metrics (OTLP)
@@ -64,8 +111,30 @@ Logs carry labels for `service`, `env`, and `host` so you can filter in
64111
Grafana. Keep label cardinality low — don't add `user_id`, `request_id`,
65112
etc. as labels; use LogQL filters for those.
66113

114+
## Alerting
115+
116+
Prometheus evaluates the rules in `config/alerts/` (target down, OTel
117+
export failures, >5% span error rate) and Grafana surfaces them under
118+
Alerting → Alert rules. There is deliberately no Alertmanager: on a
119+
single-host stack, another service buys routing complexity before anyone
120+
needs it. Add one (or a Grafana contact point) when alerts must page
121+
someone.
122+
67123
## Storage
68124

69125
Everything persists to local Docker volumes (`loki_data`, `tempo_data`,
70126
`prometheus_data`, `grafana_data`). Swap Loki / Tempo storage to S3-compatible
71127
(Backblaze B2, Cloudflare R2, Hetzner, MinIO) when you outgrow local disk.
128+
129+
## Design decisions
130+
131+
Everything enters through **one OTLP gateway** (the collector), so projects
132+
configure a single endpoint and the backends can be swapped without touching
133+
any app. The stack is **single-host by design** — CML's telemetry volume
134+
doesn't justify distributed ingest, and one compose file is auditable — with
135+
a documented scale-out path (S3-backed Loki/Tempo above) when it stops
136+
fitting. Logs keep **label cardinality low** (`service`, `env`, `host` only;
137+
everything else is query-time filtering) because high-cardinality labels are
138+
how Loki installs fall over. RED metrics are **derived from traces** by
139+
Tempo's metrics generator, so any service that sends traces gets the
140+
Service Health dashboard for free, instrumented or not.

docs/img/service-health.png

153 KB
Loading

0 commit comments

Comments
 (0)