feat: reliability hardening (auth, self-monitoring, alert delivery, backups)

.env.example

@@ -6,6 +6,17 @@ GRAFANA_ADMIN_PASSWORD=change-me
 #   GRAFANA_ROOT_URL=https://grafana.example.com
 GRAFANA_ROOT_URL=http://localhost:3000
 
+# Bearer token every telemetry sender must present (Authorization: Bearer <token>).
+# The default only suits local use — generate a real one for production, e.g.: openssl rand -hex 32
+OTLP_AUTH_TOKEN=local-dev-token
+
+# Where Alertmanager delivers alert notifications (any webhook: ntfy, Slack, …).
+# Leave empty to run without delivery; failures are logged and harmless.
+ALERT_WEBHOOK_URL=
+# Dead man's switch ping target (e.g. https://hc-ping.com/<uuid>). The Watchdog
+# alert posts here every 5m; alert externally when pings stop.
+HEARTBEAT_URL=
+
 # Only needed for `just up-tunnel` (production exposure via Cloudflare Tunnel).
 # From: Cloudflare Zero Trust → Networks → Tunnels → <your tunnel> → Configure → token
 CLOUDFLARE_TUNNEL_TOKEN=

.github/dependabot.yml

@@ -8,12 +8,22 @@ updates:
       monitoring-stack-updates:
         patterns:
           - "*"
+    ignore:
+      # Tempo 3.x is a re-architecture requiring a config migration; upgrade
+      # deliberately, not via a routine bump. See comment in compose.yml.
+      - dependency-name: "grafana/tempo"
+        update-types: ["version-update:semver-major"]
 
   - package-ecosystem: "docker"
     directory: "/demo"
     schedule:
       interval: "monthly"
 
+  - package-ecosystem: "pip"
+    directory: "/demo"
+    schedule:
+      interval: "monthly"
+
   - package-ecosystem: "github-actions"
     directory: "/"
     schedule:

.github/workflows/ci.yml

@@ -7,6 +7,9 @@ on:
     branches: [main]
   pull_request:
 
+permissions:
+  contents: read
+
 jobs:
   check:
     runs-on: ubuntu-latest

.gitignore

@@ -1,3 +1,7 @@
 .env
 *.local.yml
 *.local.yaml
+backups/
+infra/.terraform/
+infra/terraform.tfstate*
+infra/*.tfvars

CHANGELOG.md

@@ -3,6 +3,38 @@
 Notable changes to this stack. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
 versions follow [SemVer](https://semver.org/).
 
+## [Unreleased]
+
+Reliability hardening: the stack becomes operable, not just runnable.
+
+### Added
+
+- Bearer-token auth on OTLP ingestion (`OTLP_AUTH_TOKEN`); senders use
+  `OTEL_EXPORTER_OTLP_HEADERS`.
+- Host self-monitoring: node-exporter, `HostDiskSpaceLow` alert (the
+  storage backstop — Loki/Tempo have no total-size cap), **Stack Health**
+  dashboard.
+- Alert delivery: Alertmanager routes to `ALERT_WEBHOOK_URL`; an
+  always-firing `Watchdog` heartbeats to `HEARTBEAT_URL` (dead man's
+  switch).
+- `just backup` / `just restore` — crash-consistent volume snapshots,
+  exercised end-to-end.
+- `docs/RUNBOOK.md` and `docs/ONBOARDING.md` with verified telemetry
+  templates (zero-code Python, plain OTLP, Loki Docker driver, Grafana
+  Alloy).
+- `infra/`: OpenTofu for the Cloudflare tunnel, ingress routes, and DNS.
+
+### Changed
+
+- Prometheus storage now bounded by size as well as time
+  (`--storage.tsdb.retention.size=15GB`).
+
+### Fixed
+
+- Tempo held at 2.x: a routine bump to 3.0 crash-looped on the 2.x config
+  (3.0 is a re-architecture). Dependabot now skips Tempo majors so the
+  migration happens deliberately.
+
 ## [0.1.0] - 2026-07-03
 
 First tagged release: the stack is runnable, demoable, and validated in CI.

README.md

@@ -4,19 +4,37 @@ One host running Grafana, Loki, Tempo, Prometheus, and an OpenTelemetry
 Collector, wired so logs, traces, and metrics cross-link. Point any number of
 projects at it over OTLP and their telemetry lands in one place.
 
+## Architecture
+
+Everything enters through one gateway — the OTel Collector — so a project
+configures a single OTLP endpoint and a backend can be swapped without
+touching any app. Tempo derives RED metrics from spans, so a service that
+only sends traces still gets dashboards and error alerting. It all runs on
+one host on purpose: at CML's telemetry volume, distributed ingest would add
+operational weight for no gain. Rationale and alternatives:
+[ADR 0001](docs/adr/0001-observability-stack.md).
+
 ```mermaid
 flowchart LR
-    apps["Project apps<br/>(OTLP gRPC 4317 / HTTP 4318)"] --> otel["OTel Collector<br/>(ingestion gateway)"]
-    otel -- logs --> loki[Loki]
-    otel -- traces --> tempo[Tempo]
-    otel -- metrics --> prom[Prometheus]
-    tempo -- "span metrics (RED)" --> prom
-    loki --> grafana[Grafana]
-    tempo --> grafana
-    prom --> grafana
-    grafana --- cf["Cloudflare Tunnel<br/>(optional overlay)"]
+    apps["Project apps"] -->|"OTLP over HTTPS<br/>(gRPC/HTTP direct on private nets)"| cf["Cloudflare Tunnel<br/>(production, optional)"]
+    user["Browser"] -->|HTTPS| cf
+    cf --> otel
+    cf --> grafana
+
+    subgraph host["Monitoring host — Docker Compose, ports bound to 127.0.0.1"]
+        otel["OTel Collector<br/>(ingestion gateway)"]
+        otel -->|logs| loki["Loki"]
+        otel -->|traces| tempo["Tempo"]
+        otel -->|metrics| prom["Prometheus"]
+        tempo -->|"span metrics (RED)"| prom
+        grafana["Grafana"] -. queries .-> loki & tempo & prom
+    end
 ```
 
+Solid arrows are the write path; dotted arrows are Grafana reading at query
+time. Locally (`just up` / `just demo`) there is no tunnel — everything
+talks over the compose network and Grafana is on `localhost:3000`.
+
 ## Demo: see it work in one command
 
 ```sh
@@ -52,9 +70,12 @@ config/
   loki.yaml             # logs
   tempo.yaml            # traces
   prometheus.yaml       # metrics
+  alertmanager.yaml     # alert routing (webhook + watchdog heartbeat)
   alerts/               # Prometheus alert rules
   grafana/              # provisioned datasources + dashboard loader
 dashboards/             # drop JSON dashboards here; Grafana auto-loads them
+docs/                   # runbook, onboarding templates, ADRs, screenshots
+infra/                  # OpenTofu: Cloudflare tunnel, ingress routes, DNS
 ```
 
 ## Run
@@ -67,6 +88,11 @@ just up-tunnel          # core stack + Cloudflare Tunnel (needs CLOUDFLARE_TUNNE
 
 Grafana: <http://localhost:3000> (admin / whatever you set).
 
+The Cloudflare side of the tunnel (hostnames, DNS, the tunnel itself) is
+code too: `infra/` holds a small OpenTofu config whose output is the
+`CLOUDFLARE_TUNNEL_TOKEN` the overlay needs — bootstrap instructions in
+[infra/main.tf](infra/main.tf).
+
 `just check` validates everything (compose files, Prometheus config and
 alert rules, collector config, YAML, workflows, dashboard JSON) in pinned
 containers — no host installs. CI runs the same command on every push and
@@ -75,71 +101,29 @@ healthy.
 
 ## Sending telemetry from a project
 
-### Traces + metrics (OTLP)
-
-Point your app (or its local OTel Collector) at this host's OTLP endpoints:
-
-- gRPC: `<host>:4317`
-- HTTP: `<host>:4318`
-
-Do **not** publish those ports to the public internet. Expose the monitoring
-host via Cloudflare Tunnel, Tailscale, or WireGuard — the compose file binds
-them to `127.0.0.1` to make that the obvious path.
-
-### Logs (Loki Docker driver)
-
-Container stdout/stderr is best shipped directly by the Docker daemon using
-the Loki log driver plugin. On each project host:
-
-```sh
-# 1. Install the driver plugin (once per host)
-docker plugin install grafana/loki-docker-driver:latest \
-  --alias loki --grant-all-permissions
-
-# 2. Set LOKI_URL in the host's .env, e.g.
-#    LOKI_URL=https://logs.example.org/loki/api/v1/push
-```
-
-How the project wires it up is project-specific. The RELab repo, for
-example, uses an optional overlay (`compose.logging.loki.yml`) that the
-justfile auto-includes when `LOKI_URL` is set — hosts without Loki keep
-Docker's default json-file driver. Other projects can do the equivalent:
-set the daemon-wide `log-driver` in `/etc/docker/daemon.json`, or add a
-per-service `logging:` block.
+One OTLP endpoint (`<host>:4317` gRPC / `:4318` HTTP), bearer-token auth
+(`OTLP_AUTH_TOKEN`), and three naming conventions. Copy-paste templates for
+every ingestion form — zero-code Python/FastAPI, plain OTLP env vars, the
+Loki Docker driver, and Grafana Alloy for file logs — live in
+**[docs/ONBOARDING.md](docs/ONBOARDING.md)**.
 
-Logs carry labels for `service`, `env`, and `host` so you can filter in
-Grafana. Keep label cardinality low — don't add `user_id`, `request_id`,
-etc. as labels; use LogQL filters for those.
+Never publish 4317/4318 directly; the compose file binds them to
+`127.0.0.1` and the tunnel is the exposure path.
 
 ## Alerting
 
 Prometheus evaluates the rules in `config/alerts/` (target down, OTel
-export failures, >5% span error rate) and Grafana surfaces them under
-Alerting → Alert rules. There is deliberately no Alertmanager: on a
-single-host stack, another service buys routing complexity before anyone
-needs it. Add one (or a Grafana contact point) when alerts must page
-someone.
+export failures, >5% span error rate, disk >80%); Alertmanager delivers
+them to any webhook via `ALERT_WEBHOOK_URL`, and the always-firing
+`Watchdog` posts to `HEARTBEAT_URL` every 5 minutes — point that at a dead
+man's switch so you hear about it when the monitoring host itself dies.
+Both are optional; unset means alerts are visible in Grafana only.
+Operations (rotating tokens, disk pressure, backup/restore):
+**[docs/RUNBOOK.md](docs/RUNBOOK.md)**.
 
 ## Storage
 
 Everything persists to local Docker volumes (`loki_data`, `tempo_data`,
 `prometheus_data`, `grafana_data`). Swap Loki / Tempo storage to S3-compatible
 (Backblaze B2, Cloudflare R2, Hetzner, MinIO) when you outgrow local disk —
-`compose.storage-s3.yml` documents the concrete shape of that change, and
-`docs/adr/0001-observability-stack.md` records the architecture rationale.
-
-## Design decisions
-
-Everything enters through one gateway — the collector — so a project
-configures a single OTLP endpoint, and a backend can be swapped without
-touching any app. It runs on one host on purpose: CML's telemetry is a handful
-of services at single-digit requests per second, so distributed ingest would
-add operational weight for no gain, while one compose file stays auditable by a
-single person. When local disk runs short, S3-backed Loki and Tempo (above)
-are the documented way out.
-
-Logs carry only low-cardinality labels — `service`, `env`, `host` — and
-everything else is a query-time filter; high-cardinality labels are the usual
-way a Loki install falls over. And because Tempo's metrics generator derives
-RED metrics from spans, any service that sends traces gets the Service Health
-dashboard whether or not it emits metrics of its own.
+`compose.storage-s3.yml` documents the concrete shape of that change.

compose.demo.yml

@@ -1,39 +1,40 @@
 # Demo overlay: a tiny auto-instrumented FastAPI service plus a load
 # generator, so the stack has real traces, metrics, and correlated logs
-# to show. Not for production.
+# to show. Not for production — no restart-across-reboots on purpose.
 #
 #   just demo        # core stack + this overlay
-#   just demo-down
+#   just demo-down   # remove just the demo services
+
+x-demo-logging: &demo-logging
+  driver: json-file
+  options:
+    max-size: "10m"
+    max-file: "3"
 
 services:
   demo-api:
     build: ./demo
-    restart: unless-stopped
+    restart: on-failure
     depends_on: [ otel-collector ]
     environment:
       OTEL_SERVICE_NAME: demo-api
       OTEL_RESOURCE_ATTRIBUTES: env=demo
       OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
       OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
-      OTEL_TRACES_EXPORTER: otlp
-      OTEL_METRICS_EXPORTER: otlp
-      OTEL_LOGS_EXPORTER: otlp
+      OTEL_EXPORTER_OTLP_HEADERS: Authorization=Bearer ${OTLP_AUTH_TOKEN:?set OTLP_AUTH_TOKEN in .env}
+      # Traces/metrics/logs exporters all default to otlp via opentelemetry-distro.
       # Ship Python log records via OTLP with trace context attached.
       OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED: "true"
       OTEL_METRIC_EXPORT_INTERVAL: "10000"
       # Emit the *stable* HTTP semconv (http.server.request.duration in seconds,
       # http.route) rather than the legacy names. Stable metric names are frozen,
       # so the Service Health dashboard's app-SDK panel survives library bumps.
       OTEL_SEMCONV_STABILITY_OPT_IN: http
-    logging:
-      driver: json-file
-      options:
-        max-size: "10m"
-        max-file: "3"
+    logging: *demo-logging
 
   demo-load:
     image: curlimages/curl:8.21.0
-    restart: unless-stopped
+    restart: on-failure
     depends_on: [ demo-api ]
     command:
       - sh
@@ -44,8 +45,4 @@ services:
           curl -s -o /dev/null demo-api:8000/work
           sleep 1
         done
-    logging:
-      driver: json-file
-      options:
-        max-size: "10m"
-        max-file: "3"
+    logging: *demo-logging

compose.tunnel.yml

@@ -3,10 +3,9 @@
 #
 #   just up-tunnel
 #
-# Token-based tunnel: routes are configured in the Cloudflare Zero Trust
-# dashboard. Create a tunnel at dash.cloudflare.com → Zero Trust → Networks →
-# Tunnels, then set the public hostnames there (e.g. grafana.example.com →
-# http://grafana:3000, otlp.example.com → http://otel-collector:4318).
+# The tunnel, its public hostnames, and DNS are managed as code in infra/
+# (OpenTofu); `tofu output -raw tunnel_token` yields the token for .env.
+# See infra/main.tf for the bootstrap steps.
 
 services:
   cloudflared: