feat: reliability hardening (auth, self-monitoring, alert delivery, backups)#4
Merged
Conversation
Accurate diagram (Loki named, tunnel fronts both Grafana and OTLP ingestion, Tempo span-metrics edge, theme-neutral styling) under a brief Architecture section that absorbs the Design-decisions prose and links ADR 0001.
The OTLP hostnames are public behind the tunnel; without auth anyone who finds them can inject telemetry. bearertokenauth on both receivers, token via OTLP_AUTH_TOKEN (safe local default keeps 'just demo' one command). Verified: no/wrong token → 401, correct token → 200, demo still populates Grafana. Also syncs justfile validator image pins with the bumped compose versions.
…orage node-exporter (pinned) + a HostDiskSpaceLow alert at 80% — Loki and Tempo have no total-size cap, so the disk alert is the storage backstop; Prometheus additionally gets --storage.tsdb.retention.size=15GB. New provisioned Stack Health dashboard: host disk/CPU/memory + collector ingest and export-failure rates. Collector 0.155.0 dropped the _total suffix on internal telemetry; alert and dashboard queries updated and verified against live data.
Alertmanager (pinned) routes all alerts to ALERT_WEBHOOK_URL and an always-firing Watchdog to HEARTBEAT_URL every 5m — wire the latter to a dead man's switch to catch the one failure the host can't report: its own death. URLs are injected via url_file at start (Alertmanager has no env expansion); unset means logged-and-harmless delivery errors, so the stack still runs with zero config. amtool check-config joins just check. Verified live: Watchdog delivered to a local webhook sink.
Tempo 3.0.2 (bumped in passing) crash-loops on the 2.x config: 3.0 is a re-architecture that removes the ingester/compactor blocks and changes the metrics-generator. Pin back to 2.10.4 (verified working) and tell Dependabot to skip Tempo majors so the migration happens deliberately. The OtelExportFailures alert flagged the outage — working as designed.
Crash-consistent tarball of all stateful volumes (services paused for the seconds the copy takes); restore wipes and repopulates them. Exercised end-to-end: backup → docker compose down -v → restore → a pre-backup Grafana annotation survived.
RUNBOOK: service-down triage, disk-pressure response with the full retention/size-limit table, backup/restore, secret rotation, image upgrades. ONBOARDING: endpoint + conventions plus verified copy-paste templates for zero-code Python/FastAPI, plain OTLP env vars, the Loki Docker driver, and Grafana Alloy file logs (Promtail is EOL). README slims: telemetry section now points at ONBOARDING, alerting section reflects Alertmanager + Watchdog.
The tunnel, its two ingress routes (grafana, otlp), and DNS were the only stack config living outside git. infra/ declares them (provider v5, validated); the tunnel-token output feeds .env for just up-tunnel. Local state on purpose — one host, one operator. tofu fmt joins just check; full validate is 'just infra-validate' (downloads the provider, so kept out of the CI hot path).
Rendered from the live stack; it happens to capture today's real Tempo outage and the backup/restore drill, which makes it a better runbook illustration than a clean one.
- restore: validate the tarball exists and is readable BEFORE stopping the stack and wiping volumes; absolute paths now work (a typo'd filename previously destroyed all data) - backup: always unpause even when the tar step fails; refuse to run under COMPOSE_PROJECT_NAME (hardcoded volume names would silently back up empty volumes); tarballs now 0600 (they contain secrets); service/ volume lists deduplicated into variables - demo-down: remove only the demo services instead of downing the entire stack - up-tunnel: refuse to expose the stack with the default OTLP token or the documented Grafana password - check: promtool/otelcol/amtool images are read from compose.yml so Dependabot bumps can't drift from the validators; dashboards JSON check runs in a jq container (recipe claim 'no host installs' is now true); yamllint ignores .git/, backups/, infra/.terraform/ - smoke: also fail on crash-looping/exited services, not just Grafana health All guards exercised live: typo'd restore fails before wipe, corrupt tarball fails before down, up-tunnel rejects defaults, demo-down leaves the core stack running.
- drop platform: linux/amd64 on the collector: the image is multi-arch, and the pin crash-loops arm64 Linux hosts / forces emulation on Apple Silicon (verified: native arm64 now runs) - Alertmanager writes webhook/heartbeat URLs to a tmpfs instead of the persistent volume, keeping secrets out of backup tarballs - OTLP_AUTH_TOKEN loses its weak 'local-dev-token' compose fallback: :?-guarded in compose.yml, single source of truth in .env; demo overlay no longer duplicates the literal - remove dead config: Grafana feature toggles absent from v13's registry, Tempo's unused OTLP HTTP receiver, redundant demo exporter env vars (distro defaults all three to otlp) - demo services restart: on-failure so a forgotten demo doesn't survive reboots; in-file logging anchor replaces duplicated blocks - infra: DNS record names must be FQDNs under cloudflare provider v5 (terraform-provider-cloudflare#5620) — fixes perpetual plan drift - fix stale stack.yaml header (claimed 'No Alertmanager') and stale tunnel overlay header (claimed click-ops config)
Demo package pins move to requirements.txt with a Dependabot pip ecosystem entry (previously nothing watched them), and the CI workflow gets an explicit read-only permissions block.
The tunnel only routes HTTPS to the collector's OTLP HTTP receiver — the endpoint table and diagram claimed a public gRPC :4317 path that doesn't exist. The Loki push templates now say the stack exposes no push hostname by default. Runbook: backup tarballs are 0600 and carry secrets; long backups (>5 min) drop telemetry; rotating the OTLP token requires re-running just demo if the overlay is up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Features
Ingestion now requires a token: the OTLP endpoint is public behind the tunnel, so anonymous senders get a 401. just up-tunnel refuses to start with default credentials.
Fixes