feat: Make the stack runnable, demoable, and tested#2
Merged
Conversation
Move cloudflared into an optional compose.tunnel.yml overlay (pinned to 2026.6.1 instead of :latest) so a fresh clone + .env.example + 'just up' brings up Grafana locally. 'just up-tunnel' is the production path.
compose.demo.yml runs a tiny auto-instrumented FastAPI service plus a curl load loop, emitting real OTLP traces, metrics, and trace-correlated logs. 'just demo' brings up the core stack with the overlay.
Target-down, OTel export failures, and >5% span error rate (from Tempo span metrics). Evaluated by Prometheus, surfaced in Grafana Alerting; no Alertmanager to keep the stack minimal.
check validates compose files (incl. overlays), Prometheus config+rules via promtool, the collector config via otelcol validate, yamllint, actionlint, and dashboard JSON — all in pinned containers, no host installs. CI runs 'just check' plus a smoke job that waits for Grafana /api/health. Also fixes a trailing space yamllint caught in dependabot.yml.
Prometheus 3.x replaced the otlp-write-receiver feature flag with --web.enable-otlp-receiver; with the old flag the collector's metrics export 404ed and all OTLP metrics were silently dropped (surfaced by the new demo). Also add a Loki derived field so trace_id arriving as OTLP structured metadata links to Tempo, not just trace ids in the log body.
Request rate, error rate, and latency quantiles from Tempo span metrics (with exemplars linking to traces), p95 from the service's own OTLP SDK metrics, and a correlated logs panel. Works for any service that sends traces, selected via a service variable. Demo app now logs at INFO so non-error lines ship too.
Keeps CI a thin wrapper around the just contract; the same boot test now runs locally with 'just smoke'.
…gelog README gains a Mermaid diagram, a one-command demo section up top, a genuine screenshot of the Service Health dashboard rendered from the running demo, run/check/alerting docs, and a design-decisions section. CHANGELOG.md starts the release history at v0.1.0.
Docstrings, return annotations, a named ERROR_RATE constant, targeted S311 suppressions (randomness is jitter, not crypto), and an __init__.py so the demo dir isn't an implicit namespace package.
ADR records the single-host OTLP-native architecture decision and its alternatives; compose.storage.s3.yml is a deliberately commented stub documenting the concrete Loki/Tempo → S3 path the README mentions.
…h dashboard queries
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before: you needed a Cloudflare token just to start the stack, and there was no way to see it working or to validate changes. Now: