feat(observability): wire OTel tracing end-to-end with Tempo (post-MVP Phase 0)

[ivanprytula]

Summary

Post-MVP NFR roadmap Phase 0: make the already-coded OTel tracing actually work. Tracing was a silent no-op for three stacked root causes — none of them the one the roadmap suspected:

1. The ingestor image never installed the OTel packages — Dockerfile had --extra ai but not --extra tracing, so setup_tracing() degraded to a no-op in every container. → added --extra tracing to the uv sync.
2. No OTLP collector existed in any compose file — spans exported into the void. → added Grafana Tempo behind the monitoring profile (infra/monitoring/tempo.yml, OTLP gRPC :4317, loopback-exposed for host-run dev).
3. setup_tracing() ran in the lifespan — too late. Starlette builds its middleware stack when the first ASGI scope (the lifespan event) arrives, so FastAPIInstrumentor's middleware never entered the request path: Tempo received only SQLAlchemy connect spans, zero HTTP spans, and logs had no trace_id. → moved the call to module level after the last add_middleware, making the OTel span outermost so request_start/request_end logs run inside it.

Premise correction: the except ImportError, AttributeError, ... clause at libs/platform/tracing.py:120 is valid Python ≥3.14 (PEP 758 re-legalized unparenthesized multi-exception tuples) — it was never syntax-dead on this repo's pinned runtime. Parenthesized anyway for readability and cross-version portability.

Changes

• Tempo service (grafana/tempo:2.7.2) in docker-compose.yml monitoring profile + just up-monitoring
• Grafana correlation both directions: Tempo datasource with tracesToLogsV2 → Loki (service.name maps to the promtail container label; OTEL_SERVICE_NAME=api-obs-ingestor keeps them identical), and a Loki derived field extracting trace_id → Tempo
• Compose defaults OTEL_ENABLED=true (overridable via .env; export failures non-fatal when monitoring is down); .env.example + config descriptions updated Jaeger → Tempo
• Regression tests: get_trace_id() returns the active span's 32-hex trace ID, degrades to None outside a span (pytest.importorskip pattern, consistent with the ai extra tests)

Verification (live, docker compose)

• otel_tracing_initialized logged with endpoint=http://tempo:4317
• GET /api/v1/observations → request_start/request_end JSON logs carry trace_id
• That exact trace ID fetched from Tempo's API returns the GET /api/v1/observations HTTP server span (+ SQLAlchemy connect span)
• Same log lines queryable in Loki via promtail; Grafana provisions all three datasources cleanly (loki, tempo uids pinned)
• Unit suite: 221 passed, 4 skipped; all pre-commit hooks green

Notes

• docs/.plans/post-mvp-nfr-roadmap.md is gitignored, so the Phase 0 status/premise corrections live on local disk only — the substance is recorded here and in the commit message.

🤖 Generated with Claude Code