telemetry: emit grid-load pipeline timing to the companion (bespoke, session-correlated spans)

[paddymul]

tallyman is building a per-project activity/perf log and wants a timeline of a single grid load, but most of that pipeline is invisible outside buckaroo. The companion (tallyman) only sees the first browser request and its own /load_expr POST; everything after — summary-stats compute, whether the stat cache was hit, and the WebSocket row-range requests between the browser and the buckaroo server — happens inside buckaroo's server and the buckaroo-js-core widget. The row traffic flows browser ↔ server directly over the WS and never reaches the companion, so tallyman is structurally blind to it.

Proposal: instrument the buckaroo server and packages/buckaroo-js-core (widget) to emit OpenTelemetry spans for one grid load, correlated by the buckaroo session id, so a consumer (tallyman, or a Jaeger/Tempo backend) can render the waterfall.

Trace model

One trace per grid load, correlation keyed on the session id minted at /load_expr. Spans:

• load_expr (server) — POST received → build/replay ready. Continue trace context injected by the caller (tallyman puts it on the POST headers).
• summary_stats (server) — start/end, attributes cache_hit: bool, cache_path, n_columns, n_stats. (tallyman passes the stat-cache path but never learns hit/miss — only the server knows.)
• row_range (server) — one span per row-range request received over the WS: received_at, replied_at, start, end, n_rows, served-from-row-cache.
• row_request / first_paint (widget, buckaroo-js-core) — the browser side: when the widget issues its first row request and receives the first rows, plus subsequent requests. Continue the trace over the WS handshake.

Emission / propagation

• Server: opentelemetry-sdk + an OTLP exporter (env-configurable endpoint); accept/continue trace context on the /load_expr POST and on the WS connect.
• Widget: @opentelemetry/api + a web exporter; propagate context on the WS messages.
• Ingestion (tallyman side, decided at integration): (a) a custom OTLP→events.jsonl exporter in the companion so spans land in tallyman's Log UI, and/or (b) a standard collector (Tempo/Jaeger) for the waterfall.

Why this lives here, not in tallyman

The companion is structurally blind to the WS row traffic and to buckaroo's internal stat compute. Only the server and the widget observe these timings. tallyman captures the request→load_expr slice on its own side (Phase 1, tracked in the tallyman repo).

Context

Split out from tallyman's project-activity-log work: Phase 1 (in tallyman) lands the discrete event log (MCP build success/failure, alias creation, Buckaroo grid-load failures) plus the companion-visible load_expr timing. This issue is Phase 2 — the in-buckaroo pipeline timing. Touches the same /load_expr/session surface as #925 (per-session RSS) and #930 (JS backend).

[paddymul]

Resolved approach (design grill)

Walked the transport design and pivoted from a full OpenTelemetry SDK integration to bespoke, OTel-shaped emission. The reason: OTel's headline win — cross-process trace-context propagation — is nearly moot here, because every emitter already has the correlation key. The buckaroo session_id is minted/received by the companion, owned by the server, and is the widget's wsUrl (ws://…/ws/{session_id}). And the timings only need to land in the companion's own UI, not a Jaeger/Tempo backend. So we POST span-shaped records to the companion and skip the SDK + collector + propagation plumbing. Records stay OTel-shaped so an OTLP exporter is a later transport swap, not a re-instrumentation.

Decisions

#	Decision	Resolved
1	Transport	Bespoke, OTel-shaped records, correlated by session_id
2	Scope	trace = session_id; build initial_load (load → first paint) first; per-row-range streaming deferred to v2
3	Records + emission	Bridge perf_log — reuse the existing firstpull.* spans; flat {trace, source, name, t_start_ms, t_end_ms, attrs}; wall-clock (single machine → directly comparable, no parent_id, no clock sync)
4	Server → companion	Direct POST to the companion's /internal/telemetry; telemetry_url passed in the /load_expr payload
5	Gating	telemetry_url present → enable perf_log + set the sink; absent → no-op (normal buckaroo usage unaffected). Default-on for initial_load (once per load, cheap)
6	Storage / render	Companion side: traces/<session>.jsonl + a waterfall in the companion's Log UI

buckaroo-side work (this issue)

• perf_log: add (a) a telemetry sink (POST span records to a URL) and (b) a session_id + source contextvar. perf_span captures wall-clock t_start/t_end and POSTs to the sink when one is set. The existing firstpull.load_expr / expr_load / dataflow_construct / metadata / summary_stats / ws_first_payload spans become telemetry for free.
• /load_expr handler: read telemetry_url from the payload → if present, enable perf_log, set the sink, set the session contextvar.
• WS handler (/ws/<session_id>): set the session contextvar so the first-pull spans (summary_stats, ws_first_payload) are tagged.
• summary_stats span: attach the cache_run_stats() attrs (status, hits, misses, secs) landed in feat(stats): expose summary-stat cache hit/miss + timing as a structured signal #944.
• Widget (packages/buckaroo-js-core): spans for first_row_request + rows_received, POSTed same-origin with the session_id from wsUrl. Deferred — see MVP.

MVP cut

Ship the server + companion spans first; the widget spans are a fast-follow. In the motivating case the entire cold-load cost was the /load_expr POST (server-side), so the firstpull.* breakdown answers "where do the 6.9 s go" (dataflow_construct vs metadata vs summary_stats) without touching buckaroo-js-core or rebuilding the widget. Add the widget spans once the server breakdown shows whether the bottleneck is even past load_expr.

Companion side (not this repo)

The /internal/telemetry receiver, the traces/<session>.jsonl store, the telemetry_url plumbing (the companion threading its own host:port down to the load_expr payload), and the Log-tab waterfall live in tallyman and are tracked there.