feat(llmobs): realtime audio-timing metadata for full-conversation playback#19112
Draft
ZStriker19 wants to merge 17 commits into
Draft
feat(llmobs): realtime audio-timing metadata for full-conversation playback#19112ZStriker19 wants to merge 17 commits into
ZStriker19 wants to merge 17 commits into
Conversation
Auto-instruments client.realtime.connect (sync + async): a workflow span for the session and an llm child span per response turn, capturing input/output transcripts, token usage, session config, and audio_parts for renderable formats. Realtime audio is raw PCM by default (not renderable), so those turns keep the transcript as content. Adds reusable audio helpers (realtime_audio_format_to_mime, is_renderable_audio_mime, concat_base64_audio, format_audio_part_with_guard with a 5MB inline size guard) and an activate flag on BaseLLMIntegration.trace so long-lived/child spans don't disturb the active trace context.
- Tag in-flight response spans on close instead of finishing them empty. - Record client send events only after the underlying send succeeds. - Observe server events via parse_event (covers recv, iteration, and the manual recv_bytes()+parse_event() path) instead of recv. - Handle input_audio_buffer.clear/cleared to drop discarded audio. - Prune the input-transcript map (pop on assign, clear on close). - Skip non-user items in conversation.item.create input absorption.
Records the deferred-by-design gaps (late input transcription after response.done, out-of-band response.create inline input, single pending-input turn) in the module docstring.
Wrap raw PCM16 realtime audio (the API default, 24kHz mono) in a WAV container so it is emitted as a playable audio/wav audio_part instead of being dropped as non-renderable. The 4MB inline size guard runs on the WAV bytes and debug-logs when oversize audio is dropped (transcript is kept). G.711 (pcmu/pcma) stays unwrapped and falls back to the [audio] marker. Captures the input/output audio sample rate from the session config.
…ate transcripts From live testing feedback: - Drop the realtime session workflow span; each turn is its own llm span (its own trace) carrying the session config as metadata, and all turns of a connection share a session_id so the UI groups them. Keeps each trace one-turn small (per-event size headroom) and renders cleanly. - Capture input-audio transcriptions that arrive after response.done: hold the turn span open until the transcript lands (matched by item_id), with fallbacks on the next response.created and on close so spans never leak.
openai>=1.66's RealtimeConnection.recv_bytes() asserts the underlying recv(decode=False) returns bytes; the test double returned the scripted JSON as str, failing the integration test on the openai==1.66.0 CI venv. Encode str messages to bytes.
Address PR review: - Type-annotate ddtrace/contrib/internal/openai/_realtime.py fully and drop the per-module mypy override (the new code is now type-safe). - The inline-audio size guard now measures the base64-encoded size (what actually rides the span event, ~4/3 the raw bytes) instead of the raw byte count, so it can't admit audio that exceeds the per-span-event limit after encoding.
From a second Codex + subagent review pass: - Only defer a turn's span for a late input transcription when transcription is actually enabled in the session config; otherwise finalize at response.done (no transcript is coming, so waiting needlessly delayed every turn and held the last turn until close). - Handle conversation.item.input_audio_transcription.failed: finalize any turn awaiting that item so its span can't hang on an idle session. - Stop the input-transcript cache from leaking: pop each turn's entry on finalize (every finalize path) instead of only when the transcript was unset. - Finalize the session when the connection closes mid-iteration without with/close(): the recv wrappers detect ConnectionClosed and call finish_session (idempotent). - Derive total_tokens from input+output when usage omits it (matches chat/responses). Adds tests for each plus an async-connection integration test.
The module docstring is never rendered as Markdown/RST, so drop the inline-literal backticks and bold/italic markup for plain-text readability (review nit).
G.711 (audio/pcmu / audio/pcma) is the codec for Realtime phone-call integrations (Twilio/SIP). Decode it to PCM16 with a dependency-free mu-law/A-law decoder (the stdlib wave module can't emit G.711 WAV and audioop was removed in 3.13) and WAV-wrap at 8kHz so those turns produce a playable audio/wav audio_part instead of an [audio] marker.
Realtime agents call tools, but turns only captured audio/transcripts before. Parse response.done output items for function_call and mcp_call and attach them as tool_calls on the turn's llm span (MCP results, which run server-side, ride along as tool_results). The function_call_output the app feeds back is captured as a tool_result on the next turn's input. Tool-call-only turns (no audio/text) still emit a span.
The function_call_output event the app returns carries only call_id + output (no name), so the tool result showed as 'unknown' in the UI. Correlate the call_id back to the originating function_call's name and set it on the ToolResult.
… shadow metrics - Move audio helpers to a dedicated ddtrace/llmobs/_integrations/audio_utils.py (re-exported from utils.py) so the growing audio surface lives on its own. - Add a DD_OPENAI_REALTIME_ENABLED kill-switch to patch_realtime(). - Cap accumulated realtime audio via _AudioAccumulator so a long turn can't grow unbounded before the size guard runs; free chunks once oversize. - Apply shadow metrics on realtime LLM spans for APM parity. - Add multi-turn output-audio, tool-call, and patch/unpatch tests.
…ation playback Realtime turn spans now carry absolute (unix ns) audio-timing anchors in metadata (dd.llmobs.audio.input/output.start_time_unix_nano, input.speech_end_time_unix_nano), captured on the tracer's own clock as each segment is observed. These let a full-conversation audio playback UI place each turn's audio on one shared timeline and measure time-to-response. Text-only and tool-only turns carry no timing keys. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Codeowners resolved as |
|
Rename the realtime audio-timing metadata keys from dd.llmobs.audio.* to _dd.llmobs.audio.* so they are treated as internal and hidden from the LLMObs metadata UI, while still available to the full-conversation-playback feature. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ch config Address PR review follow-ups for the OpenAI Realtime API instrumentation: - add realtime wrapped-method assertions to the canonical OpenAI patch test - document DD_OPENAI_REALTIME_ENABLED in the release note - register DD_OPENAI_REALTIME_ENABLED in supported-configurations.json (and regenerate _supported_configurations.py) so the config-registry check passes Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e-audio-timing # Conflicts: # ddtrace/llmobs/_integrations/utils.py # tests/llmobs/test_integrations_utils.py
Circular import analysis
|
BenchmarksBenchmark execution time: 2026-07-16 21:09:17 Comparing candidate commit 03b3b4f in PR branch Found 0 performance improvements and 4 performance regressions! Performance is the same for 443 metrics, 1 unstable metrics. scenario:iast_aspects-re_expand_aspect
scenario:span-start
scenario:telemetryaddmetric-1-count-metric-1-times
scenario:tracer-small
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft / eval branch — not intended to merge as-is. Stacks on #18759 (base branch
llmobs-openai-realtime).What
Adds absolute (unix ns) audio-timing metadata to each OpenAI Realtime turn span so a full-conversation audio playback experience can place each turn's audio on one shared timeline and measure time-to-response. The base realtime integration captures audio + transcripts but no timing.
New span metadata keys, under the internal
_dd.prefix so they are hidden from the metadata UI (omitted when a turn has no audio):_dd.llmobs.audio.input.start_time_unix_nano— first user-audio chunk of the turn_dd.llmobs.audio.output.start_time_unix_nano— firstresponse.audio.deltaof the turn_dd.llmobs.audio.input.speech_end_time_unix_nano—input_audio_buffer.committedTimestamps are captured on the tracer's own clock as each segment is observed (provider-agnostic), and emitted as metadata, not tags (timestamps are high-cardinality).
How
_AudioAccumulator.start_ns(set on first chunk) gives both the input and output segment starts;_InputTurn.speech_end_nsset on commit._audio_timing(turn)assembles the keys; threaded through_llmobs_set_tags_from_realtime_response, which merges them into span metadata.Tests
test_realtime_state_audio_turn_carries_timing_anchors— three keys present,input.start <= speech_end <= output.start.test_realtime_state_text_only_turn_has_no_audio_timing— no keys on a text-only turn.Ran
pytest tests/contrib/openai/test_openai_realtime.py(editable ddtrace, openai 2.45.0): 24 passed including both new tests. The 5 errored cases are the pre-existingtest_realtime_integration_*tests, which need the full ddtrace test-harness fixtures (DummyWriter tracer / pytest-asyncio) not present in the minimal venv used here — unrelated to this change. Also verified end-to-end by driving the real state machine and confirming acreateRealtimeResponsespan with the_dd.llmobs.audio.*metadata POSTs to the staging intake (HTTP 202).Part of the full-conversation-audio-playback effort (RFC + spec authored separately).
🤖 Generated with Claude Code