feat(llmobs): instrument openai realtime api for audio#18759
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 21 commits intoJul 18, 2026
Conversation
Auto-instruments client.realtime.connect (sync + async): a workflow span for the session and an llm child span per response turn, capturing input/output transcripts, token usage, session config, and audio_parts for renderable formats. Realtime audio is raw PCM by default (not renderable), so those turns keep the transcript as content. Adds reusable audio helpers (realtime_audio_format_to_mime, is_renderable_audio_mime, concat_base64_audio, format_audio_part_with_guard with a 5MB inline size guard) and an activate flag on BaseLLMIntegration.trace so long-lived/child spans don't disturb the active trace context.
Codeowners resolved as |
Contributor
🎉 All green!🧪 All tests passed 🔗 Commit SHA: eab4cc8 | Docs | Datadog PR Page | Give us feedback! |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c45b55f71e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
- Tag in-flight response spans on close instead of finishing them empty. - Record client send events only after the underlying send succeeds. - Observe server events via parse_event (covers recv, iteration, and the manual recv_bytes()+parse_event() path) instead of recv. - Handle input_audio_buffer.clear/cleared to drop discarded audio. - Prune the input-transcript map (pop on assign, clear on close). - Skip non-user items in conversation.item.create input absorption.
Records the deferred-by-design gaps (late input transcription after response.done, out-of-band response.create inline input, single pending-input turn) in the module docstring.
Wrap raw PCM16 realtime audio (the API default, 24kHz mono) in a WAV container so it is emitted as a playable audio/wav audio_part instead of being dropped as non-renderable. The 4MB inline size guard runs on the WAV bytes and debug-logs when oversize audio is dropped (transcript is kept). G.711 (pcmu/pcma) stays unwrapped and falls back to the [audio] marker. Captures the input/output audio sample rate from the session config.
…ate transcripts From live testing feedback: - Drop the realtime session workflow span; each turn is its own llm span (its own trace) carrying the session config as metadata, and all turns of a connection share a session_id so the UI groups them. Keeps each trace one-turn small (per-event size headroom) and renders cleanly. - Capture input-audio transcriptions that arrive after response.done: hold the turn span open until the transcript lands (matched by item_id), with fallbacks on the next response.created and on close so spans never leak.
openai>=1.66's RealtimeConnection.recv_bytes() asserts the underlying recv(decode=False) returns bytes; the test double returned the scripted JSON as str, failing the integration test on the openai==1.66.0 CI venv. Encode str messages to bytes.
Address PR review: - Type-annotate ddtrace/contrib/internal/openai/_realtime.py fully and drop the per-module mypy override (the new code is now type-safe). - The inline-audio size guard now measures the base64-encoded size (what actually rides the span event, ~4/3 the raw bytes) instead of the raw byte count, so it can't admit audio that exceeds the per-span-event limit after encoding.
From a second Codex + subagent review pass: - Only defer a turn's span for a late input transcription when transcription is actually enabled in the session config; otherwise finalize at response.done (no transcript is coming, so waiting needlessly delayed every turn and held the last turn until close). - Handle conversation.item.input_audio_transcription.failed: finalize any turn awaiting that item so its span can't hang on an idle session. - Stop the input-transcript cache from leaking: pop each turn's entry on finalize (every finalize path) instead of only when the transcript was unset. - Finalize the session when the connection closes mid-iteration without with/close(): the recv wrappers detect ConnectionClosed and call finish_session (idempotent). - Derive total_tokens from input+output when usage omits it (matches chat/responses). Adds tests for each plus an async-connection integration test.
The module docstring is never rendered as Markdown/RST, so drop the inline-literal backticks and bold/italic markup for plain-text readability (review nit).
KowalskiThomas
approved these changes
Jun 29, 2026
G.711 (audio/pcmu / audio/pcma) is the codec for Realtime phone-call integrations (Twilio/SIP). Decode it to PCM16 with a dependency-free mu-law/A-law decoder (the stdlib wave module can't emit G.711 WAV and audioop was removed in 3.13) and WAV-wrap at 8kHz so those turns produce a playable audio/wav audio_part instead of an [audio] marker.
Realtime agents call tools, but turns only captured audio/transcripts before. Parse response.done output items for function_call and mcp_call and attach them as tool_calls on the turn's llm span (MCP results, which run server-side, ride along as tool_results). The function_call_output the app feeds back is captured as a tool_result on the next turn's input. Tool-call-only turns (no audio/text) still emit a span.
The function_call_output event the app returns carries only call_id + output (no name), so the tool result showed as 'unknown' in the UI. Correlate the call_id back to the originating function_call's name and set it on the ToolResult.
ncybul
reviewed
Jun 30, 2026
… shadow metrics - Move audio helpers to a dedicated ddtrace/llmobs/_integrations/audio_utils.py (re-exported from utils.py) so the growing audio surface lives on its own. - Add a DD_OPENAI_REALTIME_ENABLED kill-switch to patch_realtime(). - Cap accumulated realtime audio via _AudioAccumulator so a long turn can't grow unbounded before the size guard runs; free chunks once oversize. - Apply shadow metrics on realtime LLM spans for APM parity. - Add multi-turn output-audio, tool-call, and patch/unpatch tests.
ncybul
reviewed
Jul 10, 2026
ncybul
approved these changes
Jul 10, 2026
ncybul
left a comment
ncybul
left a comment
Contributor
There was a problem hiding this comment.
A couple of remaining nits (pls ignore if you already fixed and I just missed it)!
…ch config Address PR review follow-ups for the OpenAI Realtime API instrumentation: - add realtime wrapped-method assertions to the canonical OpenAI patch test - document DD_OPENAI_REALTIME_ENABLED in the release note - register DD_OPENAI_REALTIME_ENABLED in supported-configurations.json (and regenerate _supported_configurations.py) so the config-registry check passes Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
# Conflicts: # ddtrace/llmobs/_integrations/utils.py # tests/llmobs/test_integrations_utils.py
Circular import analysis
|
The ast-grep `os-environ-fix-access` rule (and the ast-grep CI job) bans direct os.environ access. Read DD_OPENAI_REALTIME_ENABLED through ddtrace.internal.settings.env instead. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The pre-check spelling job (codespell) flags `datas` -> `data`. Rename the local variable in test_realtime_integration_multi_turn_with_output_audio. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ration Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Sphinx (sphinxcontrib.spelling) build flags "Realtime" in the OpenAI integration docstring. Add it to docs/spelling_wordlist.txt, alongside similar product names like "Anthropic". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
LLM Observability: auto-instruments the OpenAI Realtime API (
client.realtime.connect(...), sync and async), the only streaming-audio surface in the OpenAI SDK (chat-completions streaming carries no audio).The Realtime API is a bidirectional WebSocket event stream, so it can't reuse the request/response streaming path. Instead, this wraps the connection's
send/recv/closeat the class level (every typed sub-resource send funnels throughRealtimeConnection.send, and iteration funnels throughrecv) and feeds each observed event into a_RealtimeStatemachine that produces:response.created→response.done), capturing the user and assistant transcripts as message content, token usage, and audio.Audio is emitted as
audio_partsonly for renderable formats. The Realtime API's default raw PCM (audio/pcm|pcmu|pcma) is not renderable, so those turns keep the transcript as message content (the playable-audio path is generic and future-proof). A reusable 5 MB inline size guard (format_audio_part_with_guard) keeps audio under the per-span-event limit.Builds on the audio
audio_partsfoundation (#18622) and the audio-in-chat work (#18695).Testing
realtime_audio_format_to_mime,is_renderable_audio_mime,concat_base64_audio,format_audio_part_with_guard)._RealtimeStateunit tests driving full turns (PCM transcripts-only, renderableaudio_parts, session metadata, failed-status error, idempotent close, audio-only fallback marker).RealtimeConnectionover a fake websocket, asserting the resulting session + per-response LLMObs spans.lint(fmt/style/typing/spelling/suitespec/error-log) all pass.Parametric/system tests are not included here, consistent with the audio-feature PRs that preceded this (#18622, #18695).
Risks
Low. Realtime instrumentation is gated on LLMObs being enabled and is fully wrapped in defensive try/except so it can never break a user's connection. Class-method wrapping is unpatched cleanly in
unpatch(). No changes to existing chat/responses/completions paths beyond a backward-compatibleactivatekwarg onBaseLLMIntegration.traceand a_get_model_providerrefactor (covered by existing tests).Additional Notes
Claude session:
57bda7ea-70ec-482f-a5fe-973dbf97f8c6Resume:
claude --resume 57bda7ea-70ec-482f-a5fe-973dbf97f8c6