Skip to content

feat(session/llm): scaffold OpenAI Realtime (WebSocket) transport#29359

Draft
WonderJL wants to merge 1 commit into
anomalyco:devfrom
WonderJL:feat/openai-realtime-scaffold
Draft

feat(session/llm): scaffold OpenAI Realtime (WebSocket) transport#29359
WonderJL wants to merge 1 commit into
anomalyco:devfrom
WonderJL:feat/openai-realtime-scaffold

Conversation

@WonderJL
Copy link
Copy Markdown

Summary

  • Scaffolds an OpenAI Realtime API (WebSocket) transport for the session LLM stream.
  • Ships behind a new experimental.openai_realtime flag (default false) — no behavior change until the flag is enabled and the wiring lands.
  • Includes a pure event mapper (Realtime JSON → LLMEvent) with unit tests, a stub stream() that fails loudly, and a design note documenting the protocol mapping and open questions.

Why a scaffold, not a full transport?

The Realtime event surface is wide and additive; the right dispatch shape inside session/llm.ts is a maintainer call. Landing the scaffold first lets reviewers shape that integration without a churny diff. This PR is intentionally inert at runtime — stream() throws RealtimeNotImplemented if accidentally invoked.

What changes

  • packages/opencode/src/session/llm/realtime.ts (new) — adapterState(), toLLMEvents(), REALTIME_URL, RealtimeNotImplemented, stub stream(). The mapper shape mirrors ai-sdk.ts's adapterState() / toLLMEvents() on purpose so the eventual dispatch site treats both adapters uniformly.
  • packages/opencode/src/session/llm/REALTIME-DESIGN.md (new) — scope, non-goals, protocol mapping table, lifecycle plan, auth, open questions for maintainers.
  • packages/opencode/src/config/config.ts — adds experimental.openai_realtime: Schema.optional(Schema.Boolean).
  • packages/opencode/test/session/llm-realtime.test.ts (new) — four unit tests covering text turns, function-call argument streaming, unknown-event passthrough, and failed-response status.

Realtime → LLMEvent mapping (summary)

Realtime event LLMEvent
response.created stepStart
response.output_item.added (message) textStart
response.output_text.delta textDelta
response.output_text.done textEnd
response.output_item.added (function_call) toolInputStart
response.function_call_arguments.delta toolInputDelta
response.function_call_arguments.done toolInputEnd
response.output_item.done (function_call) toolCall (with parsed arguments)
response.done stepFinish + finish

Unknown event types (audio., rate_limits., etc.) drop to [] rather than fail.

Open questions (called out in the design note)

  1. Register Realtime as a separate providerID (openai-realtime) or keep it under openai and switch transport by flag?
  2. Reuse ProviderTransform.tools for tool-schema translation, or build a Realtime-specific path?
  3. How to treat gpt-5* reasoning items (which aren't first-class in Realtime yet) — block at status() or pass through?

Test plan

  • bun run typecheck (packages/opencode) — clean.
  • bun test test/session/llm-realtime.test.ts — 4/4 pass.
  • Once wiring lands in a follow-up: live smoke against gpt-4o-realtime-preview / gpt-5.x-realtime with a tool-call prompt.

Notes for reviewers

  • The scaffold deliberately stops short of opening a socket — happy to push the lifecycle (acquireRelease + AbortSignal bridge + idle timeout) in a follow-up once the integration shape is agreed.
  • This is independent of fix(session): add idle timeout to LLM streaming response #29348 (idle-timeout PR) but the eventual live wiring will reuse the same Stream.timeoutOrElse idle guard so both transports fail-fast on stalls.

Adds a scaffold for the OpenAI Realtime API (wss://api.openai.com/v1/realtime)
behind a new `experimental.openai_realtime` config flag. Not wired into the
live stream path yet — see REALTIME-DESIGN.md for protocol mapping, lifecycle
plan, and open questions for maintainers.

What ships:
- realtime.ts: pure event mapper (Realtime JSON -> LLMEvent) + adapterState +
  a stream() stub that fails with RealtimeNotImplemented so accidental wiring
  is loud.
- Config flag (default off) so the live wiring can land incrementally.
- Unit tests for the mapper covering text turns, function-call argument
  streaming, unknown-event passthrough, and failed-response status.

Default behavior unchanged.
@github-actions
Copy link
Copy Markdown
Contributor

ghost commented May 26, 2026

This PR doesn't fully meet our contributing guidelines and PR template.

What needs to be fixed:

  • PR description is missing required template sections. Please use the PR template.

Please edit this PR description to address the above within 2 hours, or it will be automatically closed.

If you believe this was flagged incorrectly, please let a maintainer know.

@github-actions
Copy link
Copy Markdown
Contributor

ghost commented May 26, 2026

Hey! Your PR title feat(session/llm): scaffold OpenAI Realtime (WebSocket) transport doesn't follow conventional commit format.

Please update it to start with one of:

  • feat: or feat(scope): new feature
  • fix: or fix(scope): bug fix
  • docs: or docs(scope): documentation changes
  • chore: or chore(scope): maintenance tasks
  • refactor: or refactor(scope): code refactoring
  • test: or test(scope): adding or updating tests

Where scope is the package name (e.g., app, desktop, opencode).

See CONTRIBUTING.md for details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs:compliance This means the issue will auto-close after 2 hours. needs:title

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant