Skip to content

Fix OpenAI messages empty stream handling#3808

Open
stgmt wants to merge 10 commits into
Wei-Shaw:mainfrom
stgmt:fix/openai-messages-empty-stream-errors
Open

Fix OpenAI messages empty stream handling#3808
stgmt wants to merge 10 commits into
Wei-Shaw:mainfrom
stgmt:fix/openai-messages-empty-stream-errors

Conversation

@stgmt

@stgmt stgmt commented Jul 7, 2026

Copy link
Copy Markdown

Summary

Fix the OpenAI /v1/messages Anthropic-compat streaming bridge so pre-output upstream failures are not surfaced to Claude-compatible clients as successful empty streams or generic 502s.

This patch:

  • pre-estimates input tokens from the converted Responses request and uses that value in Anthropic message_start.usage.input_tokens
  • buffers initial Anthropic SSE events until real visible assistant output or tool output starts
  • treats terminal OpenAI Responses streams with no visible assistant/tool output as failover errors before any client bytes are written
  • maps upstream context_length_exceeded / no-visible-output client failures to non-retryable Anthropic-compatible 4xx errors instead of masking them as Upstream service temporarily unavailable
  • lets the messages failover loop retry only when no model output has actually reached the client
  • adds diagnostics for terminal Responses event types, output item types, usage, and context-window errors

Why

Claude-compatible clients can treat an Anthropic stream that only contains message_start / message_stop as a completed assistant turn even when OpenAI Responses actually ended with response.failed, response.incomplete, or reasoning-only output. In large-context sessions this hides actionable upstream errors such as context_length_exceeded behind an empty reply or a generic 502.

Tests

  • go test ./internal/pkg/apicompat -run TestStreamingMessageStartUsesPrefilledInputTokens -count=1
  • go test ./internal/service -run 'TestOpenAIMessagesAnthropicStream(ContextLengthBeforeVisibleOutputReturnsClientError|BuffersStartUntilVisibleOutput)' -count=1
  • go test ./internal/handler -run TestOpenAIHandleAnthropicFailoverExhaustedPreservesClientError -count=1

Notes

I did not run the full unit/integration/lint matrix locally; the change is covered by targeted regression tests around the affected bridge path.

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

All contributors have signed the CLA. ✅
Posted by the CLA Assistant Lite bot.

@stgmt

stgmt commented Jul 7, 2026

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

github-actions Bot added a commit that referenced this pull request Jul 7, 2026
Clears obsolete OpenAI/Codex quota-origin model_rate_limits when fresh Codex usage snapshots show quota headroom, while preserving upstream model-not-found and image cooldowns.\n\nRefs #1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant