Fix #278: bound the OpenAI-compatible ESC-cancel queue#310
Open
ericleepi314 wants to merge 1 commit into
Open
Conversation
…#278) The worker-thread ESC-cancel pattern used an unbounded queue.Queue: a pathological proxy that keeps sending bytes after ESC (and never closes the SDK iterator) accumulated chunks without limit while the consumer had already raised AbortError and left. - chunk_queue bounded at 64 (post-abort staleness drains trivially; the producer gets slack against transient consumer pauses) - the worker drops items and stops READING the stream once the abort trips or the consumer exits (consumer_gone, set via ExitStack on every consumer exit path) — without the latter, a consumer dying for a non-abort reason (on_text_chunk raising) would leave the worker retrying a full queue forever, pinning the httpx connection - a genuine stream error losing the race against ESC is logged at debug instead of vanishing Closes #278 Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This was referenced Jun 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #278
Summary
The ESC-cancel worker-thread pattern in
OpenAICompatibleProvider.chat_stream_responseused an unboundedqueue.Queue— flagged as a follow-up in PR #148. A pathological proxy that keeps sending bytes after ESC (and never closes the SDK iterator) accumulated megabytes while the consumer had already raisedAbortErrorand left._put_or_drop_on_abortstops the worker from reading the stream at all once the abort trips — the queue drains, the consumer hits the Empty tick, andAbortErrorraises within ~100ms even against a firehose.consumer_goneevent (from critic review): without it, the bounded queue introduced a new failure mode the unbounded one didn't have — a consumer that unwinds for a non-abort reason (e.g.on_text_chunkraising) left the worker retrying a full queue forever, pinning the httpx connection open. The consumer scope sets the event viacontextlib.ExitStackon every exit path; the worker bails within one 0.25s poll.test_in_loop_check_catches_abort_between_chunks); the worker-side drop is what guarantees nothing arrives after.Test plan
close(), yields forever) → AbortError <2s and the yield counter settles (worker stopped reading); consumer-crash (callback raises ValueError) → error propagates with class intact and the worker is released; 150-chunk backpressure happy path → exact content/order through the bounded queue🤖 Generated with Claude Code