fix(reasoning): prevent streaming end-token desync in base and other parsers#39044
fix(reasoning): prevent streaming end-token desync in base and other parsers#39044kaiisfree wants to merge 1 commit into
Conversation
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add If you have any questions, please reach out to us on Slack at https://slack.vllm.ai. Agent GuidelinesIMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban. 🚀 |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request modifies the streaming reasoning parsers in vllm/reasoning/ to better handle cases where token IDs are received before their corresponding text is flushed from the buffer. By shifting the primary checks from token IDs to the actual delta_text and adding explicit waits when a token ID is present without its text, the changes ensure more reliable extraction of reasoning content. I have no feedback to provide.
…parsers When stop sequences are configured, output_text_buffer_length can delay visible text while token IDs arrive immediately. This causes reasoning parsers to misroute </think> tags into content. Fix: check for end token in delta_text first (resilient to buffering), use token IDs only as a secondary signal. If token ID arrives but text is still buffered, skip the chunk and wait for the text flush. Fixes the same bug class as vllm-project#38864 (Qwen3-specific) but in the base parser and other affected parsers: DeepSeek-R1, Ernie45, Step3p5. Related: vllm-project#38789 Signed-off-by: kaiisfree <letkaibefree@yahoo.com>
kaiisfree
force-pushed
the
fix/reasoning-parser-streaming-desync
branch
from
1203200 to
0e5d2a6
Compare
… in streaming When stop sequences set output_text_buffer_length > 0, token IDs arrive in delta_token_ids before their text is flushed into delta_text. Without a guard, find() returns -1 and the reasoning/content split is silently corrupted. Add text-presence checks before both find() calls in extract_reasoning_streaming: - </think> end token path (line 215) - <|tool_calls_section_begin|> section start path (line 223) Return None (wait for flush) when the token ID is present but the text is not, matching the fix pattern from PR vllm-project#39044 (BaseThinkingReasoningParser / DeepSeekR1) and PR vllm-project#40352 (Step3ReasoningParser). Fixes vllm-project#41067 Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: <>
… in streaming When stop sequences set output_text_buffer_length > 0, token IDs arrive in delta_token_ids before their text is flushed into delta_text. Without a guard, find() returns -1 and the reasoning/content split is silently corrupted. Add text-presence checks before both find() calls in extract_reasoning_streaming: - </think> end token path (line 215) - <|tool_calls_section_begin|> section start path (line 223) Return None (wait for flush) when the token ID is present but the text is not, matching the fix pattern from PR vllm-project#39044 (BaseThinkingReasoningParser / DeepSeekR1) and PR vllm-project#40352 (Step3ReasoningParser). Fixes vllm-project#41067 Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Keyi Li <likey6688@gmail.com>
… in streaming When stop sequences set output_text_buffer_length > 0, token IDs arrive in delta_token_ids before their text is flushed into delta_text. Without a guard, find() returns -1 and the reasoning/content split is silently corrupted. Add text-presence checks before both find() calls in extract_reasoning_streaming: - </think> end token path (line 215) - <|tool_calls_section_begin|> section start path (line 223) Return None (wait for flush) when the token ID is present but the text is not, matching the fix pattern from PR vllm-project#39044 (BaseThinkingReasoningParser / DeepSeekR1) and PR vllm-project#40352 (Step3ReasoningParser). Fixes vllm-project#41067 Co-authored-by: Claude <noreply@anthropic.com> Signed-off-by: Keyi Li <likey6688@gmail.com>
1 participant
Summary
Fixes text/token-ID desync in streaming reasoning parsers when
stopsequences are configured. PR #38864 fixed this for Qwen3 only — this PR applies the same fix pattern to:BaseThinkingReasoningParser(basic_parsers.py) — fixes SeedOSS, Gemma4, Step3p5, and Mistral parsers via inheritanceDeepSeekR1ReasoningParser— fixes NemotronV3 and DeepSeekV3 via inheritanceErnie45ReasoningParser— standalone fixStep3p5ReasoningParser— standalone compatibility path fixRoot cause
When
stopsequences setoutput_text_buffer_length, visible text is delayed while token IDs arrive immediately. Parsers checkedend_token_id in delta_token_idsand assumed the end token string was indelta_text— but it was still buffered. This causeddelta_text.find(end_token)to return -1 and misroute the</think>tag into content.Fix pattern
self.end_token in delta_textfirst (text-based, resilient to buffering)end_token_idis indelta_token_idsbut text hasn't arrived yet, returnNone(wait for flush)Same pattern as #38864 (Qwen3).
Parsers affected
Related: #38789, #38864, #17468
Test plan
stopsequences on models using base parser