Skip to content

Commit 7910018

Browse files
authored
fix(vllm): non-streaming tool-call regression after #10351 (#10638)
fix(vllm): non-streaming tool-call regression after #10351 (native_streaming is a capability flag, not a state flag) #10351 introduced native streaming via `parser.extract_tool_calls_streaming` and gated the post-loop `extract_tool_calls` block on `native_streaming and not native_streaming_error`. That works for streaming requests, but for non-streaming requests the same flag is still True (it only means "the parser can stream", not "we actually streamed"), so the block was skipped and the `elif` cleared `content = ""` — the tool call was silently lost. Symptom: non-streaming chat.completions with `tools=[...]` returns `finish_reason: "stop"` with `content: ""` and no `tool_calls`. Streaming requests are unaffected. Fix: gate both branches on `streaming` too, so the extract_tool_calls block runs for non-streaming requests (and for streaming requests that fell back to the buffered path). Reproduction (vLLM 0.24, Qwen3-Coder-Next-NVFP4, qwen3_coder parser): curl -s -X POST http://localhost:8080/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"model":"coder","stream":false, "messages":[{"role":"user","content":"7*8 via calc"}], "tools":[{"type":"function","function":{"name":"calc", "parameters":{"type":"object", "properties":{"expression":{"type":"string"}}}}}]}' Before: finish_reason: "stop", content: "", tool_calls: [] After: finish_reason: "tool_calls", tool_calls[0].function.name: "calc" Streaming path re-verified in the same setup: delta.tool_calls arrives token-by-token, finish_reason: "tool_calls", no raw XML in content. Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
1 parent 1a03712 commit 7910018

1 file changed

Lines changed: 7 additions & 2 deletions

File tree

backend/python/vllm/backend.py

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -748,7 +748,12 @@ async def _predict(self, request, context, streaming=False):
748748
# When (A) native streaming ran cleanly, per-delta yields above already
749749
# delivered everything — do NOT extract again on the full text or we'd
750750
# duplicate content/tool_calls into the final chunk.
751-
if has_tool_parser and not (native_streaming and not native_streaming_error):
751+
# NOTE: `native_streaming` is a capability flag ("streaming parser is
752+
# available"), not a state flag ("streaming actually ran"). For
753+
# non-streaming requests it is still True but the per-delta loop was
754+
# never entered, so we MUST still run extract_tool_calls here. Hence
755+
# the explicit `streaming and …` guard on both branches.
756+
if has_tool_parser and not (streaming and native_streaming and not native_streaming_error):
752757
try:
753758
tp = tp_instance
754759
if tp is None:
@@ -770,7 +775,7 @@ async def _predict(self, request, context, streaming=False):
770775
))
771776
except Exception as e:
772777
print(f"Tool parser error: {e}", file=sys.stderr)
773-
elif native_streaming and not native_streaming_error:
778+
elif streaming and native_streaming and not native_streaming_error:
774779
# Per-delta path already emitted content + tool_calls; the final
775780
# chat_delta should carry only metadata (token counts, logprobs).
776781
content = ""

0 commit comments

Comments
 (0)