fix(vllm): don't stream raw tool-call markup as content when a tool parser is active

When a tool_parser is configured and the request carries tools, the streaming
loop emitted every text delta as delta.content — including the model's raw
tool-call markup (e.g. <tool_call>...) — because extract_tool_calls only runs
on the full output after the stream. Clients streaming a tool call therefore
saw the unparsed tool-call syntax as assistant content, with the structured
tool_calls arriving only in the final chunk.

Buffer the text while a tool parser is active and let the existing end-of-stream
ChatDelta carry the parsed tool_calls; when no tool call is found, flush the
buffered content as a single content delta before the final chunk. Non-tool-parser
streaming is unchanged.

Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>

Author: pos-ei-don

backend/python/vllm/backend.py

@@ -597,12 +597,18 @@ async def _predict(self, request, context, streaming=False):
         # Stream the results
         generated_text = ""
         last_output = None
+        # When a tool parser is active for a tool-enabled request, the model's
+        # raw tool-call markup (e.g. <tool_call>...) must not be streamed as
+        # delta.content — clients would otherwise see the unparsed syntax. Buffer
+        # the text and emit structured tool_calls (or the cleaned content) once
+        # the full output is available and the parser has run, below.
+        has_tool_parser = bool(self.tool_parser_cls and request.Tools)
         try:
             async for request_output in outputs:
                 iteration_text = request_output.outputs[0].text
                 last_output = request_output
 
-                if streaming:
+                if streaming and not has_tool_parser:
                     # Remove text already sent as vllm concatenates the text from previous yields
                     delta_iteration_text = iteration_text.removeprefix(generated_text)
                     # Send the partial result
@@ -698,6 +704,14 @@ async def _predict(self, request, context, streaming=False):
         )
 
         if streaming:
+            # If content was buffered (has_tool_parser) but the parser found no
+            # tool call, the buffered text was never streamed — flush it as a
+            # content delta before the final structured chunk.
+            if has_tool_parser and not tool_calls_proto and content:
+                yield backend_pb2.Reply(
+                    message=bytes(content, encoding='utf-8'),
+                    chat_deltas=[backend_pb2.ChatDelta(content=content)],
+                )
             # Final chunk with structured data
             yield backend_pb2.Reply(
                 message=b"",