examples/vllm-bench: add long-text scenario (8 paragraphs, 1500 tokens)

The long-text scenario shows the buffering vs streaming difference most
dramatically: with the buffer-all path, the client receives nothing for
20+ seconds and then the entire 1500-token response at once. With native
streaming, the first token arrives in tens of milliseconds and the
response flows progressively.

Author: Philipp Wacker

examples/vllm-bench/README.md

@@ -15,26 +15,36 @@ non-streaming — the client received nothing until the model finished.
 With native parser-side streaming (`parser.extract_tool_calls_streaming`,
 implemented by every concrete vLLM 0.23+ tool parser), each delta can be
 classified per-token: emit as content, emit as a structured tool_call, or
-suppress. This benchmark shows the difference.
+suppress.
 
-## Two scenarios
+## Three scenarios
 
 | Scenario | Request | Expected outcome |
 |---|---|---|
-| `tool_call` | "What is the weather in Paris? Please use the tool." | Model calls `get_weather`. `delta.tool_calls` chunks; no content leak. |
-| `plain_text` | "Explain in 3 short sentences what a hash table is. Do NOT call any tool." | Model writes prose. With the streaming parser, content streams progressively; without it, the entire response arrives in one chunk. |
+| `tool_call`         | "What is the weather in Paris? Please use the tool." | Model calls `get_weather`. `delta.tool_calls` chunks; no content leak. |
+| `plain_text_short`  | "Explain in 3 short sentences what a hash table is. Do NOT call any tool." | Model writes ~3 sentences. |
+| `plain_text_long`   | "Write a thorough 8-paragraph explanation of how Python's GIL works…" | Model writes ~1500 tokens of prose. |
+
+The **long scenario** is where the streaming/buffering difference is most
+dramatic: with the buffer-all path, the client sees nothing for 20+ seconds
+and then everything at once; with native streaming, the first token arrives
+in <100ms and the response flows progressively.
 
 ## What the script reports
 
 For each scenario, across N runs:
 
 - `ttf_content_s` — time until the first `delta.content` chunk
 - `ttf_tool_s` — time until the first `delta.tool_calls` chunk
-- `n_content_chunks` — total number of distinct content deltas (1 = bundled, >1 = streamed)
+- `n_content_chunks` — total content deltas (1 = bundled, >>1 = streamed)
 - `n_tool_chunks` — total tool_call deltas
 - `total_s` — total wall-clock until `[DONE]`
 - `finish_reason` — `tool_calls` / `stop` / `length`
 
+The big tell is **`n_content_chunks` vs `total_s` ratio**:
+- Buffer-all: `n_content_chunks` ≈ 1, `ttf_content_s` ≈ `total_s` (one chunk at end)
+- Streaming: `n_content_chunks` ≈ token count, `ttf_content_s` ≈ first-token latency
+
 ## Usage
 
 ```bash

examples/vllm-bench/ttft_streaming_tool_parser.py

@@ -2,15 +2,14 @@
 """
 TTFT benchmark for the vLLM backend's streaming + tool-parser path.
 
-Measures time-to-first-token (TTFT) for two scenarios against a running
-LocalAI instance with a vLLM-backed chat model:
+Three scenarios:
+  1. tool_call        — request mentions a tool; model is expected to call it
+  2. plain_text_short — request offers a tool but explicitly asks for ~3 sentences
+  3. plain_text_long  — same as above but asks for ~8 paragraphs (1500 tokens)
 
-  1. tool_call  — request mentions a tool; model is expected to call it
-  2. plain_text — request offers a tool but explicitly asks for prose
-
-Useful to validate the difference between:
-  - the buffer-all path  (#10346):       plain_text TTFT ≈ total response time
-  - the native-streaming path (this PR): plain_text TTFT ≈ true first-token time
+The long scenario shows the dramatic difference between buffering and
+streaming most clearly: with buffer-all, the client sees nothing for
+20+ seconds; with native streaming, the first token arrives in <100 ms.
 
 Usage:
   python ttft_streaming_tool_parser.py \\
@@ -45,12 +44,21 @@
         "max_tokens": 80,
     },
     {
-        "label": "plain_text",
+        "label": "plain_text_short",
         "messages": [{"role": "user",
                       "content": "Explain in 3 short sentences what a hash table is. "
                                  "Do NOT call any tool."}],
         "max_tokens": 200,
     },
+    {
+        "label": "plain_text_long",
+        "messages": [{"role": "user",
+                      "content": "Write a thorough 8-paragraph explanation of how "
+                                 "Python's GIL works, including history, current "
+                                 "state, no-GIL build, and alternatives. Be "
+                                 "detailed. Do NOT call any tool."}],
+        "max_tokens": 1500,
+    },
 ]
 
 
@@ -125,7 +133,7 @@ def main():
                    help="LocalAI base URL (default: %(default)s)")
     p.add_argument("--model", default="coder", help="Model name (default: %(default)s)")
     p.add_argument("--runs", type=int, default=3, help="Repetitions per scenario (default: %(default)s)")
-    p.add_argument("--timeout", type=int, default=120, help="Per-request timeout in seconds")
+    p.add_argument("--timeout", type=int, default=180, help="Per-request timeout in seconds")
     p.add_argument("--label", default="run",
                    help="Tag for the JSON output file (default: %(default)s)")
     args = p.parse_args()