extension/llm/server: token-ID prompt segments for tool-use resume (V2b.1.5)

Warm resume (V2b.1) misses on agent loops: an assistant turn re-rendered from
its parsed tool call almost never re-tokenizes to the tokens the model actually
generated, so the resident state isn't an exact prefix and the worker resets.
On BFCL multi_turn the warm-resume hit rate was 0%.

Fix: carry the exact tokens instead of re-deriving them from text. The worker
returns generated_token_ids on `done` and accepts a `prompt_segments` form of
the prompt -- an ordered list of {"text"} chunks to tokenize and {"ids"} runs of
literal token ids (mutually exclusive with the plain `prompt` string); the
WorkerClient/SessionRuntime transport for that form was introduced with the
SessionRuntime boundary, and this commit makes the worker assemble and emit it.
The adapter-specific transcript glue lives in a new module, openai_transcript.py
(OpenAITranscriptState): it stores one record per assistant turn ({fingerprint,
ids}) and, on the next request, rebuilds the prompt as segments -- each prior
assistant turn is replaced with a unique sentinel, the conversation is rendered
once, and the rendered text is split on the sentinels with the stored ids spliced
back in. Tool results stay text (they re-tokenize deterministically). This logic
is the OpenAI adapter's concern, not the runtime's: SessionRuntime only sees a
PromptInput (text or segments).

Splicing is guarded so stale ids are never injected: a turn is substituted only
when the incoming assistant message fingerprint-matches the response we returned
(an edited or branched history, or a session reused for another conversation ->
text fallback; splicing stops at the first divergence and the now-stale tail is
pruned, and a regenerated turn is recorded at its position so it replaces the
stale record instead of shadowing later hits), and only when its ids faithfully
decode to what the client saw -- a stop-string trim kept post-stop tokens
resident but dropped them from the output, so the worker omits the ids and the
turn is re-rendered as text. Sentinel collisions / dropped sentinels also fall
back to text, and the worker's exact-token prefix check backstops the rest.

The context-window preflight counts what the worker actually assembles: for a
segment prompt it sums the literal {ids} run lengths and the tokenized {text}
chunks (not the rendered string), so a near-limit request agrees with the worker
rather than false-rejecting or failing mid-decode.

On BFCL multi_turn (per-conversation sessions) this moves the reuse fraction
from 0% to ~50% (exact_prefix hits where there were none); the single-turn AST
suite is unchanged (no prior assistant turn -> plain text prompt).

Review order: worker_loop.h (segment assembly + faithful generated_token_ids);
then the control plane (the new openai_transcript.py store + fingerprint-guarded
sentinel rendering, and the serving_chat wiring that builds the segments and
counts them for the context preflight); then tests and docs.

Part of #20001

ghstack-source-id: 8f15d6a
ghstack-comment-id: 4661784137
Pull-Request: #20161

Author: mergennachin

examples/models/qwen3_5_moe/README.md

@@ -243,6 +243,16 @@ Each `done` event reports
 (`new`/`exact_prefix`/`dirty`/`mismatch`/`equal`) for measuring the hit rate.
 `--no-warm-resume` forces a full prefill every request (for A/B comparison).
 
+**Tool-call turns (token-ID continuation):** an assistant turn re-rendered from
+its parsed tool call rarely re-tokenizes to the tokens the model actually
+generated, so plain warm resume misses on agent loops. The server stores the
+exact generated token ids per session and, on the next turn, sends the prompt as
+segments (`{"text"}` / `{"ids"}`) that splice those ids back in for prior
+assistant turns instead of re-rendering them — so the resident state stays an
+exact token prefix and resume hits. Tool *results* remain text (re-tokenized
+deterministically). The worker's exact-token check still backstops everything, so
+a mismatch just falls back to a full prefill.
+
 This is **isolation + warm resume, not concurrency**: execution is still
 synchronous (one in-flight request; `--num-runners > 1` is rejected since more
 workers would duplicate the weights). Fair interleaving across in-flight requests

extension/llm/server/cpp/CMakeLists.txt

@@ -95,3 +95,14 @@ target_include_directories(
   test_worker_prefill_plan PUBLIC ${_common_include_directories}
 )
 add_test(NAME worker_prefill_plan COMMAND test_worker_prefill_plan)
+
+# Worker-loop harness (worker_handle_request + WorkerSessions) driven by a
+# scriptable fake LLMSession/Tokenizer/LLMEngine -- no model/GPU. It includes
+# the full worker_loop.h, so it needs the JSON include + the runtime/tokenizer
+# libs.
+add_executable(test_worker_loop test_worker_loop.cpp)
+target_include_directories(
+  test_worker_loop PUBLIC ${_common_include_directories} ${_json_include}
+)
+target_link_libraries(test_worker_loop PUBLIC ${link_libraries})
+add_test(NAME worker_loop COMMAND test_worker_loop)