[UPDATE] Update

[ghstack-poisoned]

Author: mergennachin

extension/llm/server/README.md

@@ -25,9 +25,10 @@ Hugging Face chat templates (`--hf-tokenizer`), `temperature` / `max_tokens` /
 `max_completion_tokens` / `stop`, Hermes tool calling by default
 (`<tool_call>...</tool_call>` JSON, complete calls only; model-specific launchers
 may select the Qwen XML format) with `tool_choice="none"`,
-structured API errors, and best-effort cancellation. V1 serving is single-slot
-(one worker, one session) with no prefix cache; KV prefix reuse, if it returns,
-lives inside the worker/session, not the control plane. Unsupported params (including `top_p`,
+structured API errors, and best-effort cancellation. One worker process with
+serialized execution; it hosts many isolated sessions on one weight load (warm
+append-only resume across turns). KV/prefix state lives inside the
+worker/session, not the control plane. Unsupported params (including `top_p`,
 `seed`, `n>1`, `reasoning_effort`, penalties, `logit_bias`, `response_format`,
 `logprobs`, and `tool_choice="required"`) are rejected with a structured 400
 rather than silently ignored. See `python/README.md` to run it and

extension/llm/server/python/README.md

@@ -69,7 +69,7 @@ Key flags:
 | `--allow-chatml-fallback` | opt into approximate ChatML when no HF tokenizer |
 | `--no-think` | default `enable_thinking=False` (e.g. Qwen3) |
 | `--max-context N` | reject over-long prompts with 400 instead of failing mid-gen |
-| `--num-runners N` | V1 supports **1 only** (single-slot: one worker serves one session; concurrent requests queue) |
+| `--num-runners N` | Worker processes — **1 only** (one worker hosts many isolated sessions on one weight load; more would duplicate weights) |
 | `--worker-bin PATH` | path to the `text_llm_worker` binary (default: `cmake-out/extension/llm/server/cpp/text_llm_worker`) |
 
 ## Use from an agent harness
@@ -101,16 +101,19 @@ pytest tests/
 OPENAI_BASE_URL=http://127.0.0.1:8000/v1 pytest ../conformance/test_openai_contract.py
 ```
 
-`tests/` builds a `RunnerPool` over a single `FakeRunner` worker handle, so the
+`tests/` builds a `SessionRuntime` over a single `FakeRunner` worker, so the
 real server/protocol/streaming code is tested over HTTP without a `.pte`. The
 worker JSONL protocol is covered separately by `tests/test_worker_client.py`.
 
 ## Architecture
 
-Control plane (this dir, Python): server, OpenAI protocol, chat templating,
-streaming bridge, tool parsing — no CUDA, no model, no pybind. Data plane (C++):
-a worker process (`text_llm_worker`) owns one model session and does all token
-stepping and KV mutation; it speaks one JSON object per line on stdin/stdout.
+Control plane (this dir, Python): an OpenAI adapter (`serving_chat`) over a
+stateful `SessionRuntime` over one `WorkerClient` — server, protocol, chat
+templating, streaming bridge, tool parsing — no CUDA, no model, no pybind. Data
+plane (C++): a worker process (`text_llm_worker`) that owns all model state
+(many isolated sessions on one weight load, warm-resume prefix logic) and does
+all token stepping and KV mutation; it speaks one JSON object per line on
+stdin/stdout.
 
 JSONL protocol (stdout carries protocol JSON only; logs go to stderr):
 
@@ -132,9 +135,9 @@ does blocking pipe I/O on its executor thread.
 | `server.py` | FastAPI app, routes, CLI entrypoint, worker spawn |
 | `protocol.py` | OpenAI request/response schemas |
 | `chat_template.py` | messages (+tools) → prompt string |
-| `worker_client.py` | spawn a worker process + drive it over JSONL |
-| `runner_pool.py` | worker pool (one in-flight request per worker) + async streaming bridge |
-| `serving_chat.py` | `/v1/chat/completions` (streaming + non-streaming, stop, tools) |
+| `worker_client.py` | spawn a worker process + drive it over JSONL (raw transport) |
+| `session_runtime.py` | stateful runtime over one worker: open/generate/reset/close + streaming bridge |
+| `serving_chat.py` | `/v1/chat/completions` OpenAI adapter (streaming + non-streaming, stop, tools) |
 | `tool_parsers/` | Hermes/Qwen `<tool_call>` parser only |
 | `cpp/text_llm_worker.cpp` | the generic C++ worker binary |
 
@@ -151,11 +154,11 @@ imports an example. Backend specifics (CUDA/AOTI, Metal) stay inside the worker.
 ## Scope & caveats
 
 Deliberately narrow (reliability-first): Hermes/Qwen tool calling only;
-unsupported sampling params are rejected, not ignored. V1 is **single-slot**: one
-worker hosts one session, so `--num-runners` accepts 1 and concurrent requests
-queue. Serving capacity is worker capacity, chosen by the launcher (each worker
-is its own process with its own weights, so N workers cost N × the weight memory)
-— an operator decision, not something the pool infers.
+unsupported sampling params are rejected, not ignored. **One worker process,
+serialized execution** (one in-flight request; concurrent requests queue).
+Session capacity is determined by the worker/engine — a single worker hosts many
+isolated sessions on one weight load — so `--num-runners` accepts 1; extra worker
+processes would each carry their own copy of the weights.
 
 Cancellation is best-effort: a worker request runs to completion and is not
 interruptible mid-generation in V1, so `runner.stop()` means "the control plane

extension/llm/server/python/runner_pool.py

@@ -14,9 +14,10 @@
 no runtime pybind. Model execution lives in a separate C++ worker process
 (``text_llm_worker``) driven over JSONL via WorkerClient.
 
-V1 is single-slot: one worker hosts one session, so concurrent requests queue.
-There is no prefix cache in V1 serving; caching, if it returns, lives inside the
-worker/session, not the control plane.
+One worker process, serialized execution (one in-flight request; concurrent
+requests queue). Session capacity is set by the worker/engine -- a single worker
+hosts many isolated sessions on one weight load; extra worker processes would
+duplicate the weights, so `--num-runners` accepts 1.
 
 Example:
     python -m executorch.extension.llm.server.python.server \\
@@ -35,8 +36,8 @@
 from .chat_template import ChatTemplate
 from .errors import APIError
 from .protocol import ChatCompletionRequest, ModelCard, ModelList
-from .runner_pool import RunnerPool
 from .serving_chat import ServingChat
+from .session_runtime import SessionRuntime
 from .tool_parsers import HermesDetector
 from .worker_client import spawn_worker
 
@@ -143,7 +144,8 @@ def main() -> None:
         "--num-runners",
         type=int,
         default=1,
-        help="V1 supports 1 only (single-slot: one worker serves one session).",
+        help="Worker processes. 1 only: one worker hosts many isolated sessions "
+        "on a single weight load; more workers would duplicate the weights.",
     )
     p.add_argument(
         "--worker-bin",
@@ -158,7 +160,8 @@ def main() -> None:
 
     if args.num_runners != 1:
         p.error(
-            "V1 is single-slot: one worker serves one session; concurrent requests queue."
+            "Only 1 worker process is supported (it hosts many isolated sessions "
+            "on one weight load); more workers would duplicate the weights."
         )
 
     default_template_kwargs = {"enable_thinking": False} if args.no_think else None
@@ -168,10 +171,10 @@ def main() -> None:
         default_template_kwargs=default_template_kwargs,
         allow_fallback=args.allow_chatml_fallback,
     )
-    worker = _spawn(args)  # one worker == one session (single-slot V1)
-    pool = RunnerPool([worker])
+    worker = _spawn(args)  # one worker hosting many isolated sessions
+    runtime = SessionRuntime(worker)
     serving = ServingChat(
-        pool,
+        runtime,
         template,
         args.model_id,
         max_context=args.max_context,
@@ -184,7 +187,7 @@ def main() -> None:
 
     @app.on_event("shutdown")
     def _stop_worker():
-        pool.close()
+        runtime.close_worker()
 
     import uvicorn  # imported here so build_app() is usable without the ASGI server