Document realtime WS partial rendering and concurrency (#3092)

Author: LauraGPT

docs/vllm_guide.md

@@ -617,7 +617,7 @@ Speaker diarization is disabled by default; add `--enable-spk` only when the `sp
 {"event": "stopped"}
 ```
 
-**Fields**: `sentences[]` = locked segments, `partial` = text being spoken (may change), `is_final` = true after STOP. When `--enable-spk` is enabled, `sentences[]` also includes `spk`.
+**Fields**: `sentences[]` = locked segments, `partial` = text being spoken (may change), `partial_start_ms` = where the current provisional `partial` begins, `is_final` = true after STOP. When `--enable-spk` is enabled, `sentences[]` also includes `spk`.
 
 **Sequence diagram**:
 ```
@@ -678,6 +678,17 @@ While the user is speaking, the streaming service periodically (default `decode_
 
 > Note: `serve_vllm.py`'s `/ws` (§5) has **no partial** and only returns at sentence end; use `serve_realtime_ws.py` for live preview.
 
+**Frontend rendering rule**
+Treat `partial` as a replaceable preview, not as text to append. A good UI keeps locked text and preview text separate:
+
+```js
+const committed = data.sentences.map((s) => s.text).join("");
+const preview = data.partial || "";
+render(committed + preview);
+```
+
+If `partial_start_ms` moves forward because `--partial-window-sec` is active, the preview only describes the current bounded decode window. Replace the preview area on each message; append only VAD-locked `sentences` or the final `is_final=true` result.
+
 **Principle: why each partial re-encodes the whole segment from the start**
 Fun-ASR-Nano's acoustic encoder (SenseVoice) is a **full-context, non-streaming** encoder — each frame's representation depends on the context of the entire segment. When the sentence continues and the audio grows, the context of the earlier frames changes, so **the previously computed encoding no longer holds**. It therefore cannot cache history and encode only the new frames the way a streaming / causal encoder would; it must run the whole "start → now" segment through the encoder again.
 
@@ -702,6 +713,7 @@ Because each refresh re-encodes from the sentence start, the longer a sentence,
 
 - **The single-process concurrency ceiling comes from event-loop serialization, not GPU compute.** Under high concurrency GPU utilization stays low and the encoder runs at ~86× real time; mistaking this for insufficient GPU and adding cards or tensor parallelism yields little (TP only splits the LLM, not the standalone encoder).
 - **The right way to scale (currently) = multiple independent processes on one card + CUDA MPS + nginx round-robin**: each process has its own GIL and CUDA context, sidestepping the single-loop serialization; MPS lets the processes truly share the GPU concurrently and fill the idle compute; nginx round-robins across the WebSocket backends. Beyond a single card's headroom, scale out horizontally (one instance per card + a load balancer).
+- **vLLM is not always more efficient for small real-time streams than several PyTorch processes.** vLLM helps most when requests can be batched or when LLM token decoding dominates. The current real-time WebSocket path submits many small, synchronous per-connection decode calls through one event loop, so it may reserve much more memory while still leaving GPU utilization modest. For a small number of continuous microphone streams, several lighter PyTorch processes can be easier to pack on one card. For vLLM, benchmark with the real traffic shape and start with lower `--gpu-memory-utilization` plus multiple service processes instead of assuming one vLLM process should carry every stream.
 - **Sustainable concurrency has no universal "supports N connections" number.** The ceiling is set not by the number of connections but by **how many are speaking at the same moment** — each speaking connection triggers a partial decode roughly once per second, all serialized on that single event loop. It mainly varies with: **① silence ratio** — in real turn-taking users spend most of the time listening, so far fewer are decoding simultaneously than are connected, whereas a continuous monologue keeps nearly every connection decoding; **② sentence length** — longer sentences make each partial encode more expensive (see 6.5's O(L²)), raising load at the same connection count. So the same "single L20 + multi-process + MPS" setup can sustain dozens of connections under turn-taking-like load but markedly fewer under long, pauseless speech. **Any "supports X connections" figure holds only for the traffic profile it was measured under** — benchmark with your own real traffic (sentence length, pauses, continuous or not) rather than treating someone else's number as your spec.
 
 ---

docs/vllm_guide_zh.md

@@ -621,7 +621,7 @@ CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/
 {"event": "stopped"}
 ```
 
-**字段**：`sentences[]` = 已锁定，`partial` = 正在说（会变），`is_final` = STOP 后为 true。启用 `--enable-spk` 后，`sentences[]` 会包含 `spk`。
+**字段**：`sentences[]` = 已锁定句子，`partial` = 当前正在说的临时文本（可能变化），`partial_start_ms` = 当前 `partial` 对应音频窗口的起点，`is_final` = STOP 后为 true。启用 `--enable-spk` 后，`sentences[]` 会包含 `spk`。
 
 **时序**：
 ```
@@ -682,6 +682,17 @@ asyncio.run(stream("audio.wav"))
 
 > 注：`serve_vllm.py`（§5）的 `/ws` **没有 partial**、只在句尾返回；要实时预览请用 `serve_realtime_ws.py`。
 
+**前端渲染原则**
+`partial` 只能当作“可替换预览”，不要把连续两次 `partial` 直接追加到最终文本里。推荐把已锁定文本和临时预览分开：
+
+```js
+const committed = data.sentences.map((s) => s.text).join("");
+const preview = data.partial || "";
+render(committed + preview);
+```
+
+如果启用了 `--partial-window-sec`，`partial_start_ms` 可能随着窗口向前滑动；这时 `partial` 只描述当前受限窗口内的临时识别结果。前端应每次替换 preview 区域，只把 VAD 已锁定的 `sentences` 或最终 `is_final=true` 结果追加到正式转写区。
+
 **原理：为什么每次 partial 都从句首整段重编**
 Fun-ASR-Nano 的声学编码器（SenseVoice）是**全上下文、非流式**编码器——每一帧的表示都依赖整段音频的前后文。当这句话又往下说了一截、音频变长时，先前那些帧的上下文随之改变，**之前算出的编码不再成立**，因此无法像流式 / 因果编码器那样"缓存历史、只算新增帧"，只能把"句首→当前"的整段重新过一遍编码器。
 
@@ -706,6 +717,7 @@ Fun-ASR-Nano 的声学编码器（SenseVoice）是**全上下文、非流式**
 
 - **单进程并发墙来自事件循环串行，不是 GPU 算力**。
 - **目前扩展可行方案 = 单卡多个独立进程 + CUDA MPS + nginx 轮询**：每个进程有独立的 GIL 与 CUDA 上下文，绕开单循环串行；MPS 让多进程真正并发共享 GPU、填满空闲算力；nginx 在多个 WebSocket 后端间轮询。超过单卡余量后，再横向加卡（每卡一实例 + 负载均衡）。
+- **小并发实时流不一定比多个 PyTorch 进程更适合 vLLM**。vLLM 的优势主要来自批处理和 LLM token decode 调度；而当前实时 WebSocket 路径会把多路小请求通过单事件循环同步送进解码，无法自然形成大 batch，所以可能显存占用更高但 GPU 利用率仍不高。对于少量连续麦克风流，多个轻量 PyTorch 进程有时更容易在一张卡上排布；使用 vLLM 时请按真实话务压测，先配合较低的 `--gpu-memory-utilization` 和多进程服务，而不是假设一个 vLLM 进程就应该承载所有连接。
 - **可持续并发没有通用的"支持 N 路"数字**：决定上限的不是在线连接数，而是**同一时刻有多少路正在"说话"**——每路只要在说，就每约 1 秒触发一次 partial 解码，全部串行在那个单事件循环上。它主要随两点变化：**① 停顿 / 静音占比**——真实一问一答中用户大半时间在听、不出声，同时解码的路数远少于在线连接数；连续独白则几乎每路都在持续解码，负载高得多。**② 句长**——句子越长，单次 partial 的编码越贵（见 6.5 的 O(L²)），同样路数下负载更高。因此同一套"单卡 L20 + 多进程 + MPS"，在接近真实 turn-taking 的负载下可稳定支撑数十路，而在长句、连续不停顿的负载下会显著更低。**任何"支持 X 路"的数字都只在它被测出来的那种话务下成立**——请按自己的真实话务（句长、停顿、是否连续说话）压测确定，别把别处测出的某个并发数当成自己的规格。
 
 ---

docs/vllm_guide_zh_v2.md

@@ -621,7 +621,7 @@ CUDA_VISIBLE_DEVICES=0 python examples/industrial_data_pretraining/fun_asr_nano/
 {"event": "stopped"}
 ```
 
-**字段**：`sentences[]` = 已锁定，`partial` = 正在说（会变），`is_final` = STOP 后为 true。启用 `--enable-spk` 后，`sentences[]` 会包含 `spk`。
+**字段**：`sentences[]` = 已锁定句子，`partial` = 当前正在说的临时文本（可能变化），`partial_start_ms` = 当前 `partial` 对应音频窗口的起点，`is_final` = STOP 后为 true。启用 `--enable-spk` 后，`sentences[]` 会包含 `spk`。
 
 **时序**：
 ```
@@ -682,6 +682,17 @@ asyncio.run(stream("audio.wav"))
 
 > 注：`serve_vllm.py`（§5）的 `/ws` **没有 partial**、只在句尾返回；要实时预览请用 `serve_realtime_ws.py`。
 
+**前端渲染原则**
+`partial` 只能当作“可替换预览”，不要把连续两次 `partial` 直接追加到最终文本里。推荐把已锁定文本和临时预览分开：
+
+```js
+const committed = data.sentences.map((s) => s.text).join("");
+const preview = data.partial || "";
+render(committed + preview);
+```
+
+如果启用了 `--partial-window-sec`，`partial_start_ms` 可能随着窗口向前滑动；这时 `partial` 只描述当前受限窗口内的临时识别结果。前端应每次替换 preview 区域，只把 VAD 已锁定的 `sentences` 或最终 `is_final=true` 结果追加到正式转写区。
+
 **原理：为什么每次 partial 都从句首整段重编**
 Fun-ASR-Nano 的声学编码器（SenseVoice）是**全上下文、非流式**编码器——每一帧的表示都依赖整段音频的前后文。当这句话又往下说了一截、音频变长时，先前那些帧的上下文随之改变，**之前算出的编码不再成立**，因此无法像流式 / 因果编码器那样"缓存历史、只算新增帧"，只能把"句首→当前"的整段重新过一遍编码器。
 
@@ -706,6 +717,7 @@ Fun-ASR-Nano 的声学编码器（SenseVoice）是**全上下文、非流式**
 
 - **单进程并发墙来自事件循环串行，不是 GPU 算力**。
 - **目前扩展可行方案 = 单卡多个独立进程 + CUDA MPS + nginx 轮询**：每个进程有独立的 GIL 与 CUDA 上下文，绕开单循环串行；MPS 让多进程真正并发共享 GPU、填满空闲算力；nginx 在多个 WebSocket 后端间轮询。超过单卡余量后，再横向加卡（每卡一实例 + 负载均衡）。
+- **小并发实时流不一定比多个 PyTorch 进程更适合 vLLM**。vLLM 的优势主要来自批处理和 LLM token decode 调度；而当前实时 WebSocket 路径会把多路小请求通过单事件循环同步送进解码，无法自然形成大 batch，所以可能显存占用更高但 GPU 利用率仍不高。对于少量连续麦克风流，多个轻量 PyTorch 进程有时更容易在一张卡上排布；使用 vLLM 时请按真实话务压测，先配合较低的 `--gpu-memory-utilization` 和多进程服务，而不是假设一个 vLLM 进程就应该承载所有连接。
 - **可持续并发没有通用的"支持 N 路"数字**：决定上限的不是在线连接数，而是**同一时刻有多少路正在"说话"**——每路只要在说，就每约 1 秒触发一次 partial 解码，全部串行在那个单事件循环上。它主要随两点变化：**① 停顿 / 静音占比**——真实一问一答中用户大半时间在听、不出声，同时解码的路数远少于在线连接数；连续独白则几乎每路都在持续解码，负载高得多。**② 句长**——句子越长，单次 partial 的编码越贵（见 6.5 的 O(L²)），同样路数下负载更高。因此同一套"单卡 L20 + 多进程 + MPS"，在接近真实 turn-taking 的负载下可稳定支撑数十路，而在长句、连续不停顿的负载下会显著更低。**任何"支持 X 路"的数字都只在它被测出来的那种话务下成立**——请按自己的真实话务（句长、停顿、是否连续说话）压测确定，别把别处测出的某个并发数当成自己的规格。
 
 ---