Streaming adapter can accumulate failed frames past context and image limits

[xiaowangzhixiao]

Problem

The WebInfer streaming adapter appends each incoming frame to current_chunk before calling the main model. If that call fails (for example because the prompt exceeds the model context window), the appended turn remains in session state. Retrying then appends the frame again, so the prompt keeps growing until the backend's multimodal item limit is also exceeded.

Observed failure sequence with an 8,192-token backend context and a 32-image vLLM limit:

1. The prompt first fails with Input length (...) exceeds ... 8192.
2. Each retry adds more images to the same session.
3. The backend eventually returns At most 32 image(s) may be provided in one prompt.
4. The adapter exposes the backend 400 as a 502 response.

The current CHUNK setting bounds summary cadence, but async-summary carry-over can still leave enough unsummarized turns for the main-model prompt to exceed a backend-specific context or image limit.

Expected behavior

• Deployments should be able to configure a recent-turn/recent-image window for main-model prompts without discarding full session history used by summaries and long-term memory.
• A failed main-model call should not permanently append the failed turn, so a retry starts from the same prior state.
• Existing deployments should retain their current behavior unless the optional limits are enabled.

Proposed fix

Add opt-in MAIN_MAX_PROMPT_TURNS and MAIN_MAX_PROMPT_IMAGES settings (both defaulting to 0, unlimited), and snapshot/restore append-only turn state around main-model calls.

I have a tested implementation ready and will link the pull request here.