Skip to content

fix(chat): stop silent output-token truncation from stranding agent writes#151

Merged
ABB65 merged 1 commit into
mainfrom
fix/chat-output-token-truncation
Jul 14, 2026
Merged

fix(chat): stop silent output-token truncation from stranding agent writes#151
ABB65 merged 1 commit into
mainfrom
fix/chat-output-token-truncation

Conversation

@ABB65

@ABB65 ABB65 commented Jul 14, 2026

Copy link
Copy Markdown
Member

Problem

Two staging conversations (Lanista/collabers project) froze on a "…saving now:" preamble whose save never ran, with no error shown. Reproduced from the staging DB: the stuck assistant turns had content_blocks = [text] only and token_count_output = 4096 exactly.

Root cause: the chat tool loop called the model with a hardcoded max_tokens: 4096 (conversation-engine.ts). A large write tool call — a full ui-strings dictionary, or several articles across locales in one turn — exceeded 4096 output tokens, so generation was cut off mid tool_use. No tool_use_end fires, the incomplete call never lands in currentToolCalls, and the loop's generic "not tool_use → end turn" check closed the turn silently. The user saw a frozen preamble and nothing saved.

Fix — three complementary changes

1. Raise the output-token ceiling (catalog-driven). maxOutputTokens is now a ChatModelEntry field (16K for every chat model, safely under each model's documented limit) plus maxOutputTokensFor() with an 8K fallback for Conversation-API / legacy model IDs. It's a cap, not a target — billed on actual output — so this is cost-neutral for normal turns and only prevents realistic bulk writes from truncating. Threaded through ConversationConfig and both callers (Studio chat + ee Conversation API).

2. Handle stop_reason: max_tokens explicitly. Instead of falling through to the silent end-of-turn: keep the partial text for the transcript, emit a typed output_truncated error, and close the turn. The client localizes it via chat.output_truncated (falls back to the generic send error for other error events).

3. rules.batch_writes system-prompt rule. Steers the agent to split large writes across smaller save_content calls (a few collection entries per call) so it stops emitting oversized single tool calls. Reviewed against the write path for safety — the rule is scoped to the collection-entry axis and explicitly preserves id stability (same entry id across locales, existing id on updates), so batching cannot cause i18n id drift. Collection sibling/field merge, dictionary merge, document merge, and relation integrity are all preserved/enforced by existing code (save-content.ts, MCP planContentSave, and the relation pre-flight, which returns a self-correcting error rather than silently breaking).

Testing

  • New regression test: a max_tokens stop yields an output_truncated error and still closes the turn with the partial text preserved.
  • pnpm test (unit + integration + nuxt): 1049/1049 pass.
  • pnpm typecheck, lint: clean.

Follow-up

Merge-path change — worth a real save on staging after deploy (e.g. the same ui-strings capitalization edit) to confirm the large-write path end-to-end; mocked git tests don't exercise the provider truncation behavior.

…rites

Two staging conversations froze on a "…saving now:" preamble whose save
never ran. Root cause: the tool loop called the model with a hardcoded
`max_tokens: 4096`. A large write tool call (a full ui-strings dictionary,
or several articles across locales in one turn) exceeded 4096 output
tokens, so generation was cut off mid tool_use — no `tool_use_end` fired,
the incomplete call never landed in `currentToolCalls`, and the loop's
generic "not tool_use → end turn" check closed the turn silently. The
user saw a frozen preamble with no error and nothing saved.

Three complementary fixes:

- Raise the output-token ceiling via the model catalog. `maxOutputTokens`
  is now a `ChatModelEntry` field (16K for every chat model, safely under
  each model's documented limit) plus `maxOutputTokensFor()` with an 8K
  fallback for Conversation-API / legacy model IDs. It is a cap, not a
  target — billed on actual output — so this is cost-neutral for normal
  turns and only prevents realistic bulk writes from truncating. Threaded
  through `ConversationConfig` and both callers (Studio chat + ee
  Conversation API).

- Handle `stop_reason: max_tokens` explicitly in the loop instead of
  falling through to the silent end-of-turn: keep the partial text for the
  transcript, emit a typed `output_truncated` error, and close the turn.
  The client localizes it via `chat.output_truncated` (falls back to the
  generic send error for other error events).

- Add the `rules.batch_writes` system-prompt rule steering the agent to
  split large writes across smaller save_content calls (a few entries per
  call, one locale per call) so it stops emitting oversized single tool
  calls in the first place.

Adds a regression test asserting a `max_tokens` stop yields an
`output_truncated` error and still closes the turn with the partial text.
@ABB65 ABB65 merged commit 0619a7e into main Jul 14, 2026
2 checks passed
@ABB65 ABB65 deleted the fix/chat-output-token-truncation branch July 14, 2026 13:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant