bug: Chat streaming error handling bugs

[tgasser-nv]

Did you check docs and existing issues?

• I have read all the NeMo-Guardrails docs
• I have updated the package to the latest version before submitting this issue
• (optional) I have used the develop branch
• I have searched the existing issues of NeMo-Guardrails

Python version (python --version)

Python 3.13.2

Operating system/version

Tahoe 26.3

NeMo-Guardrails version (if you must use a specific version and not the latest

0.21

Describe the bug

Two bugs discovered by Greptile in #1766 which were out-of-scope for that PR. Capturing here to be fixed later. Quoting from the comment:

Two P2 issues exist in the CLI streaming path worth addressing before or after merge.

Empty assistant message appended to history on streaming error (cli/chat.py lines 106-107): when a streaming error chunk is received and the loop breaks, bot_message_list is empty; the resulting {"role": "assistant", "content": ""} is unconditionally appended to history at line 181, corrupting subsequent turns with a blank assistant message.
Fragile error-chunk detection in CLI (cli/chat.py line 94): errors are identified via chunk.startswith('{"error"') — a raw string prefix check — while the server correctly uses ChunkError.model_validate_json(). Any whitespace or key-ordering change in the emitted JSON would silently pass the error through as normal content in the CLI.

Steps To Reproduce

1. Run nemoguardrails chat application with streaming enabled.
2. Insert streaming error chunk into App LLM response or with an "error" that doesn't match the expected format.

Expected Behavior

Handle error chunks gracefully and not corrupt conversation turns.

Actual Behavior

Corrupted conversation turns and potential missed error chunk responses

[ianliuy]

What's broken? (Symptoms)

When the nemoguardrails chat CLI runs with streaming enabled and an error occurs mid-stream, two things go wrong:

1. Empty assistant message corrupts history: A blank {"role": "assistant", "content": ""} is silently appended to the conversation history, corrupting subsequent turns.
2. Error chunks can slip through undetected: The CLI uses a raw string prefix check (chunk.startswith('{"error"')) that would silently miss error chunks with leading whitespace or different JSON key ordering.

Who is affected? (Impact)

Anyone using nemoguardrails chat with streaming enabled (--streaming flag or streaming config). The server path (nemoguardrails server) is not affected — it already uses proper Pydantic-based ChunkError.model_validate_json() parsing.

When does it trigger? (Conditions)

• Bug 1: Every time a streaming error occurs — the empty message is unconditionally appended.
• Bug 2: When the LLM or rail produces an error chunk whose JSON serialization doesn't start with the exact string {"error" (e.g., leading whitespace, pretty-printed JSON, or reordered keys).

Where is the bug? (Code Location)

Both bugs are in nemoguardrails/cli/chat.py, inside _run_chat_v1_0():

• Bug 1 — Lines 84→98→173: bot_message_list = [] is never populated when break fires on error (line 93), so bot_message_text = "", and history.append(bot_message) at line 173 is unconditional.
• Bug 2 — Line 86: if isinstance(chunk, str) and chunk.startswith('{"error"'):

Why does it happen? (Root Cause)

Bug 1: The streaming loop (PR #1538) builds bot_message after the async for loop without tracking whether an error occurred. The break on error exits the loop, but bot_message_list remains [] → "".join([]) = "" → an empty assistant message is created and unconditionally appended to history.

Bug 2: PR #1766 added chunk.startswith('{"error"') as a quick heuristic for error detection in the CLI, explicitly noted as a follow-up to fix properly (this issue). The server already has the robust approach via ChunkError.model_validate_json() in server/api.py:430-431.

How did we fix it? (Solution)

Bug 2: Replace startswith('{"error"') with json.loads() + check for "error" key in the parsed dict. This mirrors the existing pattern in iorails.py:360-364 and handles any valid JSON error format regardless of whitespace or key ordering. No new imports needed — json is already imported.

Bug 1: Add if bot_message.get("content"): guard before history.append(bot_message) at line 173. When an error breaks the streaming loop, bot_message_text is "" which is falsy, so the empty message is not appended to history.

Alternative considered: Moving ChunkError to a shared module (streaming.py) and importing it in both CLI and server (T2 root-cause approach). Deferred to a future refactoring PR to keep this fix minimal and focused.

How do we know it works? (Testing)

Will be filled after implementation.

---

I'm working on a fix.

[ianliuy]

Implemented as described above. Key refinement from code review: tightened the error detection to require error_data["error"] to be a dict (not just checking for the "error" key), preventing false positives on LLM-generated JSON like {"error": "rate limited"}. PR incoming shortly.

[ianliuy]

PR: #1792