Bug: append mode corrupts `original_text`, breaking conversation-aware chunking on subsequent retains

[qxxaa]

Bug Description

When update_mode=append is used with conversation-format content (flat JSON arrays of {role, content, timestamp} dicts), the original_text stored in the documents table becomes invalid JSON after the second retain cycle. This happens because combined_content is built by newline-joining multiple content items:

combined_content = "\n".join([c.get("content", "") for c in contents_dicts])

When contents_dicts contains the previously stored text (a valid JSON array) and the new delta (another valid JSON array), the join produces two JSON arrays separated by \n, which is not valid JSON.

On subsequent append cycles, this corrupted original_text is fetched and passed to chunk_text(). The JSON parse fails, _chunk_jsonl() also rejects it (each line is an array, not a dict), and the text falls through to RecursiveCharacterTextSplitter. This splits the raw JSON string on sentence boundaries (. ) inside message content, producing chunks that begin mid-sentence with no speaker attribution. The extraction LLM then has no basis for determining who said what.

Affected file: hindsight-api-slim/hindsight_api/engine/retain/orchestrator.py

Steps to Reproduce

1. Retain a conversation-format document with update_mode=append:

{
  "document_id": "session-abc",
  "update_mode": "append",
  "items": [{"content": "[{\"role\":\"user\",\"content\":\"hello\"},{\"role\":\"assistant\",\"content\":\"hi\"}]"}]
}

2. Send a second retain with new turns to the same document_id:

{
  "document_id": "session-abc",
  "update_mode": "append",
  "items": [{"content": "[{\"role\":\"user\",\"content\":\"how are you\"},{\"role\":\"assistant\",\"content\":\"fine\"}]"}]
}

3. Send a third retain with any new content to the same document_id.

4. Inspect the chunks produced for the document.

Minimal script demonstrating the storage corruption:

import json

turn1 = json.dumps([
    {"role": "user", "content": "User: hello", "timestamp": "2026-06-23T22:00:00+00:00"},
    {"role": "assistant", "content": "Assistant: hi there", "timestamp": "2026-06-23T22:00:01+00:00"},
])

turn2 = json.dumps([
    {"role": "user", "content": "User: how are you", "timestamp": "2026-06-23T22:01:00+00:00"},
    {"role": "assistant", "content": "Assistant: doing well", "timestamp": "2026-06-23T22:01:01+00:00"},
])

# After 2nd retain, original_text in the DB becomes:
original_text = "\n".join([turn1, turn2])

# 3rd retain fetches this and tries to chunk it:
try:
    parsed = json.loads(original_text)
    print("JSON parse succeeded")
except json.JSONDecodeError as e:
    print(f"JSON parse failed: {e}")
    print("Falls through to RecursiveCharacterTextSplitter -> mid-sentence splits")

Output:

JSON parse failed: Extra data: line 2 column 1 (char 175)
Falls through to RecursiveCharacterTextSplitter -> mid-sentence splits

Expected Behavior

All chunks produced from conversation-format append documents should go through _chunk_conversation(), splitting at turn boundaries. Every chunk should begin with a complete {"role": ...} object so the extraction LLM always has speaker attribution.

Actual Behavior

After the second append cycle, original_text becomes newline-joined JSON arrays (not valid JSON). On subsequent retains, chunk_text() fails to parse it, _chunk_jsonl() rejects it (lines are arrays not dicts), and it falls through to sentence-boundary text splitting.

Corruption mechanism

On the second retain, the server fetches original_text from step 1 and prepends it as a new content item:

contents_dicts = [
    {"content": '[{"role":"user","content":"hello"},{"role":"assistant","content":"hi"}]'},
    {"content": '[{"role":"user","content":"how are you"},{"role":"assistant","content":"fine"}]'}
]

Then builds combined_content:

combined_content = "\n".join([c.get("content", "") for c in contents_dicts])

Producing:

[{"role":"user","content":"hello"},{"role":"assistant","content":"hi"}]
[{"role":"user","content":"how are you"},{"role":"assistant","content":"fine"}]

This is stored as original_text. Not valid JSON - it's two JSON arrays separated by \n.

Chunking failure

On the third retain onward, the server fetches the corrupted original_text and passes it to chunk_text():

parsed = json.loads(text)  # Fails - Extra data: line 2 column 1

Falls through to RecursiveCharacterTextSplitter with separators ["\n\n", "\n", ". ", "! ", "? ", "; ", ", ", " ", ""]. The raw JSON string gets split on . boundaries inside message content, producing chunks like:

. The configuration was working fine until...
. In that sense it has a complet...
. I've been trying to debug this...

No role prefix, no speaker attribution.

Observable symptoms

In the Hindsight dashboard chunks view, a document retained via append mode shows an alternating pattern:

Chunk	Starts with	Source
#0	[{"role": "user", "content":	Recent delta (valid JSON, chunked correctly)
#1	. The configuration was working fine until...	Corrupted original_text (text-split)
#2	[{"role": "user", "content":	Recent delta
#3	. In that sense it has a complet...	Corrupted original_text

Chunks from the corrupted path lack speaker attribution entirely.

Suggested Fix

When building combined_content for append-mode documents, the content items need to be merged in a format-aware way rather than blindly newline-joined. For conversation arrays specifically:

Option A: Merge JSON arrays before storing. Parse each content item as JSON; if both are arrays of dicts, concatenate the lists and re-serialize as a single flat array. original_text stays valid JSON and _chunk_conversation() fires on every subsequent fetch.

Option B: Use JSONL. Store each content item as one line (already the \n join behaviour). Teach chunk_text() to detect JSONL-formatted conversation arrays (multiple lines, each a JSON array of role/content dicts) and route them to _chunk_conversation() after flattening.

Option A is simpler and preserves the existing _chunk_conversation path without modification.

Version

commit 6a10b62

LLM Provider

OpenAI

[qxxaa]

Pushed a fix in #2412. Approach: detect when all contents_dicts items are JSON arrays of dicts in the append-mode block, merge them into a single flat array before combined_content is built. Non-conversation content is unaffected.

Tested on a live deployment - chunks now consistently start with [{"role": and route through _chunk_conversation across multiple append cycles.