fix: don't drop fork messages containing U+2028/U+2029/U+0085#1092
Open
Otis0408 wants to merge 1 commit into
Open
fix: don't drop fork messages containing U+2028/U+2029/U+0085#1092Otis0408 wants to merge 1 commit into
Otis0408 wants to merge 1 commit into
Conversation
_parse_fork_transcript split the raw transcript with str.splitlines(), whose line-boundary set includes U+2028 (LINE SEPARATOR), U+2029 (PARAGRAPH SEPARATOR) and U+0085 (NEL). Those are not JSONL line breaks: they are valid *unescaped* inside a JSON string and are emitted raw by the CLI's Node JSON.stringify, so a message whose content contains one lands whole and valid inside a single .jsonl line. splitlines() fragmented such a line into pieces that each fail json.loads, so fork_session silently dropped that message (and orphaned any child pointing at it), even though the read path (get_session_messages / _parse_transcript_entries) reads it fine because it splits on "\n" only. Split on "\n" to match the read path. Adds a unit test over all three separators and an end-to-end test asserting fork preserves a message containing a raw U+2028.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fork_session()can silently drop a message whose content contains a raw U+2028 (LINE SEPARATOR), U+2029 (PARAGRAPH SEPARATOR), or U+0085 (NEL)._parse_fork_transcriptsplits the raw transcript withstr.splitlines():str.splitlines()breaks on U+2028/U+2029/U+0085 in addition to\n. But those characters are not JSONL line breaks — they are valid unescaped inside a JSON string (JSON only requires escaping U+0000–U+001F), and the CLI's NodeJSON.stringifyemits them raw. So a user/assistant message whose content contains one lands whole and valid inside a single.jsonlline.splitlines()fragments that line into pieces that each failjson.loads, so the message is dropped from the fork (and any child pointing at itsparentUuidis orphaned).The read path reads the same file fine, because it splits on
"\n"only:So
get_session_messages(sid)returns the message, butfork_session(sid)produces a fork that's missing it — an internal-contract divergence and real data loss. If the affected message is the session's only message, fork raisesValueError("... has no messages to fork")for a session that reads fine.Fix
Split on
"\n"to match the read path.line.strip()already normalizes trailing\r, so this is equivalent to the read path's tolerant behavior.fork_session_via_storeis unaffected (it iterates already-parsed store objects rather than re-splitting text).Tests
test_parse_fork_transcript_keeps_unicode_line_separators— unit test parametrized over U+2028 / U+2029 / U+0085, asserting the entry survives.test_fork_preserves_message_with_line_separator— end-to-end: a session whose assistant message contains a raw U+2028 forks with the same message count.Verified both fail on
main(message dropped) and pass with the fix. Full suite: 1060 passed, 5 skipped;ruffandmypyclean.