fix: skip corrupt/truncated lines in import_session_to_store#1093
Open
Otis0408 wants to merge 1 commit into
Open
fix: skip corrupt/truncated lines in import_session_to_store#1093Otis0408 wants to merge 1 commit into
Otis0408 wants to merge 1 commit into
Conversation
_append_jsonl_file_in_batches parsed each line with a bare json.loads(line), so a partially-written final line — the normal outcome when the CLI is killed mid-append to a .jsonl — made import_session_to_store raise json.JSONDecodeError. Every read path in the package tolerates this: _parse_transcript_entries (used by get_session_messages) wraps json.loads in try/except and skips corrupt lines, so the same file reads fine. Because entries flush in batches, a large session also left the store partially populated before the crash. Skip corrupt/truncated lines to match the read path. import_session_to_store's documented Raises (ValueError, FileNotFoundError) already excludes JSONDecodeError, so this aligns behavior with the contract. Adds a regression test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
import_session_to_storeraisesjson.JSONDecodeErroron a corrupt or truncated transcript line, instead of skipping it like every read path in the package._append_jsonl_file_in_batchesparses each non-blank line with a barejson.loads(line):A partially-written final line — the normal outcome when the CLI is killed mid-append to a
.jsonl— therefore aborts the whole import withJSONDecodeError. The read path tolerates exactly this:_parse_transcript_entries(used byget_session_messages) wrapsjson.loadsintry/except (json.JSONDecodeError, ValueError): continue, so the same file reads fine. And because entries flush in batches, a large session leaves the store partially populated before the exception.The
import_session_to_storedocstring documents onlyValueError(bad UUID) andFileNotFoundError, so the raisedJSONDecodeErroris also undocumented.Fix
Skip corrupt/truncated lines, mirroring
_parse_transcript_entries. This aligns the behavior with both the documentedRaisescontract and the package-wide convention of quietly dropping truncated transcript lines (seetest_truncated_final_line_is_dropped_not_raised,test_corrupt_lines_skipped).Tests
Adds
test_skips_corrupt_truncated_line: two valid lines followed by a truncated one; asserts the import completes without raising and the store receives exactly the two valid entries (run withbatch_size=1so the earlier batches would already have flushed on the old code). Verified it fails onmainwithJSONDecodeErrorand passes with the fix. Full suite: 1058 passed, 5 skipped;ruffandmypyclean.