Skip to content

fix: skip corrupt/truncated lines in import_session_to_store#1093

Open
Otis0408 wants to merge 1 commit into
anthropics:mainfrom
Otis0408:fix/import-session-skip-corrupt-lines
Open

fix: skip corrupt/truncated lines in import_session_to_store#1093
Otis0408 wants to merge 1 commit into
anthropics:mainfrom
Otis0408:fix/import-session-skip-corrupt-lines

Conversation

@Otis0408

@Otis0408 Otis0408 commented Jul 8, 2026

Copy link
Copy Markdown

Summary

import_session_to_store raises json.JSONDecodeError on a corrupt or truncated transcript line, instead of skipping it like every read path in the package.

_append_jsonl_file_in_batches parses each non-blank line with a bare json.loads(line):

for line in f:
    line = line.rstrip("\n")
    if not line:
        continue
    batch.append(json.loads(line))   # <-- unguarded

A partially-written final line — the normal outcome when the CLI is killed mid-append to a .jsonl — therefore aborts the whole import with JSONDecodeError. The read path tolerates exactly this: _parse_transcript_entries (used by get_session_messages) wraps json.loads in try/except (json.JSONDecodeError, ValueError): continue, so the same file reads fine. And because entries flush in batches, a large session leaves the store partially populated before the exception.

The import_session_to_store docstring documents only ValueError (bad UUID) and FileNotFoundError, so the raised JSONDecodeError is also undocumented.

Fix

Skip corrupt/truncated lines, mirroring _parse_transcript_entries. This aligns the behavior with both the documented Raises contract and the package-wide convention of quietly dropping truncated transcript lines (see test_truncated_final_line_is_dropped_not_raised, test_corrupt_lines_skipped).

Tests

Adds test_skips_corrupt_truncated_line: two valid lines followed by a truncated one; asserts the import completes without raising and the store receives exactly the two valid entries (run with batch_size=1 so the earlier batches would already have flushed on the old code). Verified it fails on main with JSONDecodeError and passes with the fix. Full suite: 1058 passed, 5 skipped; ruff and mypy clean.

_append_jsonl_file_in_batches parsed each line with a bare json.loads(line),
so a partially-written final line — the normal outcome when the CLI is killed
mid-append to a .jsonl — made import_session_to_store raise json.JSONDecodeError.
Every read path in the package tolerates this: _parse_transcript_entries (used
by get_session_messages) wraps json.loads in try/except and skips corrupt
lines, so the same file reads fine. Because entries flush in batches, a large
session also left the store partially populated before the crash.

Skip corrupt/truncated lines to match the read path. import_session_to_store's
documented Raises (ValueError, FileNotFoundError) already excludes
JSONDecodeError, so this aligns behavior with the contract. Adds a regression
test.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant