Bug
The Codex adapter fails an entire rollout file when any single JSONL record exceeds the hard-coded 16 MiB bufio.Scanner token limit, even when the file is below the configured watcher file-size limit.
Environment
- Observer: v1.19.0
- Platform: macOS arm64
- Adapter:
codex
observer.watch.max_file_size_mb = 128
Reproduction
Run:
observer scan --force --adapter codex
Against a valid 91 MiB Codex rollout JSONL containing one 23,355,441-byte record (about 22.3 MiB). The scan reports:
watcher.Scan: process failed adapter=codex err="codex.ParseSessionFile: scan: bufio.Scanner: token too long"
The record contents and local path are omitted for privacy. The failure is reproducible from the measured record length alone.
Root cause
internal/adapter/codex/adapter.go sets const maxLine = 16 * 1024 * 1024 in both the main parser and the session-context prefetch path. Raising observer.watch.max_file_size_mb only passes the outer whole-file guard; it cannot change this per-record limit.
ParseSessionFile returns the scanner error before the watcher ingests the accumulated parse result, so the whole session is skipped rather than only the oversized record.
Expected behavior
Legitimate Codex rollout records within the configured whole-file allowance should be ingestible, or the limit/error behavior should be configurable and explicit.
Suggested fix
Any of these would address the mismatch:
- Derive a bounded per-record limit from
observer.watch.max_file_size_mb.
- Add a separate
max_line_size_mb setting.
- Replace
bufio.Scanner with a reader that supports large newline-delimited records while enforcing an explicit safety bound.
As a minimum compatibility fix, increasing both Codex maxLine constants to 64 MiB would cover this real-world record. A regression test with a valid JSONL record larger than 16 MiB would prevent recurrence.
Bug
The Codex adapter fails an entire rollout file when any single JSONL record exceeds the hard-coded 16 MiB
bufio.Scannertoken limit, even when the file is below the configured watcher file-size limit.Environment
codexobserver.watch.max_file_size_mb = 128Reproduction
Run:
observer scan --force --adapter codexAgainst a valid 91 MiB Codex rollout JSONL containing one 23,355,441-byte record (about 22.3 MiB). The scan reports:
The record contents and local path are omitted for privacy. The failure is reproducible from the measured record length alone.
Root cause
internal/adapter/codex/adapter.gosetsconst maxLine = 16 * 1024 * 1024in both the main parser and the session-context prefetch path. Raisingobserver.watch.max_file_size_mbonly passes the outer whole-file guard; it cannot change this per-record limit.ParseSessionFilereturns the scanner error before the watcher ingests the accumulated parse result, so the whole session is skipped rather than only the oversized record.Expected behavior
Legitimate Codex rollout records within the configured whole-file allowance should be ingestible, or the limit/error behavior should be configurable and explicit.
Suggested fix
Any of these would address the mismatch:
observer.watch.max_file_size_mb.max_line_size_mbsetting.bufio.Scannerwith a reader that supports large newline-delimited records while enforcing an explicit safety bound.As a minimum compatibility fix, increasing both Codex
maxLineconstants to 64 MiB would cover this real-world record. A regression test with a valid JSONL record larger than 16 MiB would prevent recurrence.