fix(opencode): honor --since/--until in SQLite loader and JSON dump scan#1188
fix(opencode): honor --since/--until in SQLite loader and JSON dump scan#1188justi wants to merge 1 commit into
Conversation
The OpenCode adapter loaded every row from `opencode.db` and parsed every file under `storage/message/*.json` on every invocation, regardless of the `--since` / `--until` filters that `SharedArgs` already carries. On long-lived installs (33 GB DB, 100k+ legacy JSON files) the loader appeared to hang. - SQLite query now uses the existing `time_created` index when bounds are set: `WHERE time_created >= ?1 AND time_created < ?2` (plus the open-ended variants). - The `storage/message/*.json` loop now skips files whose `mtime` is outside the inferred range, before any read or parse. - Bounds are inflated by -1 / +2 days so the existing summary-time filter (`summary.rs:265-271`) remains authoritative; the SQL/mtime short-circuits never drop legitimate rows under timezone offsets or FAT32-style 2-second mtime rounding. Adds 4 tests in `adapter::opencode::loader::tests` covering the SQL bounds and mtime skip paths. All 285 workspace tests pass; `cargo clippy --workspace --all-targets -- -D warnings` is clean; `cargo fmt --all -- --check` is clean. Refs ryoppippi#801, ryoppippi#867, ryoppippi#960. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
This PR was auto-closed. Only contributors approved with Maintainers review auto-closed issues and reopen worthwhile ones. Issues that do not meet the quality bar in CONTRIBUTING.md may not be reopened or receive a reply. If a maintainer replies See CONTRIBUTING.md. |
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR adds time-based filtering with slack-adjusted bounds to the OpenCode data loader. Date bounds are parsed once from ChangesDate-range filtering with slack windows
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
Summary
ccusage opencode(and the aggregateccusage) reads every row from the OpenCode SQLitemessagetable on every invocation, regardless of--since/--until, and then re-reads every JSON file understorage/message/*.json. On long-lived OpenCode installs (especially those migrated from the pre-SQLite layout) the loader can hang for minutes or appear to lock up.SharedArgs.since/SharedArgs.untilwere already plumbed into the adapter (ccusage-cli/src/types.rs:34-35), but the loader did not use them.Reproduction (local data, before this PR)
On
main @ 9d90e1b, opencode storage with:opencode.db: 33.6 GB, 392,340 rows inmessagestorage/message/: 118,619 JSON files (legacy pre-SQLite dump, ~2.0 GB)$ ccusage opencode --since 2026-05-04 --until 2026-05-10 # never returns; the loader scans 392k DB rows + parses 118k JSON filesRaw SQL with
WHERE time_created BETWEEN ? AND ?against the same DB returns in <50 ms.Fix
rust/crates/ccusage/src/adapter/opencode/loader.rs:SQLite query uses the existing index. The
messagetable already hasCREATE INDEX message_session_time_created_id_idx ON message (session_id, time_created, id). Whenshared.since/shared.untilare set, the prepared statement now addsWHERE time_created >= ?1 AND time_created < ?2(or open-ended variants).storage/message/*.jsonloop skips by mtime. Files outside the inferred range are skipped viafs::metadata(...).modified()before any read/parse.Slack. since/until are inflated by
-1/+2days before being applied, so the existing string-based summary filter (summary.rs:265-271) remains authoritative; the SQL/mtime checks only short-circuit work that the summary would have discarded anyway. This absorbs timezone offsets and any FAT32-style 2-second mtime rounding.Performance on local data
--since 2026-05-04 --until 2026-05-10)main @ 9d90e1bWHEREonlyWHERE+ mtime-skipLoaded tokens are identical across variants and match a hand-written
sqlite3query against the same DB: 414,644 input / 4,646 output.Cross-OS compatibility
Runtime:
std::fs::metadata().modified()works on macOS APFS/HFS+, Linux ext4/btrfs/xfs, and Windows NTFS. FAT32's 2-second mtime granularity is covered by the slack.Tests: new tests use the
filetimecrate (FileTime::from_unix_time) which maps toutimensat(Linux),setattrlist(macOS), andSetFileTime(Windows). Same code path runs on all three CI targets.Tests added
4 new tests in
adapter::opencode::loader::tests:since_filter_drops_db_rows_older_than_lower_bounduntil_filter_drops_db_rows_at_or_after_upper_boundsince_filter_skips_json_files_with_older_mtimeno_since_until_keeps_all_json_files_regardless_of_mtimeAll 285 workspace tests pass;
cargo clippy --workspace --all-targets -- -D warningsis clean;cargo fmt --all -- --checkis clean.Why this is a fresh narrow PR
The same user-visible gap was reported in #801, requested in #867, and previously fixed against the now-removed TypeScript
@ccusage/opencodepackage in #960 (closed with: "If a gap remains, it needs a fresh narrow PR against main"). This PR is that narrow PR against the current Rust adapter. No CLI surface, docs, or configuration schema changes.Summary by cubic
Make
ccusage opencodehonor --since/--until in the OpenCode adapter. This stops full DB scans and legacy JSON parsing, cutting runtime on large installs from “never returns” to seconds.Bug Fixes
time_createdindex.storage/message/*.jsonby file mtime before reading.Dependencies
filetimefor setting mtimes in tests.Written for commit e36d426. Summary will update on new commits.
Summary by CodeRabbit
Release Notes
New Features
Tests