Skip to content

fix(opencode): honor --since/--until in SQLite loader and JSON dump scan#1188

Open
justi wants to merge 1 commit into
ryoppippi:mainfrom
justi:fix/opencode-since-until-adapter
Open

fix(opencode): honor --since/--until in SQLite loader and JSON dump scan#1188
justi wants to merge 1 commit into
ryoppippi:mainfrom
justi:fix/opencode-since-until-adapter

Conversation

@justi
Copy link
Copy Markdown

@justi justi commented May 29, 2026

Summary

ccusage opencode (and the aggregate ccusage) reads every row from the OpenCode SQLite message table on every invocation, regardless of --since / --until, and then re-reads every JSON file under storage/message/*.json. On long-lived OpenCode installs (especially those migrated from the pre-SQLite layout) the loader can hang for minutes or appear to lock up.

SharedArgs.since / SharedArgs.until were already plumbed into the adapter (ccusage-cli/src/types.rs:34-35), but the loader did not use them.

Reproduction (local data, before this PR)

On main @ 9d90e1b, opencode storage with:

  • opencode.db: 33.6 GB, 392,340 rows in message
  • storage/message/: 118,619 JSON files (legacy pre-SQLite dump, ~2.0 GB)
$ ccusage opencode --since 2026-05-04 --until 2026-05-10
# never returns; the loader scans 392k DB rows + parses 118k JSON files

Raw SQL with WHERE time_created BETWEEN ? AND ? against the same DB returns in <50 ms.

Fix

rust/crates/ccusage/src/adapter/opencode/loader.rs:

  1. SQLite query uses the existing index. The message table already has CREATE INDEX message_session_time_created_id_idx ON message (session_id, time_created, id). When shared.since / shared.until are set, the prepared statement now adds WHERE time_created >= ?1 AND time_created < ?2 (or open-ended variants).

  2. storage/message/*.json loop skips by mtime. Files outside the inferred range are skipped via fs::metadata(...).modified() before any read/parse.

  3. Slack. since/until are inflated by -1 / +2 days before being applied, so the existing string-based summary filter (summary.rs:265-271) remains authoritative; the SQL/mtime checks only short-circuit work that the summary would have discarded anyway. This absorbs timezone offsets and any FAT32-style 2-second mtime rounding.

Performance on local data

Variant Runtime (--since 2026-05-04 --until 2026-05-10)
main @ 9d90e1b did not return
this PR, SQL WHERE only 186.68 s
this PR, SQL WHERE + mtime-skip 9.57 s

Loaded tokens are identical across variants and match a hand-written sqlite3 query against the same DB: 414,644 input / 4,646 output.

Cross-OS compatibility

Runtime: std::fs::metadata().modified() works on macOS APFS/HFS+, Linux ext4/btrfs/xfs, and Windows NTFS. FAT32's 2-second mtime granularity is covered by the slack.

Tests: new tests use the filetime crate (FileTime::from_unix_time) which maps to utimensat (Linux), setattrlist (macOS), and SetFileTime (Windows). Same code path runs on all three CI targets.

Tests added

4 new tests in adapter::opencode::loader::tests:

  • since_filter_drops_db_rows_older_than_lower_bound
  • until_filter_drops_db_rows_at_or_after_upper_bound
  • since_filter_skips_json_files_with_older_mtime
  • no_since_until_keeps_all_json_files_regardless_of_mtime

All 285 workspace tests pass; cargo clippy --workspace --all-targets -- -D warnings is clean; cargo fmt --all -- --check is clean.

Why this is a fresh narrow PR

The same user-visible gap was reported in #801, requested in #867, and previously fixed against the now-removed TypeScript @ccusage/opencode package in #960 (closed with: "If a gap remains, it needs a fresh narrow PR against main"). This PR is that narrow PR against the current Rust adapter. No CLI surface, docs, or configuration schema changes.


Summary by cubic

Make ccusage opencode honor --since/--until in the OpenCode adapter. This stops full DB scans and legacy JSON parsing, cutting runtime on large installs from “never returns” to seconds.

  • Bug Fixes

    • SQLite: add time bounds in the query (open-ended variants) and bind params; uses the existing time_created index.
    • JSON dump: skip storage/message/*.json by file mtime before reading.
    • Safety: widen bounds by -1/+2 days so the summary-time filter remains authoritative and tolerates TZ drift and coarse mtimes.
    • Tests: add 4 tests for SQL bounds and mtime skipping; all checks pass.
  • Dependencies

    • Add dev-dependency filetime for setting mtimes in tests.

Written for commit e36d426. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

Release Notes

  • New Features

    • Date-range filtering is now available when loading OpenCode data from SQLite databases and JSON files, including configurable slack window support that extends specified time boundaries.
  • Tests

    • Test coverage expanded to validate date-range filtering functionality, including verification of time-based row exclusion from SQLite sources and file modification time-based filtering for JSON files.

Review Change Stack

The OpenCode adapter loaded every row from `opencode.db` and parsed every
file under `storage/message/*.json` on every invocation, regardless of
the `--since` / `--until` filters that `SharedArgs` already carries.
On long-lived installs (33 GB DB, 100k+ legacy JSON files) the loader
appeared to hang.

- SQLite query now uses the existing `time_created` index when bounds
  are set: `WHERE time_created >= ?1 AND time_created < ?2` (plus the
  open-ended variants).
- The `storage/message/*.json` loop now skips files whose `mtime` is
  outside the inferred range, before any read or parse.
- Bounds are inflated by -1 / +2 days so the existing summary-time
  filter (`summary.rs:265-271`) remains authoritative; the SQL/mtime
  short-circuits never drop legitimate rows under timezone offsets
  or FAT32-style 2-second mtime rounding.

Adds 4 tests in `adapter::opencode::loader::tests` covering the
SQL bounds and mtime skip paths. All 285 workspace tests pass;
`cargo clippy --workspace --all-targets -- -D warnings` is clean;
`cargo fmt --all -- --check` is clean.

Refs ryoppippi#801, ryoppippi#867, ryoppippi#960.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@github-actions
Copy link
Copy Markdown
Contributor

This PR was auto-closed. Only contributors approved with lgtm can open PRs. Open an issue first.

Maintainers review auto-closed issues and reopen worthwhile ones. Issues that do not meet the quality bar in CONTRIBUTING.md may not be reopened or receive a reply.

If a maintainer replies lgtmi, your future issues will stay open. If a maintainer replies lgtm, your future issues and PRs will stay open.

See CONTRIBUTING.md.

@github-actions github-actions Bot closed this May 29, 2026
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 29, 2026

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4b350b27-a045-441a-b2cf-0fca35c82a04

📥 Commits

Reviewing files that changed from the base of the PR and between 9d90e1b and e36d426.

⛔ Files ignored due to path filters (1)
  • rust/Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (2)
  • rust/crates/ccusage/Cargo.toml
  • rust/crates/ccusage/src/adapter/opencode/loader.rs

📝 Walkthrough

Walkthrough

This PR adds time-based filtering with slack-adjusted bounds to the OpenCode data loader. Date bounds are parsed once from YYYYMMDD strings into UTC milliseconds, then applied to skip JSON files and SQLite rows outside the range. Tests validate filtering behavior across both data sources.

Changes

Date-range filtering with slack windows

Layer / File(s) Summary
Time utilities and parsing
rust/crates/ccusage/src/adapter/opencode/loader.rs
Adds imports for UNIX_EPOCH and time zone utilities; defines MS_PER_DAY and helpers to parse YYYYMMDD strings into UTC epoch milliseconds and compute slack-adjusted (since_ms, until_ms) bounds (since minus 1 day, until plus 2 days).
JSON message file filtering via mtime
rust/crates/ccusage/src/adapter/opencode/loader.rs
Computes date bounds once inside load_entries_from_directory and applies mtime-based filtering before reading JSON files; files outside the range are skipped.
SQLite database filtering via time_created
rust/crates/ccusage/src/adapter/opencode/loader.rs
Modifies load_entries_from_database to filter message table rows by time_created using conditional SQL with bound parameters; handles prepare/bind failures by logging and returning empty results.
Test helper, dependency, and coverage
rust/crates/ccusage/Cargo.toml, rust/crates/ccusage/src/adapter/opencode/loader.rs
Adds filetime = "0.2" dev dependency; introduces create_db_message_with_time test helper for creating SQLite rows with explicit timestamps; validates DB filtering by since/until and JSON mtime filtering when bounds are set.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

A rabbit hops through time with glee, 🐰
Filtering dates from JSON and DB with ease,
Slack windows buffering days,
In mtime and time_created ways—
No old records shall escape, you'll see! ✨

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@socket-security
Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security
Vulnerability Quality Maintenance License
Addedcargo/​filetime@​0.2.2910010093100100

View full report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants