Skip to content

perf(snapshot): optimize snapshot listing#202

Merged
XuPeng-SH merged 1 commit into
matrixorigin:mainfrom
gouhongshen:codex/snapshot-list-filter
May 21, 2026
Merged

perf(snapshot): optimize snapshot listing#202
XuPeng-SH merged 1 commit into
matrixorigin:mainfrom
gouhongshen:codex/snapshot-list-filter

Conversation

@gouhongshen
Copy link
Copy Markdown
Contributor

@gouhongshen gouhongshen commented May 18, 2026

What type of PR is this?

  • feat (new feature)
  • fix (bug fix)
  • docs (documentation)
  • style (formatting, no code change)
  • refactor (code change that neither fixes a bug nor adds a feature)
  • perf (performance improvement)
  • test (adding or updating tests)
  • chore (maintenance, tooling)
  • build / ci (build or CI changes)

Which issue(s) this PR fixes

Fixes matrixorigin/memoria-website#132

What this PR does / why we need it

Root cause

/v1/snapshots called SHOW SNAPSHOTS without a predicate. On a cluster with many users, MatrixOne returned every visible snapshot first, and Memoria filtered the current user's database in Rust. With 10K snapshots this made snapshot listing spend seconds in SHOW SNAPSHOTS even when the current user only had 50 snapshots.

Changes

  • Filter MatrixOne snapshot listing at the source with SHOW SNAPSHOTS WHERE DATABASE_NAME = '<user db>'.
  • Decode the lower-case output column names returned by MatrixOne's SHOW ... WHERE rewrite.
  • Store derived snapshot metadata in mem_snapshots.extra JSON instead of adding a narrow memory_count column.
  • Persist extra.memory_count when creating snapshots.
  • Lazily backfill extra.memory_count for old snapshot registrations on read, while still returning a concrete memory_count in /v1/snapshots.
  • Keep focused snapshot-listing timing logs so regressions are easy to identify.

Compatibility

  • No user schema version bump.
  • Existing user DBs get a targeted mem_snapshots.extra JSON column migration when accessed.
  • Old registrations with missing extra.memory_count are read-through backfilled for returned snapshots only.
  • New users get the extra column in the initial mem_snapshots schema.

Validation

  • SQLX_OFFLINE=true cargo check -p memoria-storage -p memoria-git -p memoria-mcp -p memoria-api
  • Remote MatrixOne real-snapshot stress reuse: 200 users, 100K memories, 10K physical snapshots, 400/400 /v1/snapshots requests succeeded.
  • SHOW SNAPSHOTS rows per request dropped from 10004 to 50.
  • SHOW SNAPSHOTS latency improved from p50 1.67s / p95 4.35s / max 9.93s to p50 185ms / p95 376ms / max 421ms.
  • API/MCP walkthrough on a complete-schema test user: 56 checks, 0 failures.

@gouhongshen gouhongshen marked this pull request as ready for review May 18, 2026 08:38
@gouhongshen gouhongshen changed the title [codex] Optimize snapshot listing perf(snapshot): optimize snapshot listing May 18, 2026
@XuPeng-SH XuPeng-SH merged commit 3207091 into matrixorigin:main May 21, 2026
5 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants