fix: fjall compaction, Drop deadlock, and pre-commit hook toolchain

[mwiewior]

Summary

• load_sift_cache: add missing major_compact() after persist — without this, every SIFT lookup probes hundreds of L0 SSTs
• CacheLoader: add mem::forget after persist + compact to avoid fjall Drop deadlock on large datasets (bounded flume channel fills up)
• cache_builder: apply mem::forget fix for both build_variation_fjall_from_parquet() and build_sift_fjall() (cherry-pick from unmerged fix: use mem::forget to avoid fjall Drop deadlock (#86) #87)
• pre-commit: switch from doublify/pre-commit-rust v1.0 (2019) to local hooks using the project's own cargo toolchain — the old hooks used a rustfmt that didn't support edition 2024 let-chains
• compact_fjall: new utility example for re-compacting existing uncompacted stores without re-ingesting data

Supersedes #87. Closes #86.

Root cause of ~10% perf degradation

Both variation.fjall (1,829 files) and translation_sift.fjall (437 files) were fully uncompacted — every table file was an L0 SST at ~62MB with zero merged levels. This means every point lookup probes bloom filters across all files instead of a handful of compacted segments.

Store	L0 SST files	Total size	Expected after compaction
variation.fjall	1,829	109 GB	~5-10 files
translation_sift.fjall	437	26 GB	~3-5 files

Cause: load_sift_cache never called major_compact(). For variation, either the build was killed during the Drop hang (issue #86) or compaction itself was interrupted.

Test plan

• All 490 lib tests pass
• cargo clippy --all-targets --all-features clean
• cargo fmt --all --check clean
• Pre-commit hooks pass with new local config
• Rebuild fjall stores and verify compaction completes
• Run compact_fjall on existing uncompacted stores

🤖 Generated with Claude Code

[mwiewior]

@claude pls review

[claude]

Claude finished @mwiewior's task in 1m 56s —— View job

---

PR Review

• Gather context and read changed files
• Review core logic changes (fjall compaction, Drop deadlock fixes)
• Review pre-commit config changes
• Review new examples (compact_fjall.rs, strip_fjall.rs)
• Review kv_store and sift_store additions
• Summarize findings

---

Overall the PR is well-motivated and well-documented. The root cause analysis is solid and the fixes are generally correct. One medium-severity bug, two minor issues.

---

Bug: compact_fjall.rs will create the extra keyspaces it's trying to avoid

datafusion/bio-function-vep/examples/compact_fjall.rs:43-51

let known_keyspaces = ["data", "meta", "sift", "translations", "exons"];

for ks_name in &known_keyspaces {
    // ...
    let ks = match db.keyspace(ks_name, fjall::KeyspaceCreateOptions::default) {
        Ok(ks) => ks,
        Err(_) => continue,
    };

db.keyspace() with KeyspaceCreateOptions::default creates the keyspace if it doesn't already exist. Running compact_fjall on variation.fjall will open/create "sift", "translations", and "exons" — exactly the extra keyspaces this PR removes for the ~10% perf regression. By contrast, strip_fjall.rs correctly calls db.keyspace_exists(name) first and skips missing ones.

Fix: add an existence guard before the db.keyspace() call:

if !db.keyspace_exists(ks_name) {
    continue;
}
let ks = match db.keyspace(ks_name, fjall::KeyspaceCreateOptions::default) {

Fix this →

---

Minor: strip_fjall.rs lists "meta" as expected for translation_sift.fjall

datafusion/bio-function-vep/examples/strip_fjall.rs:120

check_keyspaces(&sift_fjall, &["meta", "sift"]);

SiftKvStore::create() only opens the "sift" keyspace — there's no "meta" keyspace in translation_sift.fjall (confirmed by test_create_produces_minimal_keyspaces asserting exactly 2 dirs: fjall default + sift). Listing "meta" as expected is harmless (it just won't appear since keyspace_exists("meta") will be false), but it can mislead users of the tool who see no ✓ 'meta' line and wonder if the store is corrupt.

Suggest: check_keyspaces(&sift_fjall, &["sift"]).

---

Minor: mem::forget in cache_builder.rs lacks the #[cfg(not(test))] guard

datafusion/bio-function-vep/src/cache_builder.rs:908 and :1459

loader.rs wraps its mem::forget in #[cfg(not(test))] so that tests can correctly release the fjall lock file and allow subsequent VepKvStore::open() calls in the same test directory. The cache_builder.rs builder functions don't have this guard. This is probably safe today since tests don't call build_variation_fjall_from_parquet() / build_sift_fjall() directly, but adding the guard consistently would prevent a hard-to-diagnose test failure if that changes.

---

What's well done

• #[cfg(not(test))] guard in loader.rs — exactly right; this is the correct way to handle the test vs. production trade-off for mem::forget.
• Compaction before mem::forget — persist() + major_compact() is called before forgetting the store in all paths, so no data is lost.
• Deadlock explanation comments — clear, with issue links, explaining the bounded flume channel mechanism. Future maintainers will understand why this isn't a memory leak.
• Regression tests for keyspace count — test_create_produces_minimal_keyspaces / test_open_does_not_create_extra_keyspaces in both kv_store.rs and sift_store.rs will catch the previous extra-keyspace bug at build time.
• Pre-commit migration — switching from the 2019 doublify/pre-commit-rust (which shipped an old rustfmt without edition 2024 support) to local cargo fmt/cargo clippy is the right call.
• load_context_kv.rs behavioral fix — writing context keyspaces to a separate context.fjall instead of polluting variation.fjall is architecturally cleaner; the comment explaining the perf implication is helpful.