fix(planner): defer index search and provider I/O to execute() time#2
Merged
Conversation
Previously plan_extension performed all real work — HNSW index search, full provider scan with collect(), and fetch_by_keys — during physical planning. This caused O(N) memory spikes at plan time for large tables, hid I/O from DataFusion execution metrics, and prevented cancellation. plan_extension is now purely structural: validates the registry entry and compiles filter Exprs to PhysicalExprs. All I/O moves into USearchExec::execute(), which returns a SendableRecordBatchStream via RecordBatchStreamAdapter + futures::stream::once. The filtered path streams the provider scan batch-by-batch (O(1) memory) instead of collecting the entire table into memory. The provider scan plan is pre-planned in plan_extension (cheap: creates a MemoryExec-like plan from the in-memory MemTable) and executed lazily at query time. Also: add Debug impl for USearchRegistry; fix udtf.rs to use a local BatchExec instead of the old 3-arg USearchExec::new.
- cargo fmt: reformat execute() stream chain, usearch_execute signature, usearch_search call, and BatchExec::new struct literal - clippy: extract SearchParams struct to group the 11 search-config fields, reducing usearch_execute to 2 args and adaptive_filtered_execute to 4 args (eliminates both too_many_arguments warnings without #[allow]) - private_interfaces: make USearchExec::new private (only called within planner.rs; SearchParams is module-private)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Root cause:
plan_extensionwas doing all real work (HNSW index search, full providercollect(),fetch_by_keys) during physical planning. With 1M+ rows this causes an O(N) memory spike at plan time, blocks the planner thread, hides I/O fromEXPLAIN ANALYZE, and prevents query cancellation.Unfiltered path:
usearch_search+fetch_by_keysnow run insideUSearchExec::execute()via afutures::stream::onceasync block bridged throughRecordBatchStreamAdapter.Filtered path: The provider scan is pre-planned (cheap: creates an in-memory execution plan) in
plan_extensionusingSessionState, then executed lazily at query time usingscan_plan.execute(0, task_ctx). Iteration is now a streaming loop (stream.next().await) instead ofcollect(), giving O(1) memory for the scan phase.No behaviour change for existing tests — all 27 pass unchanged.
Test plan
cargo check— no errorscargo clippy— no errors (two expectedtoo_many_argumentswarnings on private async fns)cargo test— all 27 tests passEXPLAIN ANALYZEon a vector search query should show real execution time insideUSearchExecinstead of near-zero cost