feat(ftindex): primary-key full-text and hybrid search read path (full scope)#568
Draft
JunRuiLee wants to merge 22 commits into
Draft
feat(ftindex): primary-key full-text and hybrid search read path (full scope)#568JunRuiLee wants to merge 22 commits into
JunRuiLee wants to merge 22 commits into
Conversation
Route single-payload validation failures in the PK full-text bucket state to stale during classification (row-count mismatch, row-range mismatch, and an overflowing source row total via checked_add), keeping only cross-payload conflicts across the surviving current set (duplicate payload file name, double-covered source data file) as fail-loud errors. Preserve blank tokens when parsing the PK full-text index columns option so it matches the reference split-and-trim.
Add the primary-key hybrid execute_read path: dispatch when every route resolves to a physical primary-key route, run each route's candidate producer against one pinned snapshot, convert hits to shared physical positions, fuse them via the configured ranker, select the physical source files the fused positions reference, build indexed splits carrying the fused raw scores, materialize, and reorder best-fused-first while stripping the internal position column. Mixed physical/global routes and execute/execute_scored on a primary-key hybrid fail loud; the global path is unchanged.
…ey hybrid routes The primary-key hybrid full-text route searched compaction-visible payloads without checking the global-index search mode, so FULL/DETAIL would silently degrade instead of failing. Reject non-FAST modes loud in the route producer, mirroring the full-text read path. Each route also resolved its own snapshot and the code only checked the returned ids were equal afterward, so a concurrent commit between route plans could split routes across versions. Resolve one snapshot up front on the read-latest path and pin it into every route via scan.snapshot-id; keep the equality check as a defensive guard.
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Implements #567 — the complete primary-key full-text and hybrid search read path for the Rust side, mirroring Java Paimon. This is the full picture of what I've implemented locally (posted as one draft so the whole scope is visible); it builds on the reader foundation in #563. Depends on the shared native core
paimon-ftindex-core(apache/paimon-full-text, tantivy 0.26.1) — the same engine the Java/Python bindings use — so Rust reads the exact on-disk archives Java writes.Scope (four parts, all implemented + tested locally)
1. Reader foundation (= #563)
paimon-ftindex-core, gated behind thefulltextfeature.crates/paimon/src/ftindex/:FullTextArchiveReader(from_input_file/search/search_with_includewith a roaring live-row allow-list); fail-loud error mapping.2. PK full-text data layer
CoreOptions::primary_key_full_text_index_columns()(lenient, mirrors Javapk-full-text.index.columns).spec/pk_index_source.rs(PrimaryKeyIndexSourceMeta/PrimaryKeyIndexSourceFile+should_readpolicy), shared by the vector and full-text paths (mirrors Javaindex/pk).PkFullTextBucketState— per-bucket current/stale payload reconciliation ("full-text"index type), mirroring JavaPkFullTextBucketIndexState.3. PK full-text read path (FAST mode)
PrimaryKeyFullTextScan/PrimaryKeyFullTextSearchSplit(manifest scan,(partition, bucket)grouping, split invariants).PrimaryKeyFullTextBucketSearch(cumulativeSourceRangeoffsets, live-rowinclude= active ranges − deletion vectors, archive row-id → physical position).PrimaryKeyFullTextRead+ a PK branch infull_text_search_builder— materializes rows by physical position +__paimon_search_score, best-score-first;execute/execute_scoredfail loud on PK (physical positions, not global row ids).4. Hybrid search on PK tables
PrimaryKeySearchPosition+PrimaryKeySearchRanker(RRF / weighted-score / weighted-MRR,DEFAULT_RRF_K = 60), a faithful port of JavaPrimaryKeySearchRanker.HybridSearchBuilderImpl.Tests
Full
cargo test -p paimon --features fulltextgreen (1921 passed / 0 failed), including the vector regression, PK-vector baseline, and the Java-written fixture.cargo clippy -p paimon --lib --tests --features fulltext -- -D warningsclean; default (no-feature) build andcargo build -p paimon-datafusion(Send boundary) both pass.API and Format
No public API change to existing paths and no storage-format change; adds feature-gated readers over the Java-written archive formats (v1 full-text) via
paimon-ftindex-core.Documentation
Module-level doc comments describe each component.