feat(table): add primary-key full-text read data layer#594
Merged
Conversation
…hared module Relocate the primary-key-index source-metadata leaf from `pk_vector_source` to a shared `pk_index_source` module and rename its types (`PkVectorSourceMeta`/`PkVectorSourceFile` -> `PrimaryKeyIndexSourceMeta`/ `PrimaryKeyIndexSourceFile`), moving the shared `should_read_pk_index_source` policy alongside them. The metadata is shared by the primary-key vector and full-text read paths, mirroring Java's `index/pk` package, so its messages use the neutral "index"/"source" wording. The on-disk byte format is unchanged and all vector call sites are re-pointed; behavior preserving.
Add `primary_key_full_text_index_enabled()` and `primary_key_full_text_index_columns()`, mirroring the Java `pk-full-text.index.columns` option: no default, split on ',' and trim each token with blank tokens preserved, returning a plain `Vec<String>` (never an error) that is empty only when the key is absent.
Add `PkFullTextBucketState::from_active_data_files`, the per-bucket current/stale reconciliation for full-text index payloads, mirroring Java `PkFullTextBucketIndexState`. A payload is current only when it indexes the target field, exactly covers the live source files at its level, and passes its per-payload validation; per-payload defects (wrong field carrying source metadata, parse failure, source-set mismatch, bad row count or range, an overflowing source row total) are collected as stale, while a payload that is not full-text, carries no global index metadata, or targets another field without source metadata is ignored. Only cross-payload conflicts across the surviving current set (a duplicate payload name or a source data file covered twice) fail loud. The source-file to current-payload map is insertion-ordered so the read path can build deterministically ordered splits.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Part of #567 (primary-key full-text + hybrid search on the Rust read side), the second slice after #563 (the
paimon-ftindex-corereader foundation).This adds the read-side data layer for primary-key full-text search — everything the scan/read path needs before opening and searching an archive, all independently unit-testable:
index/pk/package,PkFullTextBucketIndexState.fromActiveDataFiles.No archive I/O here, so nothing new is feature-gated — this is plain data-layer code, like the existing PK-vector data layer.
Brief change log
primary_key_full_text_index_enabled()/primary_key_full_text_index_columns(), mirroring Javapk-full-text.index.columns— no default;split(',', -1)then trim with blank tokens preserved; returns a plainVec<String>(never errors);[]only when the key is absent.spec/pk_vector_source.rs→spec/pk_index_source.rs, renamePkVectorSourceMeta/PkVectorSourceFile→PrimaryKeyIndexSourceMeta/PrimaryKeyIndexSourceFile, and move the shared source policy (should_read_pk_index_source) there — mirroring Java's sharedindex/pk/layer used by both vector and full-text. Byte format unchanged; all PK-vector call sites re-pointed; behavior-preserving.PK_FULL_TEXT_INDEX_TYPE = "full-text"andPkFullTextBucketState::from_active_data_files(...)— the independent current/stale reconciliation mirroring JavaPkFullTextBucketIndexState, reusing the shared leaf. Per-payload defects (row-count / row-range / overflow / source-set mismatch / field mismatch with source-meta / parse failure) are collected as stale; only cross-payload conflicts in the surviving current set (duplicate payload name, double-covered source) fail loud; wrong index type / missing global-index-meta / field mismatch without source-meta are ignored. The source-file → current-payload map preserves insertion order (matching Java'sLinkedHashMap) so the read path can build deterministically ordered splits.Note the public rename of the two source-meta spec types with no deprecated alias: the crate is pre-1.0, these are niche read-internal types, and the rename aligns with Java's shared
index/pknaming.Tests
[];"a, b ,c"→["a","b","c"]; blank-token preservation ("a,,b"→["a","","b"]," , "→["",""]); single column.should_readtruth table (COMPACT + level>0 true; APPEND / level 0 / absent false).cargo test -p paimon(1832) and--features fulltext(1853) green;cargo clippy -p paimon --all-targets(+--features fulltext)-D warningsclean;cargo fmt --all --checkclean;cargo build -p paimon-datafusion(Send boundary) passes.API and Format
Public rename of the two PK-index source-meta spec types (
PkVectorSourceMeta/PkVectorSourceFile→PrimaryKeyIndexSourceMeta/PrimaryKeyIndexSourceFile), no alias — pre-1.0, read-internal. The on-disk source-metadata byte format is unchanged; no storage-format change. The new CoreOptions accessors are additive.Documentation
No documentation changes; module- and item-level doc comments describe the new options, the shared leaf, and the reconciliation.