feat(ftindex): read Java full-text archives via paimon-ftindex-core

[JunRuiLee]

Purpose

First step of mirroring Java Paimon's primary-key full-text search into the Rust read side. Java's full-text index is backed by a shared native Rust core, paimon-ftindex-core (public repo apache/paimon-full-text, tantivy 0.26.1), which the Java/Python bindings wrap via JNI/ctypes. This PR lets the Rust reader depend on that same core directly (no JNI needed), so it can read the on-disk archive format Java writes.

This is a deliberately small, self-contained foundation slice: dependency wiring + a thin reader wrapper + round-trip tests. It de-risks the external dependency (does it fetch and build here?) before any of the primary-key scan/read machinery is built on top.

Brief change log

• Add optional git dependency paimon-ftindex-core (tag v0.1.0-rc4), wired into the existing fulltext cargo feature via dep: syntax.
• New feature-gated module crates/paimon/src/ftindex/ with FullTextArchiveReader:
  • from_input_file(&InputFile) — reads the archive bytes and opens the core reader over an in-memory SliceReader.
  • search(query_json, limit) — JSON-DSL query, returns FullTextHits { row_ids, scores }.
  • search_with_include(query_json, limit, RoaringTreemap) — restricts results to a live-row allow-list (the withIncludeRowIds equivalent).
• Fail-loud error mapping (engine errors surface as crate::Error, no silent fallback).

Out of scope (later PRs): shared PK-index-layer generalization, CoreOptions full-text options, PK full-text scan/read, hybrid-on-PK, and migrating the existing append/data-evolution full-text path off raw tantivy 0.22 onto the core.

Tests

• ftindex::reader::tests::test_round_trip_search_returns_expected_rows — build an archive with the core writer, read it back, assert the correct row-ids/scores.
• ftindex::reader::tests::test_search_with_include_restricts_to_allow_list — all docs match; the roaring include-filter restricts results to the allow-list subset (also validates cross-crate roaring 0.11 compatibility).
• Full cargo test -p paimon --features fulltext green (lib 1724 passed / 0 failed); cargo clippy -p paimon --lib --tests --features fulltext -- -D warnings clean; default (no-feature) build and cargo build -p paimon-datafusion (Send boundary) both pass.

API and Format

Dependency note (why this is draft): paimon-ftindex-core is pinned to a git tag (v0.1.0-rc4) for now because the crate is not yet on crates.io. This builds and tests fine, but a git-only dependency blocks cargo package/publishing of paimon. Once paimon-ftindex-core is released to crates.io, this will be switched to a registry version (paimon-ftindex-core = "<ver>") and the PR marked ready — keeping it draft until then.

No public API change to existing paths and no storage-format change. Adds a new feature-gated reader over the existing Java-written full-text archive format (v1), read through the shared paimon-ftindex-core engine. The fulltext feature transitionally pulls both tantivy 0.22 (legacy append path) and 0.26 (via the core); the duplicate is removed when the append path converges onto the core in a later PR.

Documentation

No documentation changes; module-level doc comments describe the new reader.

[shyjsarah]

I’ve completed the implementation and am currently waiting for the paimon-full-text release. Once it’s available, I’ll submit the PR. I’d really appreciate your review then.

[JunRuiLee]

I’ve completed the implementation and am currently waiting for the paimon-full-text release. Once it’s available, I’ll submit the PR. I’d really appreciate your review then.

Thanks for the heads-up! Could you share which part you've been working on?
I'd originally planned to do the full primary-key full-text and hybrid read path myself, and my implementation is already quite far along — I've opened #568 as a complete draft so you can see what it covers (reader foundation → PK full-text data layer → PK full-text read path → PK hybrid search), all passing the full test suite; it's just gated on the paimon-ftindex-core crates.io release before I can mark it ready.

Tracking issue: #567. You're very welcome to review once it's up.

This PR (and #568) is only waiting on the paimon-ftindex-core crates.io release — the dependency is pinned to a GitHub git tag for now, and as soon as the crate is published I'll switch that to the crates.io version and mark the PRs ready. You're very welcome to review once they're up.

[shyjsarah]

Thanks for sharing #568. I've opened #571 as a draft so the scope and implementation are visible.

My work focuses on the existing non-primary-key/global-index full-text path and DataFusion integration, rather than the primary-key full-text and hybrid search path covered by #568. It migrates the legacy in-repository Tantivy implementation to paimon-ftindex-core and covers snapshot index-manifest reads, fast/full/detail modes, deletion vectors, and raw fallback.

There is some overlap with #563 in the dependency and reader wiring. I'm happy to rebase on or reuse that foundation to avoid duplication. Please let me know which integration order you would prefer. I'll also review #568 separately.

[JunRuiLee]

Thanks for sharing #568. I've opened #571 as a draft so the scope and implementation are visible.

My work focuses on the existing non-primary-key/global-index full-text path and DataFusion integration, rather than the primary-key full-text and hybrid search path covered by #568. It migrates the legacy in-repository Tantivy implementation to paimon-ftindex-core and covers snapshot index-manifest reads, fast/full/detail modes, deletion vectors, and raw fallback.

There is some overlap with #563 in the dependency and reader wiring. I'm happy to rebase on or reuse that foundation to avoid duplication. Please let me know which integration order you would prefer. I'll also review #568 separately.

Thanks — the scope is clear now, and I think our work is complementary with essentially no functional overlap: you're migrating the append/global-index full-text path (and DataFusion) onto paimon-ftindex-core, and #568 covers the primary-key full-text + hybrid path.

The only real intersection is the shared piece: the paimon-ftindex-core dependency + the FullTextArchiveReader wiring (currently in #563). I'd suggest we land that reader foundation first, as its own small PR (#563), and then both of our stacks — your global-index path and my PK path — rebase onto it so the dependency and reader live in one place. That also lets us settle one shared detail: the pinned tag (I'm on v0.1.0-rc4, you're on v0.1.0-rc5) — we should converge on one and switch to the crates.io version before merging, since a git-only dependency blocks publishing.

Works for me to have #563 go in first; happy to adjust the reader's API if your global path needs anything from it.

[shyjsarah]

Thanks, that ordering works for me.

One API requirement from #571: global-index archives may be stored on remote FileIO and can be large, so the current implementation intentionally avoids reading the entire archive into memory. It adapts range reads from FileRead to paimon_ftindex_core::io::SeekRead with bounded concurrency.

Could #563 make FullTextArchiveReader generic over SeekRead, or provide a from_seek_read constructor while keeping from_input_file as a convenience API? That would let both the global-index and PK paths reuse the same reader foundation without forcing whole-file reads.

I'll keep #571 as a draft and rebase it onto #563 once we agree on this API and the foundation lands.

[JunRuiLee]

Done — I've updated #563 with a generic SeekRead API (from_seek_read constructor + FullTextArchiveReader<R: SeekRead + 'static>). The from_input_file convenience wrapper is preserved for whole-file reads. You can now rebase #571 onto the updated foundation and use AsyncFileSeekRead with it. Let me know if the API needs any adjustment.

Thanks, that ordering works for me.

One API requirement from #571: global-index archives may be stored on remote FileIO and can be large, so the current implementation intentionally avoids reading the entire archive into memory. It adapts range reads from FileRead to paimon_ftindex_core::io::SeekRead with bounded concurrency.

Could #563 make FullTextArchiveReader generic over SeekRead, or provide a from_seek_read constructor while keeping from_input_file as a convenience API? That would let both the global-index and PK paths reuse the same reader foundation without forcing whole-file reads.

I'll keep #571 as a draft and rebase it onto #563 once we agree on this API and the foundation lands.

Done — I've updated #563 with a generic SeekRead API (from_seek_read constructor + FullTextArchiveReader<R: SeekRead + 'static>). The from_input_file convenience wrapper is preserved for whole-file reads. You can now rebase #571 onto the updated foundation and use AsyncFileSeekRead with it. Let me know if the API needs any adjustment.

[shyjsarah]

Thanks for the quick update! I tested #571 on top of the latest #563 and integrated AsyncFileSeekRead with FullTextArchiveReader::from_seek_read. It works well for the global-index path, and all relevant tests and Clippy checks pass. No further API changes are needed from my side. I’ll keep #571 as a draft and rebase it once #563 is merged.