feat(fulltext): read global indexes via paimon-ftindex-core

[shyjsarah]

Purpose

Migrate the existing append/data-evolution global full-text search path from the legacy in-repository Tantivy implementation to the shared paimon-ftindex-core engine.

This lets Rust and DataFusion read the current full-text index archives produced by Java Paimon and PyPaimon while preserving the existing full_text_search(table_name, column_name, query_text, limit) API.

This PR intentionally focuses on the non-primary-key/global-index path. The primary-key full-text and hybrid search work is covered separately by #568.

Brief change log

• Replace the legacy Tantivy archive reader and writer modules with an adapter over paimon-ftindex-core.
• Read full-text entries from snapshot index manifests through a bounded-concurrency SeekRead adapter.
• Support plain-text queries, the Paimon JSON query DSL, Roaring include filters, shard-local/global row-ID translation, and global Top-K merging.
• Align fast, full, and detail search modes with Java behavior, including raw fallback for uncovered row ranges.
• Filter deleted rows through data-evolution deletion vectors and keep raw fallback scoring in a single BM25 corpus.
• Add core and DataFusion coverage for native index reads, row-range filtering, raw fallback, query normalization, authorization, and branch table queries.

Testing

• cargo check -p paimon --features fulltext
• cargo test -p paimon --features fulltext full_text -- --nocapture
• cargo check -p paimon-datafusion --features fulltext
• cargo test -p paimon-datafusion --features fulltext fulltext_tests -- --nocapture
• cargo clippy -p paimon --lib --tests --features fulltext -- -D warnings
• cargo clippy -p paimon-datafusion --lib --tests --features fulltext -- -D warnings

API and format

• No public API or table data-file format changes.
• The SQL table-function signature remains unchanged.
• The fulltext feature now reads the current Paimon full-text archive format. Compatibility with the legacy paimon-rust tantivy-fulltext archive is intentionally removed.

Draft status and coordination

paimon-ftindex-core is temporarily pinned to the public v0.1.0-rc5 Git tag. The dependency will be changed to a crates.io version before this PR is marked ready, because a git-only dependency prevents publishing paimon.

This work overlaps with #563 in dependency and reader wiring. It is being opened early so we can agree on the integration order and reuse or rebase on the reader foundation rather than duplicate it. It complements, but does not implement, the primary-key full-text and hybrid search scope in #568.