feat: fulltext2 — a position-free BM25 full-text retrieval index

[cpegeric]

What type of PR is this?

• API-change
• BUG
• Improvement
• Documentation
• Feature
• Test and CI
• Code Refactoring

Which issue(s) this PR fixes:

issue ##25541

What this PR does / why we need it:

Summary

Adds fulltext2, a new first-class full-text index type built around a WAND / block-max ranked-retrieval engine, and removes the standalone bm25 index (its ranked-retrieval capability is now subsumed by fulltext2's POSITION_FREE mode). fulltext2 is a distinct algo from classic fulltext — not a variant — so it carries its own contract, on-disk format, plugin, CDC path, and query surface.

Classic fulltext keeps positional term data for exact phrase/boolean/NL semantics. fulltext2 adds a position-free build option that drops positions (keeping the FST term dictionary), yielding a much smaller, faster, ranked bag-of-words index — while still supporting a positional build for phrase/boolean parity when you want it.

Gated behind the experimental_fulltext2_index session variable.

Motivation

The earlier "WAND retrieval" experiment proved that ranked BM25 retrieval does not fit as a fulltext parser-mode variant (boolean/phrase/NL need term positions the position-free binary index cannot serve). Rather than graft ranked retrieval onto classic fulltext or maintain a separate bm25 index type, we make it its own index type with a clean contract, and fold the old bm25 engine into it.

SQL surface

DDL — a distinct index type via the USING/index-type seam, with a configurable tokenizer and a position-free option:

SET experimental_fulltext2_index = 1;

-- positional (phrase / boolean / NL capable); parser defaults to ngram
CREATE FULLTEXT2 INDEX idx ON t(body);
CREATE FULLTEXT2 INDEX idx ON t(body) WITH PARSER gojieba;

-- position-free, ranked bag-of-words — requires WITH PARSER gojieba
CREATE FULLTEXT2 INDEX idx ON t(body) WITH PARSER gojieba POSITION_FREE=TRUE;

Full CREATE option grammar — CREATE FULLTEXT2 INDEX name ON t(cols) <options>, where <options> may combine, in any order:

Option	Values	Meaning
WITH PARSER	ngram (default) · gojieba · json · json_value	tokenizer
POSITION_FREE	TRUE · FALSE (default)	drop positions → ranked bag-of-words only (requires gojieba)
AUTO_UPDATE	TRUE · FALSE (default)	enable the idxcron scheduled auto-maintenance for this index
DAY / HOUR / SECOND	integer	the auto-maintenance cadence (combined; e.g. DAY=1 = compact daily)
MAX_INDEX_CAPACITY	integer	per-segment doc capacity (drives tiered-merge fullness)
ASYNC / FORCE_SYNC	flag	build the initial index asynchronously (CDC) vs synchronously

-- gojieba, position-free, with daily auto-maintenance:
CREATE FULLTEXT2 INDEX idx ON t(body)
  WITH PARSER gojieba POSITION_FREE=TRUE AUTO_UPDATE=TRUE DAY=1;

-- capacity-bounded segments + hourly auto-maintenance:
CREATE FULLTEXT2 INDEX idx ON t(body)
  MAX_INDEX_CAPACITY=100000 AUTO_UPDATE=TRUE HOUR=6;

AUTO_UPDATE=TRUE + DAY/HOUR/SECOND are stored in the index's algo-params and read by the idxcron executor, which schedules a periodic MERGE/REBUILD (see Maintenance below). POSITION_FREE=TRUE is only valid with WITH PARSER gojieba; a bare POSITION_FREE=TRUE (defaults to ngram) and WITH PARSER ngram POSITION_FREE=TRUE are both rejected at DDL — ngram/json emit overlapping tokens meaningless without positions, so they can only build a positional index.

Query — ranked retrieval mode alongside the classic modes:

-- ranked bag-of-words top-K (works on a POSITION_FREE index)
SELECT * FROM t WHERE MATCH(body) AGAINST('query terms' IN BM25 MODE) LIMIT 10;

-- NL / BOOLEAN / phrase still available on a *positional* fulltext2 index
SELECT * FROM t WHERE MATCH(body) AGAINST('+cat -dog' IN BOOLEAN MODE);

Maintenance — ALTER … REINDEX (MERGE / REBUILD / retune)

fulltext2 maintains a tiered index: a tag=0 base (built from source) plus a tag=1 CDC tail of incremental inserts/deletes. ALTER TABLE … ALTER REINDEX <name> FULLTEXT2 <option> drives compaction and retuning:

-- MERGE: fold the tag=1 CDC tail into the tag=0 base (in place, keeps the current
-- position-free/positional mode). Cheap incremental compaction.
ALTER TABLE t ALTER REINDEX ft FULLTEXT2 MERGE;

-- REBUILD (no option): drop the tail AND rebuild the base entirely from source.
-- Use when the dead-doc ratio is high or to recover a partially-materialized index.
ALTER TABLE t ALTER REINDEX ft FULLTEXT2;

-- Retune POSITION_FREE both ways (a REBUILD from source under the new mode).
-- Tri-state parse distinguishes =FALSE from "unset", so it toggles deterministically:
ALTER TABLE t ALTER REINDEX ft FULLTEXT2 POSITION_FREE=FALSE;  -- → positional (phrase/boolean work)
ALTER TABLE t ALTER REINDEX ft FULLTEXT2 POSITION_FREE=TRUE;   -- → position-free (IN BM25 MODE only)

-- FORCE_SYNC: run the reindex synchronously instead of via async CDC.
ALTER TABLE t ALTER REINDEX ft FULLTEXT2 MERGE FORCE_SYNC;

Scheduled (idxcron) maintenance. When the index was created AUTO_UPDATE=TRUE with a DAY/HOUR/SECOND cadence, the idxcron executor runs the same reindex action automatically on that interval, choosing MERGE vs REBUILD by the dead-doc ratio (a tail past the dead-doc threshold rebuilds; otherwise it merges). No manual ALTER … REINDEX is needed in that mode.

Architecture

• Engine (pkg/fulltext2/): segment build sink, delta+varint block-postings format, FST term dictionary, block-max WAND top-k, dense boolean evaluator, block-by-block phrase intersection, multi-segment index with global corpus stats + per-segment liveness/deletes, mmap'd base segments, CDC tail builder, and MERGE compaction.
• fulltext2 AlgoPlugin (pkg/fulltext2/plugin/{plan,compile,runtime,idxcron,iscp}): five hook sets, all with var _ Hooks compile-time assertions; routed through the indexplugin.Get(algo) registry (no new per-algo switch in plan/compile). Blank-imported in indexplugin/all and indexplugin/iscp.
• SQL-plan integration: apply_indices_fulltext2.go / apply_indices_match.go route MATCH … AGAINST to the fulltext2_search TVF with ORDER BY score DESC + LIMIT pushdown; fulltext2_compact.go drives compaction.
• TVFs: fulltext2_{create,search,compact}.
• CDC + idxcron: async incremental maintenance (CREATE→sync→query→REINDEX→DROP), scheduled MERGE/REBUILD.

Tokenizers

ngram (default), gojieba (CJK word segmentation), json (JSON values flattened to text, then tokenized as ngram), and json_value (each leaf value indexed as one whole atomic token — classic-fulltext json_value parity), with tokenization parity between build and query. POSITION_FREE is supported only with gojieba — ngram and json produce overlapping/redundant tokens that can't be reconstructed without positions, so they always build a positional index (and are rejected if POSITION_FREE=TRUE is requested).

Performance & footprint

Position-free build is materially smaller and faster than both classic fulltext and the old bm25 index. Residency wins: block-compressed mmap postings + lazy directory (decode a term's block-max entry on lookup), lazy positions section, dense O(N) boolean evaluator, insertion-sort WAND pivot loop (no per-iteration alloc), and a cached cursor curDoc with inline in-block skip.

Safety / guards

• Reject a FULLTEXT + FULLTEXT2 index on the same column set at DDL.
• Reject NL/BOOLEAN/phrase modes on a POSITION_FREE index with a clear message.
• OOM defenses: bounded top-k heap (uncapped LIMIT no longer OOMs), fail-fast memory guard on CDC tail load, off-heap posting decode, corrupt-frame / uvarint-overflow guards on read.
• Injective per-row keys for temporal/decimal PKs (no data loss); CDC hands uuid/decimal/datetime as typed strings (no types.EncodeValue panics).
• ALTER ADD FULLTEXT no longer locks its not-yet-created hidden table.
• Experimental-flag gated; restore/clone supported.

Testing

New BVT suites under test/distributed/cases/fulltext2/ and pessimistic_transaction/fulltext2/: functional sync, POSITION_FREE + IN BM25 MODE, parser coverage (ngram/gojieba/json_value), membership prefilter, ORDER-BY/LIMIT pushdown, FULLTEXT+FULLTEXT2 conflict rejection, async CDC convergence, MERGE/REBUILD, restore/clone, and bugfix regressions. Extensive engine unit tests in pkg/fulltext2/ (boolean parity, phrase-exact, CJK, multi-segment, mmap, serialize round-trip).

Compatibility

New opt-in index type behind an experimental flag; no change to classic fulltext behavior or on-disk format. Fulltext2 reports as FULLTEXT in information_schema / SHOW INDEX (correct MySQL taxonomy).

[fengttt]

WHY does this PR has to have all those cuvs code? Can we have a CLEAN PR that a human can read? Please break the PR to reasonable, self contained works.