Commit b994a06
Claude Agent
FSST contains: extract scan-routing into a cost-model planner
Replaces the hardcoded `if let Some(...) { ... } else if ...` cascade
inside `FoldedContainsDfa::scan_to_bitbuf` (and the smaller cascades in
`FlatContainsDfa` / `MultiContainsDfa`) with a single `ScanPlanner`
that picks a `ScanPlan` up front and dispatches through one match.
New `dfa/planner.rs` (~430 lines) exposes:
- `ScanPlan` — one variant per legacy cascade branch, plus a reserved
`ShiftOr` slot for Task A. Slot is `cfg_attr`-gated dead_code outside
the test harness.
- `ScanContext` — borrowed inputs (n, all_bytes, ssa codes, bucket
summaries, escape-only flag) the planner reads in O(1).
- `ScanPlanner::plan_folded` / `plan_flat_or_multi` — rules-based
routing that replicates the legacy cascade exactly (locked in by
`test_planner_matches_legacy_cascade` against every fsst_contains
bench needle on every bench corpus).
- `ssa_saturated` and `escape_pair_targets` moved here as the single
source of truth.
- `ArchProfile::detect()` runs CPUID once at `ScanPlanner::new()`; the
arch is cached for the lifetime of the DFA.
- `ScanPlanner::estimated_cost_ns` returns approximate per-call cost.
Calibrated from `DESIGN.md` numbers and benches/fsst_like.rs:
* triple Teddy: AVX-512 4.28 GB/s, AVX2 2.74 GB/s, NEON 2.5, scalar 0.8
* pair Teddy: AVX-512 5.50, AVX2 3.30, NEON 3.0, scalar 1.0
* 1-byte: AVX-512 12.0, AVX2 8.0, NEON 7.0, scalar 2.0
* memmem ~25 GB/s, row-loop ~150 ns/row
Today the cost is diagnostic only (the routing is rules-based); the
constants exist for VORTEX_FSST_PLAN_TRACE and to make later
comparison-based selection mechanical.
`FoldedContainsDfa::scan_to_bitbuf` now extracts each path into a
`run_*` helper (`run_escape_only`, `run_one_byte_saturated`,
`run_triple_teddy`, `run_escape_pair`, `run_pair_teddy`,
`run_one_byte_bitset`, `run_row_loop`) and dispatches via `match
plan { ... }`. The Teddy-trace `VORTEX_FSST_TEDDY_TRACE` output is
preserved verbatim, and a new `VORTEX_FSST_PLAN_TRACE=1` prints the
planner's chosen plan plus inputs and the estimated cost.
`FlatContainsDfa` and `MultiContainsDfa` route through the same
planner (only `EscapeOnly` vs `RowLoop`) so the dispatch surface is
uniform across the three contains DFAs.
Regression guards added:
- `test_planner_matches_legacy_cascade` runs every fsst_contains
bench's underlying call (12 corpus × needle pairs) and asserts
`planner.plan() == legacy_path_for(...)`. Future changes can't
silently re-route traffic.
- 11 unit tests in `planner::tests` cover each routing decision row,
cost-model monotonicity, and `ScanPlan::name` uniqueness.
No algorithmic changes — every existing scan path is invoked under
the same conditions as before, so benches are at parity.
Checks:
- cargo test -p vortex-fsst --lib --features _test-harness: 184 passed
- cargo test -p vortex-fsst --lib: 182 passed
- cargo +nightly fmt --all: clean
- cargo clippy -p vortex-fsst --all-targets --all-features: no new
lints in changed files (pre-existing lints in dfa_compressed/,
anchor_scan.rs:3100+, mod.rs:498, multi_contains.rs:405 untouched).
- cargo bench -p vortex-fsst --bench fsst_like --features _test-harness:
benches compile and `fsst_contains_htt_{cb,urls}` /
`fsst_contains_https_urls` run inside expected timings.
Signed-off-by: Claude Agent <claude-agent@anthropic.com>1 parent f0c8b51 commit b994a06
6 files changed
Lines changed: 1248 additions & 272 deletions
File tree
- encodings/fsst/src/dfa
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
39 | 42 | | |
40 | 43 | | |
41 | 44 | | |
| |||
58 | 61 | | |
59 | 62 | | |
60 | 63 | | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
61 | 68 | | |
62 | 69 | | |
63 | 70 | | |
| |||
118 | 125 | | |
119 | 126 | | |
120 | 127 | | |
| 128 | + | |
121 | 129 | | |
122 | 130 | | |
123 | 131 | | |
| |||
176 | 184 | | |
177 | 185 | | |
178 | 186 | | |
179 | | - | |
180 | | - | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
181 | 198 | | |
182 | | - | |
183 | 199 | | |
184 | 200 | | |
185 | 201 | | |
| |||
0 commit comments