fsst opt DO NOT MERGE#6996

Closed

joseph-isaacs wants to merge 12 commits into

claude/optimize-fsst-compression-KgJdu

Commits on Mar 16, 2026

Commits on Mar 17, 2026

Major FSST compression improvement: reduce first-byte bias + add stabilization pass
claude
committed
Add diverse benchmark dataset generators (network, analytics, text)
claude
committed
feat(fsst): add diverse benchmark datasets for FSST compression testing
claude
committed
feat(fsst): expand benchmark to 25 diverse datasets
claude
committed
feat(fsst): add decompression throughput benchmarks
claude
committed
Make benchmark datasets much more diverse and realistic
claude
committed
feat(fsst): improve training convergence with stratified sampling and more generations
claude
committed
perf(fsst): hoist PHT lookup for instruction-level parallelism in compress_word
claude
committed
feat(fsst): use smoother generation ramp for better training convergence
claude
committed
feat(fsst): remove min_count pre-filter in favor of gain-based selection
claude
committed