Commit c32b58d
feat: add Parquet merge policy for compaction (Phase 2)
Adds a constant write amplification merge policy for Parquet splits,
adapted from the existing ConstWriteAmplificationMergePolicy but using
byte size instead of document count as the primary size metric. This is
Phase 2 of the Parquet compaction project — the decision layer that
determines which splits to merge within each compaction scope.
Key components:
- ParquetMergePolicy trait mirroring the MergePolicy interface
- CompactionScope grouping by (index_uid, sort_fields, window_start)
- ConstWriteAmplificationParquetMergePolicy with bounded write amp
- finalize_operations() for cold window compaction
- 33 tests: unit, proptest (MC-CONSERVE/LEVEL/WA/IDEMPOTENT), simulation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent 3fc479f commit c32b58d
6 files changed
Lines changed: 1269 additions & 0 deletions
File tree
- quickwit
- quickwit-parquet-engine
- src/merge
- policy
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
30 | 30 | | |
31 | 31 | | |
32 | 32 | | |
| 33 | + | |
33 | 34 | | |
34 | 35 | | |
35 | 36 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
| 24 | + | |
24 | 25 | | |
25 | 26 | | |
26 | 27 | | |
| |||
0 commit comments