feat(parquet): support configurable bitmap row-range refining strategies (coalesce and trim)

[zhf999]

feat(parquet): support configurable bitmap row-range refining strategies (coalesce and trim)

Purpose

Previously, when a bitmap was pushed down for page-level filtering, the only strategy was to trim page start/end rows based on the bitmap and then apply intersection across all target columns (FilterPagesByBitmap). While precise, this approach can produce many small disjoint ranges when the bitmap has scattered set bits, leading to excessive range count and degraded I/O efficiency.

This PR introduces two configurable strategies for refining row ranges from a pushed-down bitmap:

1. Coalesce strategy (default): Converts bitmap set bits into contiguous row ranges per row group via BitmapToContiguousRanges, then merges nearby ranges whose inter-range gap is ≤ coalesce-hole-size-limit (default 32) via CoalesceNearbyRanges. This reduces the number of disjoint ranges while still skipping large holes, improving I/O efficiency by trading a small amount of over-read for fewer range seeks.

2. Trim strategy (existing behavior, renamed): The previous FilterPagesByBitmap approach is renamed to RefineRowRangesByTrimming. It trims page start/end rows based on the bitmap and applies intersection among all target columns, providing the most precise range selection without over-reading.

The strategy is selected at runtime via the new option parquet.read.bitmap.row-range-refining-strategy (values: "coalesce" or "trim"; default: "coalesce").

Tests

UT — PageFilteredRowGroupReaderTest (src/paimon/format/parquet/page_filtered_row_group_reader_test.cpp):

• BitmapCoalesceTest (replaces ScatteredBitmapTest) — verifies that the coalesce strategy merges ranges with gaps ≤ 32, producing 143 rows from a bitmap of 111 set bits across 4 ranges.
• BitmapCoalesceSmallHoleSizeTest (new) — verifies that a small coalesce-hole-size-limit (5) prevents merging ranges with larger gaps, producing exactly 111 rows (no holes filled).
• BitmapTrimStrategyTest (new) — verifies the trim strategy produces 84 rows by trimming page boundaries without coalescing.
• BitmapWithPageFilteredOptionDisabled — updated to use the new options-based test helper signature.

Test helper ReadWithPredicateAndBitmapImpl was refactored: the bool enable_page_level_filter parameter is replaced with const std::map<std::string, std::string> options, allowing test cases to pass arbitrary reader options (strategy, hole size limit, page index filter toggle, etc.).

Datasets

Multiple performance testing were conducted on 2 dataset.

Dataset	File size	num_rows	num_columns	num_rowgroups	page length (in rows)
Dataset#1	6.7GB	2M	1,176	39	1024
Dataset#2	3GB	11M	4	23	1024

Test Methodology

To validate bitmap behavior, we organize tests into six groups based on bitmap distribution patterns. Each group contains five distinct datasets, and the final result reported is the average of these five runs.

Test Cases

1. Matching records are contiguous and only one row is queried.
2. Matching records are contiguous and 10,000 rows are queried.
3. Matching records are scattered in 5 segments, each segment containing 1 row.
4. Matching records are scattered in 5 segments, each segment containing 100 rows.
5. Matching records are scattered in 100 segments, each segment containing 1 row.
6. Skip 100 rows, read 100 rows, skip 100 rows, read 100 rows util EOF.
7. Skip 10000 rows, read 10000 rows, skip 10000 rows, read 10000 rows util EOF.
8. Scattered bitmap, skip 1 row, read 1 row, skip 1 row, read 1 row until EOF (i.e. 1010101010...).
9. Scattered bitmap, skip 32 rows, read 1 row, skip 32 rows, skip 1 row until EOF.
10. Scattered bitmap, skip 99 rows, read 1 row, skip 99 rows, skip 1 row until EOF.
11. Scattered bitmap, skip 9999 rows , read 1 row, skip 9999 rows, skip 1 row until EOF.

Test Results

On dataset1

Branch	Case 1	Case 2	Case3	Case 4	Case 5	Case 6	Case 7	Case 8	Case 9	Case 10	Case 11	Case 12
Without Bitmap Pushdown	2.27s	2.19s	8.74s	8.75s	105.58s	126.49s	125.34s	126.08s	124.85s	125.96s	126.00s
Trim (Branch Main)	0.33s	0.90s	1.32s	1.35s	15.78s	124.50s	74.06	128.16s	126.41s	119.94s	20.81s
Coalesce (New Feature)	0.33s	0.92s	1.32s	1.34s	16.32s	102.59s	78.81s	126.91s	82.48s	68.42s	20.52s

On dataset2

Branch	Case 1	Case 2	Case3	Case 4	Case 5	Case 6	Case 7	Case 8	Case 9	Case 10	Case 11	Case 12
Without Bitmap Pushdown	1.30s	1.26s	2.77s	2.76s	30.48s	31.80s	32.20s	31.27s	32.16s	31.28s	31.28s
Trim (Branch Main)	8ms	39ms	25ms	26ms	0.46s	35.48s	18.18s	35.18s	33.96s	32.81s	1.99s
Coalesce	8ms	38ms	26ms	26ms	0.48s	28.79s	18.12s	33.83s	23.66s	22.86s	2.00s

Perf tests on 2 datasets shows the Coalesce strategy has better performance on Case 7, Case 8, Case 10 and Case 11.

API and Format

• Internal API change (header): In parquet_file_batch_reader.h:
  • FilterPagesByBitmap renamed to RefineRowRangesByTrimming.
  • FilterRowGroupPagesByBitmap renamed to TrimRowGroupPageRanges.
  • New methods: RefineRowRangesByCoalescing, BitmapToContiguousRanges, CoalesceNearbyRanges.
• New configuration options in parquet_format_defs.h:
  • PARQUET_READ_BITMAP_ROW_RANGE_REFINING_STRATEGY = "parquet.read.bitmap.row-range-refining-strategy" (default: "coalesce").
  • PARQUET_READ_ROW_RANGES_COALESCE_HOLE_SIZE_LIMIT = "parquet.read.bitmap.coalesce-hole-size-limit" (default: 32).
• No changes to Paimon's public API in include/.
• No storage format or protocol changes.
• No Arrow dependency patch changes.

Documentation

No new user-facing feature documentation needed — this is an internal performance optimization configurable via reader options.

Generative AI tooling

GLM 5.2.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[lxy-9602]

+1

[lxy-9602]

+1