fix(parquet): support mask filtering across skipped pages

[hhhizzz]

Which issue does this PR close?

• Closes [Parquet] Support skipping pages with mask based evaluation #8845.
• Related: [regression] Error with adaptive predicate pushdown: "Invalid offset in sparse column chunk data: 754, no matching page found." #9239, [EPIC] Faster performance for parquet predicate evaluation for non selective filters #7456, [Parquet] Adaptively to pick between RowSelection and Mask filter representation #8846, and Filter pushdown selectivity threshold #9591.

Rationale for this change

Mask-backed row selection decodes every row covered by a mask chunk before applying Arrow's filter kernel. Page pruning can leave sparse in-memory column chunks where complete pages were never fetched. If a mask chunk crosses one of those gaps, decoding requests data that is not present and fails with an invalid sparse-page offset.

Previously, Auto avoided this failure by falling back to selectors when page pruning was detected, while an explicit RowSelectionPolicy::Mask could still fail. This PR makes skipped pages safe for Mask execution, allowing explicit Mask to work and Auto to retain its density-based strategy choice.

What changes are included in this PR?

The reader now computes LoadedRowRanges: absolute row-group intervals whose backing pages are loaded for every Parquet leaf in the current projection.

For each predicate projection and the final output projection, it:

1. Maps the pages selected for each projected leaf to absolute row ranges.
2. Intersects those ranges across the projected leaves.
3. Gives the resulting ranges to Mask execution.
4. Ends each decoded mask chunk at a loaded-range boundary.
5. Uses skip_records() to cross gaps without decoding missing pages.

These ranges are decoder safety boundaries, not output batch boundaries. ParquetRecordBatchReader accumulates multiple safe chunks and their mask bits until it reaches batch_size, then consumes and filters once. Sparse pages therefore do not fragment one logical batch into one batch per loaded range.

When no projected pages were skipped, LoadedRowRanges is omitted and the existing Mask fast path is unchanged. No masks are intersected or rewritten; the original selection mask is still applied by Arrow's filter kernel.

Loaded row range example

Consider two projected columns with different page boundaries and the selection mask 100000000001:

Row partition	[0, 4)	[4, 6)	[6, 8)	[8, 10)	[10, 12)
Selection mask	1000	00	00	00	01
Column A pages	loaded A0	not loaded A1 [4, 8)		loaded A2 [8, 12)
Column B pages	loaded B0 [0, 6)		not loaded B1 [6, 10)		loaded B2
LoadedRowRanges	loaded	unloaded gap			loaded

Column A:        [0, 4) U [8, 12)
Column B:        [0, 6) U [10, 12)
LoadedRowRanges: [0, 4) U [10, 12)

The intersection does not need to match any one column's page boundaries. It is a row-domain guarantee: every row in the result has backing page data in every projected leaf. For example, [0, 4) is a valid subset of Column B's [0, 6) page.

Mask execution and batch behavior

With batch_size = 12, the reader processes the example as follows:

Cursor position	Decoder action	Selected rows accumulated
0	Read [0, 4) with mask 1000	1
4	skip_records(7) to row 11, then read [11, 12) with mask 1	2
End	Consume once and apply the combined mask	one 2-row batch

The unloaded gap is never decoded, while the loaded-range boundary remains invisible to callers as a batch boundary.

Are these changes tested?

Coverage includes:

• page-to-row-range conversion and intersections across different column page boundaries;
• explicit Mask and Auto, with predicate caching enabled and disabled;
• reads with and without an initial row selection;
• multiple predicates with projection-specific loaded ranges;
• sparse loaded ranges coalescing to the requested logical batch size;
• nested List<Struct<...>> projections whose leaves have different page boundaries;
• masks trimmed before the end of their enclosing loaded range.

Validated with:

cargo test -p parquet --lib --features arrow
cargo test -p parquet --test arrow_reader --features async

The current GitHub CI run is green, including Parquet tests, Clippy, MSRV, macOS, Windows, wasm32, and PySpark integration.

Are there any user-facing changes?

There are no public API or source compatibility changes.

Explicit Mask execution no longer fails when page pruning creates sparse column data. Auto no longer falls back to selectors solely because pages were pruned, and sparse loaded ranges no longer fragment logical output batches. These changes can affect execution performance characteristics, but not output semantics.

[hhhizzz]

@alamb 👋Hi, This PR is now ready for review and updated with the latest main.
It fully fixes the sparse-page Mask failure without falling back to selectors. Also tests on my machine. All 1,679 Parquet tests and clippy pass, and SF10 TPC-DS completed without failures or performance regression ✅
When you have time, I’d appreciate a review of the overall approach 🙏😊

[alamb]

Ok, thank ou @hhhizzz -- I will try. It just takes me a qhile to review non trivial PRs like this (not just the code, but also the approach).

[hhhizzz]

Ok, thank ou @hhhizzz -- I will try. It just takes me a qhile to review non trivial PRs like this (not just the code, but also the approach).

Thanks, and no rush at all. Please take whatever time works for you. I completely understand that reviewing the overall approach of a non-trivial change takes significantly more time than just reviewing the code.

I’ve been focusing on this area recently, and I added a few diagrams to the PR description in the hope that they make the design and execution flow easier to understand.
I’d also be very interested in taking on more maintenance responsibility in this area over time, so I’d be happy to receive any feedback on the design, implementation, tests, or how I can contribute more effectively.
I’ll try to stay involved in related community discussions and reviews, and provide code suggestions where useful. Please feel free to tag me on related issues or PRs whenever I can help.
Thanks again!

[hhhizzz]

@alamb 👋 A brief update since your last note: I simplified the implementation
and added the remaining regression coverage for batch shape and nested columns
with different page boundaries.

I also realized that, in earlier iterations, I let the scope grow by pursuing
too many detailed performance improvements at once. AI-assisted tools make it
easy to explore and implement related ideas quickly, but they do not reduce the
reviewer's cognitive load. That scope was therefore asking too much from
reviewers.

I have now narrowed this PR to one behavioral goal: making Mask execution
correct across sparse pages without falling back to selectors. Future
optimizations can be evaluated and reviewed separately in focused follow-up
PRs. This is also how I plan to scope my contributions going forward: establish
the behavioral change first, then optimize incrementally with separate evidence.

The PR has 604 additions and 168 deletions overall, but when #[cfg(test)]
code is counted as tests, the core runtime change is +244/-116 across four
files, or a net increase of 128 lines. Tests are +360/-52 and account for most
of the net growth. I also revised the description around the LoadedRowRanges
invariant and how safe decode chunks are coalesced into one logical batch.
All current CI checks are green ✅

Whenever your schedule allows, I’d really appreciate your review of the
approach 🙏 I’d like to keep building context in Parquet predicate pushdown
and gradually help maintain this area, including reviewing related issues and
PRs and taking follow-up work. Please feel free to tag me wherever I can be
useful. Thank you! 😊