Skip to content

Commit 48199b9

Browse files
Dandandanclaude
andauthored
Use ParquetPushDecoder in ParquetOpener (#20839)
## Which issue does this PR close? - Closes #20841 ## Rationale for this change We want to split IO and CPU to allow for more (NUMA-aware) parallelism and utilizing IO and CPU better. This allows for e.g. more coalescing, prefetching, parallel IO, more parallel / incremental decoding etc. Also this allows doing morsels only on a CPU level and not doing IO multiple times for each morsel. ## What changes are included in this PR? Just refactor `ParquetOpener` to use `ParquetPushDecoder`. I used claude to rewrite it and to keep changes small. ## Are these changes tested? Existing tests. Nothing should change, the arrow-rs code also uses `ParquetPushDecoder`. ## Are there any user-facing changes? --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 9b7cdda commit 48199b9

File tree

1 file changed

+183
-119
lines changed

1 file changed

+183
-119
lines changed

0 commit comments

Comments
 (0)