Skip to content

fix(reader): ensure batch readers return zero-offset Arrow arrays#440

Open
lxy-9602 wants to merge 3 commits into
alibaba:mainfrom
lxy-9602:fix-parquet-page-slice
Open

fix(reader): ensure batch readers return zero-offset Arrow arrays#440
lxy-9602 wants to merge 3 commits into
alibaba:mainfrom
lxy-9602:fix-parquet-page-slice

Conversation

@lxy-9602

Copy link
Copy Markdown
Collaborator

Purpose

BatchReader results are transferred through the Arrow C Data Interface and require the returned
array, including nested child arrays, to have zero offsets.

This change fixes several paths that could return non-zero offsets:

  • DataEvolutionFileReader could directly return a sliced singleton array.
  • Parquet page filtering can produce columns with different chunk boundaries.
  • Fully matched Parquet reads can have different chunk boundaries when large binary columns are
    split by Arrow.
  • Arrow TableBatchReader aligns mismatched chunk boundaries by slicing columns, which can leave
    non-zero offsets in returned columns or nested children.

The change:

  • Documents the zero-offset contract on BatchReader.
  • Makes DataEvolutionFileReader concatenate a singleton array when its offset is non-zero.
  • Adds ArrowUtils::NormalizeRecordBatchOffsets for recursively detecting non-zero child offsets.
  • Reuses the utility in both page-filtered and fully matched Parquet read paths.
  • Keeps the normal path lightweight by returning the original RecordBatch when all offsets are
    zero and only copying columns that require normalization.

Tests

Added the following unit test coverage:

  • ArrowUtilsTest.TestNormalizeRecordBatchOffsets
    • top-level column with a non-zero offset
    • nested child with a non-zero offset
    • unchanged data after normalization
    • unaffected columns reuse their original ArrayData
    • already normalized batches use the no-copy fast path
  • PageFilteredRowGroupReaderTest.NormalizesSlicedBatchOffsets
    • verifies multi-batch page-filtered reads through the recursive batch offset checks

API and Format

Documentation

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant