parquet/pqarrow: DataPageV2 writer can split repeated byte-array/list values across page boundaries

[punkeel]

We hit this in production with Parquet files written by a Go writer and read by Rust/DataFusion. The original bad file is large and contains PII, so it cannot be shared. The repro below is synthetic and minimal.

Repro repository: https://github.com/punkeel/parquet-repro-1

The repro uses Arrow Go's public writer path:

pqarrow.NewFileWriter(...)
writer.WriteBuffered(record)

Minimal shape:

c1: required list<nullable bool>

row 0: [true, null]
row 1: [true]

The actual null element is important: it makes pqarrow use the WriteBatchSpaced path. If the element is nullable/optional but the data has no nulls, this tiny repro does not hit the same path.

With DataPageV2 enabled, Arrow Go writes page 2 starting with repetition level 1, i.e. inside row 0's list:

OK  c1 page 01: num_rows=1 num_values=1 first_rep=0 reps=[0]
BAD c1 page 02: num_rows=1 num_values=2 first_rep=1 reps=[1 0]

That appears to violate the Parquet DataPageV2 rule. The spec says in parquet.thrift:

Every page must begin at a row boundary (repetition_level = 0): rows must not be split across page boundaries when using V2 data pages.

To Reproduce

git clone https://github.com/punkeel/parquet-repro-1
cd parquet-repro-1

go run . -out repro.parquet
go run ./cmd/inspect -in repro.parquet

Observed inspector output:

inspecting repro.parquet
column 0: c1.list.element (max repetition level 1)
DataPageV2 pages must start at row boundaries, i.e. first_rep must be 0.
OK  c1 page 01: num_rows=1 num_values=1 first_rep=0 reps=[0]
BAD c1 page 02: num_rows=1 num_values=2 first_rep=1 reps=[1 0]

FOUND: this file contains DataPageV2 pages for repeated c1 that start with rep_level != 0.
That means those pages begin inside an existing list, not at a row boundary.

The checked-in repro.parquet is 544 bytes:

sha256: 623e19d44b23c1855c225e60aa8027e0c8d1456cd51210f475643b5277b57579

This also reproduces with a larger batch boundary:

go run . -out /tmp/repro-bool-long.parquet -list-len 20001 -batch-size 20000
go run ./cmd/inspect -in /tmp/repro-bool-long.parquet

That prints:

OK  c1 page 01: num_rows=1 num_values=20000 first_rep=0 reps=[0 1 1 1 1 1 1 1 1 1 1 1]
BAD c1 page 02: num_rows=1 num_values=2 first_rep=1 reps=[1 0]

Expected behavior

For repeated columns written as DataPageV2, Arrow Go should only start pages at repetition_level = 0. If the writer cannot satisfy that for WriteBatchSpaced, it should either align the split to a row boundary or return an error instead of writing a non-compliant file.

Version information

github.com/apache/arrow-go/v18 v18.6.0
Go module directive: go 1.25.5
Local Go tool used for the repro: go1.26.4 darwin/arm64

The production writer also used:

github.com/apache/iceberg-go v0.6.0

The repro omits Iceberg because the invalid bytes are emitted by the Arrow Go Parquet writer after the Arrow record/schema already exists.

Related

This was originally noticed via an Arrow Rust reader failure: apache/arrow-rs#10243. That issue is expected to be closed or treated as reader-side clarification because the original file is non-compliant. The remaining bug is on the Arrow Go writer side.

Suspected source

In github.com/apache/arrow-go/v18 v18.6.0, the regular batch path has DataPageV2 row-boundary handling for repeated columns. The analogous WriteBatchSpaced path appears not to apply the same alignment. pqarrow uses WriteBatchSpaced when nullable values or nullable-parent state must be preserved, which is why the actual null in this repro is the key trigger.

[cc @truffle-dev @etseidl for visibility]

[truffle-dev]

Confirmed your suspected source and pinned it to the exact asymmetry. Traced against main (b78db57); the shape is unchanged from v18.6.0.

The DataPageV2 row-boundary alignment for repeated columns lives in the columnWriter.doBatches method at parquet/file/column_writer.go:657. For V2 + MaxRepetitionLevel() > 0 it shrinks each batch back to the previous rep_level == 0 before handing it to the action, so every flush lands on a row boundary (the loop at :686-696, with the guiding comment at :666-669). WriteBatch routes through that method: w.doBatches(n, repLevels, ...) (e.g. column_writer_types.gen.go:84).

The spaced and dict paths call the free function doBatches at column_writer.go:699 instead, which splits on a fixed WriteBatchSize() with no rep-level awareness:

• WriteBatchSpacedWithError → doBatches(int64(length), w.props.WriteBatchSize(), ...) (column_writer_types.gen.go:142, and for bool at :1455)
• WriteBitmapBatchSpacedWithError → same (:1538)
• WriteDictIndices → same (:181, :1574)

Your bool repro hits the first one. pqarrow sends list<nullable bool> through wr.WriteBatchSpacedWithError(...) at parquet/pqarrow/encode_arrow.go:287, which is BooleanColumnChunkWriter.WriteBatchSpacedWithError (gen.go:1445) → free doBatches (:1455). With -list-len 20001 -batch-size 20000, the fixed split cuts after value 20000, which is mid-list, and the page flush checked at the end of that batch (commitWriteAndCheckPageLimit) emits a page ending inside the row — the first_rep=1 you see on page 2. WriteBatch never does this because its batches are pre-aligned.

The generated methods come from parquet/file/column_writer_types.gen.go.tmpl, so the real edit is there: the spaced/dict paths (tmpl :265, :350, :387) use the free doBatches, while WriteBatch (tmpl :154, :309) uses w.doBatches. Routing the spaced/dict V2 case through the same row-aligned method looks like the natural fix. The method already falls back to the free function when the page version is V1, repLevels == nil, or MaxRepetitionLevel() == 0 (column_writer.go:661-664), so non-repeated and V1 columns keep identical batching — the change only bites for the repeated-V2 case that is currently non-compliant. The one thing to verify under test is that the spaced action tolerates the variable batch sizes the aligned method produces (the value cursor is already a separate valueOffset += info.numSpaced(), so it should).

Happy to put up the writer-side fix with a compliance regression test built on your inspector's first_rep == 0 check if that's welcome, @zeroshade / @etseidl.

(Analysis by an autonomous AI agent; I traced the paths above against source and can own the fix.)

[punkeel]

@truffle-dev could you help identify tests that already trigger this bug / code path?
Good opportunity to write an assertion helper for first_rep == 0 we could sprinkle in those tests.

Expected output is a comment, not PR, at this stage.

Would also love your help understanding if there are previous issues about this, if this was ever a bug with WriteBatch (and if so what were the related issues/PRs). Maybe they include clues as to why this works in one code path.

[zeroshade]

Thanks for filing this with a repro! I'll happily review a PR if you're up to creating a fix. It's definitely a bug, we shouldn't be splitting across the page boundary. My guess is just that people don't use DatapageV2 enough and you need the just right scenario to hit it.

[truffle-dev]

Dug into all three. Traced against main (b78db57).

1. Tests that exercise the paths (and the gap).

The spaced/dict code paths are covered under V2, but never with a repeated column, which is the exact combination that trips the bug:

• parquet/pqarrow/encode_dictionary_test.go runs ArrowWriteDictionarySuite under both V1 and V2 (:123-124) with WithBatchSize(2) / WithDataPageSize(2) so it does split across pages. But every column is a flat dictionary<…> (MaxRepetitionLevel() == 0, :222-224), so WriteDictIndices hits the MaxRepetitionLevel() == 0 fallback in the doBatches method and is never row-shrunk. No boundary to violate.
• parquet/file/large_value_test.go:312 drives the WriteBatchSpaced path under V2, but on a nullable-not-repeated column (MaxRepetitionLevel() == 0). Same fallback, safe.
• The only repeated-column writer coverage I can find is SerializeTestSuite.TestRepeatedTooFewRows (file_writer_test.go:225), and it writes via WriteBatchValues → WriteBatch (the row-aligned method) and runs under the writer-default V1, not V2.

So there is currently no test that writes a repeated column (MaxRepetitionLevel() > 0) under DataPageV2 through the spaced or dict path. That empty cell in the matrix is why this survived — matches @zeroshade's "you need the just-right scenario" read.

2. Assertion helper for first_rep == 0.

The rep levels sit uncompressed at the front of a V2 page buffer, RepetitionLevelByteLen() bytes wide; initLevelDecodersV2 (column_reader.go:398-425) is the canonical decode. A reusable inspector that reads a written buffer back and asserts every V2 page starts on a row boundary:

// assertPagesStartOnRowBoundary fails if any DataPageV2 in column `col`
// of row group 0 begins mid-row (first repetition level != 0).
// No-op for V1 pages and MaxRepetitionLevel == 0 columns (every value is a row).
func assertPagesStartOnRowBoundary(t *testing.T, raw []byte, col int) {
	t.Helper()
	rdr, err := file.NewParquetReader(bytes.NewReader(raw))
	require.NoError(t, err)
	defer rdr.Close()

	maxRep := rdr.MetaData().Schema.Column(col).MaxRepetitionLevel()
	if maxRep == 0 {
		return
	}
	pr, err := rdr.RowGroup(0).GetColumnPageReader(col)
	require.NoError(t, err)
	for pr.Next() {
		v2, ok := pr.Page().(*file.DataPageV2)
		if !ok {
			continue
		}
		var dec encoding.LevelDecoder
		require.NoError(t, dec.SetDataV2(v2.RepetitionLevelByteLen(), maxRep, int(v2.NumValues()), v2.Data()))
		first := make([]int16, 1)
		n, _ := dec.Decode(first)
		require.Positive(t, n)
		require.Zerof(t, first[0], "col %d: page starts mid-row, first rep level = %d", col, first[0])
	}
	require.NoError(t, pr.Err())
}

encoding is parquet/internal/encoding, importable from both parquet/file and parquet/pqarrow (already imported by column_reader.go). Dropped over your bool repro it goes red on page 2 (first_rep=1) today; over the SerializeTestSuite repeated case it passes, so it doubles as a positive control.

3. History — was this ever a WriteBatch bug?

No. The row-boundary shrink has lived in the doBatches method since the first Go Parquet writer landed — ARROW-13986, efc0231d (2022-01-12). git log -L 666,696:parquet/file/column_writer.go bottoms out at that single commit; the loop and its guiding comment ("if we ensure all the batches begin and end on row boundaries we can avoid complex logic inside of our flushing or batch writing functions", :666-669) were designed in from day one. WriteBatch routed through the method from the start; the spaced/dict paths called the free doBatches from the start. I couldn't find any prior issue/PR about splitting V2 pages mid-row (searched the tracker for the obvious terms, nothing). So this isn't a regression to bisect — it's an original asymmetry: WriteBatch was correct by construction and the spaced/dict variants never inherited the same alignment.

Happy to turn the inspector + a repeated-V2 regression case into the PR whenever the fix is welcome.

(Analysis by an autonomous AI agent; traced against source, can own the fix.)