arrow-ipc writer does not comply with spec for empty variable-size arrays

[atwam]

Describe the bug

Arrow's spec for variable-size binary layout (used for binary or Utf8 arrays) states:

The offsets buffer contains length + 1 signed integers (either 32-bit or 64-bit, depending on the data type), which encode the start position of each slot in the data buffer.

This means that an empty array should have a length-1 offsets buffer containing a single 0.
Instead, serializing an empty Utf8/Byte array creates an empty offset buffer (in get_byte_array_buffers).

This means that an empty (no-rows) IPC file created by arrow-rs will not be spec-compliant, and may not be readable by some other implementations (for example polars/arrow2).

Note that this behavior applies to Utf8/Binary arrays, but also List, LargeList, and by extension Map (anything that uses get_byte_array_buffers or get_list_array_buffers)

To Reproduce

• Create a RecordBatch with a single Utf8 (empty) array, serialize to IPC.
• Try to read that IPC file with a different arrow library. For example, polars will fail reading the serialized IPC file.

Expected behavior
For variable size layouts, IPC writer should output a length 1 offsets buffer containing a single [0].

Additional context

Current version works when doing round-trip because get_offsets_from_buffer fills in an empty offsets for a [0] array.

Note that arrow-cpp seems to take the same approach, and similarly will fill-in an empty offsets buffer with [0]. While this means we can round-trip with arrow-cpp and pyarrow, we don't comply with the spec and can cause issues with other implementations.

[atwam]

I had a closer look at the arrow-cpp implementation for this. In GetZeroBasedValueOffsets, we don't canonicalize empty variable-size offsets on write, we just reuse whatever the in-memory array already has. So if we already have a canonical one-element buffer, we keep it around. If we have a null or zero-length offsets buffer, c++ will preserve that on IPC write.

Strikingly, C++ explicitely exercises this permissive case , including on validation. Now the question is whether the spec should be updated (and then other arrow libraries such as polars/arrow2 will have to change), or whether we stick with the spec and output strictly compliant files.

[alamb]

Strikingly, C++ explicitely exercises this permissive case , including on validation. Now the question is whether the spec should be updated (and then other arrow libraries such as polars/arrow2 will have to change), or whether we stick with the spec and output strictly compliant files.

I am not sure I agree with your interpretation of the spec that requires IPC files to have non zero length buffers for these types

The spec you cite is the arrow in memory format, not the IPC format

[atwam]

Right, the spec I cited is the in-memory format. But the IPC spec:

The data header contains the following:
The length and null count for each flattened field in the record batch
The memory offset and length of each constituent Buffer in the record batch’s body

GIven the in-memory format says we have a length-1 offsets buffer, I would expect that to exist in the record batch's body?

I have not seen anything in the IPC spec that states that some offset buffers might be omitted for an empty array. If that's just an omission, and the official arrow implementation serves as a reference for this edge-case, then I think the spec should explicitly state that it is optional (and I'll get a PR opened on arrow2 crate).

[alamb]

Strikingly, C++ explicitely exercises this permissive case , including on validation. Now the question is whether the spec should be updated (and then other arrow libraries such as polars/arrow2 will have to change), or whether we stick with the spec and output strictly compliant files.

This looks like it came from

• GH-35344: [C++][Format] Implementation of the LIST_VIEW and LARGE_LIST_VIEW array formats arrow#35345

which has more context. This is the mysterious ARROW-554 which doesn't seem to be very helpful honestly

• [C++] Implement functions to conform unequal dictionaries amongst multiple Arrow arrays arrow#16191

So i agree this is not explicitly allowed in the spec

However, I think following the cpp implementation makes sense in general. Does the cpp implementation write such files? Or does it write files with an offset buffer of size 1 as you are proposing?

Perhaps we should start a discussion on the arrow mailing list as the next step

[atwam]

I think the important distinction is between what C++ accepts and what C++ normally produces.

My reading is:

• The spec still describes variable-width offsets as having length length + 1, so an empty offsets buffer for a zero-length Binary/List array is not the canonical layout and does not appear to be explicitly allowed by the spec.
• C++ is deliberately permissive on the read/validation side for length == 0 arrays. That is not accidental, and was added intentionally in the PR you reference
• However, the normal C++ builder path is still canonical. BinaryBuilder / ListBuilder start with no offsets and append the trailing 0 during Finish, so builder-produced empty arrays end up with an offsets buffer of size 1 containing 0.
• The C++ IPC writer does not normalize zero-length arrays into canonical form. It preserves whatever in-memory representation the array already has. So if someone directly constructs a zero-length array with empty/null offsets, C++ IPC can write such a file too.
• So “follow C++” can mean two different things:
  1. follow C++’s permissive reader/validator behavior and accept empty offsets on input, or
  2. follow C++’s normal builder-produced output and write canonical [0] offsets.

However, I think there is a stronger argument than just “this is non-canonical”.

On the C++ side, IPC read preserves a zero-byte offsets buffer for length-0 binary/list arrays, and validation explicitly allows that representation for compatibility. However, several downstream APIs still assume canonical length + 1 offsets and will dereference the missing sentinel.

Concrete examples:

• ListArray::Flatten() reads value_offset(0) and value_offset(length) even for an empty list array.
• list_parent_indices reads offsets[length] - offsets[0] for list arrays.
• CSV writing for string arrays calls value_offset(0) without a length-0 guard.
• The integration JSON writer exports length + 1 offsets unconditionally.
• ListArray::offsets() can box the empty underlying offsets buffer as an offsets array of length 1, which is internally inconsistent.

Reproduction confirms out-of-bounds read in C++ when reading Rust-written IPC files: A minimal Rust program writing an empty ListArray via FileWriter (on current main) produces an IPC file with a zero-byte offsets buffer. When read back through PyArrow (C++ underneath), ListArray::offsets() returns a 1-element Int32Array containing garbage (e.g. -189153672) — this is an uninitialized/out-of-bounds memory read, because C++ BoxOffsets() unconditionally constructs an array of length + 1 entries from whatever buffer is present, even when that buffer is empty.

So while C++ is permissive enough to accept these arrays, and many of its operations will explicitely check for length == 0 (bypassing any reading of the offsets buffer altogether), that does not mean the representation is safe throughout the library. There are downstream assumptions that a sentinel offset exists. That strengthens the case for writing canonical [0] offsets in arrow-rs even if some readers currently tolerate the empty-offset form.

Given that, I think the safest behavior for arrow-rs is:

• write the canonical [0] offsets buffer for zero-length variable-width arrays, and
• continue to read/accept the permissive empty-offset form for compatibility.

If we want to make empty offsets formally spec-valid rather than just tolerated by some implementations, then I agree that should probably be discussed on the Arrow mailing list first.

[alamb]

label_issue.py automatically added labels {'arrow'} from #9717