Skip to content

[client-v2] NativeFormatReader misreads multi-row Array columns with non-uniform lengths #2955

Description

@polyglotAI-bot

Describe the bug

NativeFormatReader.readBlock() (client-v2) decodes Array(T) columns incorrectly when a result has more than one row and the rows have different array lengths.

In the ClickHouse Native format an Array(T) column in a block is encoded as N cumulative UInt64 offsets followed by the flattened elements. The per-row element count for row j is offsets[j] - offsets[j-1] (with offsets[-1] = 0). The reader instead reads the offsets into a local array and then uses sizes[0] (row 0's offset) as the element count for every row.

Consequences:

  • rows after the first are truncated / contain the wrong elements;
  • because fewer elements than encoded are consumed, the stream desyncs and the following columns in the same block are corrupted (often surfacing as IllegalArgumentException: Non-empty typeName is required while parsing the next column).

It is accidentally correct only when every row has the same length (all equal to row 0's length), which is why existing single-row / uniform-length tests do not catch it.

There is also a latent problem on the same path for empty array rows: BinaryStreamReader.readArrayItem(col, len) is called directly by the Native reader and, for a non-nullable primitive element type, unconditionally reads one element and writes index 0 — so len == 0 reads a phantom value and throws ArrayIndexOutOfBoundsException. (readArray avoids this by guarding len == 0 before calling readArrayItem; the Native reader does not.)

How to reproduce

  • ClickHouse version: 26.5 (any)
  • Client version: current main (0.11.0-rc1)
  • Language: Java client-v2
CREATE TABLE default.arrtest (id UInt32, arr Array(Int32)) ENGINE=Memory;
INSERT INTO default.arrtest VALUES (1,[10]),(2,[20,21]),(3,[30,31,32]);
SELECT arr FROM default.arrtest ORDER BY id FORMAT Native;

Native bytes for the arr column: offsets = [1, 3, 6] (cumulative UInt64), data = 10,20,21,30,31,32.

Reading this via client.query(..., new QuerySettings().setFormat(ClickHouseFormat.Native)) + newBinaryFormatReader(response):

  • Expected: [10], [20, 21], [30, 31, 32]
  • Actual: [10], [20], [21] (each row truncated to sizes[0] == 1), and any column after the array in the block is corrupted.

Expected behaviour

Each row's array should contain exactly offsets[j] - offsets[j-1] elements, and columns following an array column in the same block should read correctly (including rows whose array is empty).

Notes

Found during review of PR #2939 (QBit support); confirmed pre-existing and unrelated to that change (the if (column.isArray()) block dates to commit a07534070, 2024-12-03). Filing separately so the fix stays one-concern-per-PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions