Describe the bug
NativeFormatReader.readBlock() (client-v2) decodes Array(T) columns incorrectly when a result has more than one row and the rows have different array lengths.
In the ClickHouse Native format an Array(T) column in a block is encoded as N cumulative UInt64 offsets followed by the flattened elements. The per-row element count for row j is offsets[j] - offsets[j-1] (with offsets[-1] = 0). The reader instead reads the offsets into a local array and then uses sizes[0] (row 0's offset) as the element count for every row.
Consequences:
- rows after the first are truncated / contain the wrong elements;
- because fewer elements than encoded are consumed, the stream desyncs and the following columns in the same block are corrupted (often surfacing as
IllegalArgumentException: Non-empty typeName is required while parsing the next column).
It is accidentally correct only when every row has the same length (all equal to row 0's length), which is why existing single-row / uniform-length tests do not catch it.
There is also a latent problem on the same path for empty array rows: BinaryStreamReader.readArrayItem(col, len) is called directly by the Native reader and, for a non-nullable primitive element type, unconditionally reads one element and writes index 0 — so len == 0 reads a phantom value and throws ArrayIndexOutOfBoundsException. (readArray avoids this by guarding len == 0 before calling readArrayItem; the Native reader does not.)
How to reproduce
- ClickHouse version: 26.5 (any)
- Client version: current
main (0.11.0-rc1)
- Language: Java client-v2
CREATE TABLE default.arrtest (id UInt32, arr Array(Int32)) ENGINE=Memory;
INSERT INTO default.arrtest VALUES (1,[10]),(2,[20,21]),(3,[30,31,32]);
SELECT arr FROM default.arrtest ORDER BY id FORMAT Native;
Native bytes for the arr column: offsets = [1, 3, 6] (cumulative UInt64), data = 10,20,21,30,31,32.
Reading this via client.query(..., new QuerySettings().setFormat(ClickHouseFormat.Native)) + newBinaryFormatReader(response):
- Expected:
[10], [20, 21], [30, 31, 32]
- Actual:
[10], [20], [21] (each row truncated to sizes[0] == 1), and any column after the array in the block is corrupted.
Expected behaviour
Each row's array should contain exactly offsets[j] - offsets[j-1] elements, and columns following an array column in the same block should read correctly (including rows whose array is empty).
Notes
Found during review of PR #2939 (QBit support); confirmed pre-existing and unrelated to that change (the if (column.isArray()) block dates to commit a07534070, 2024-12-03). Filing separately so the fix stays one-concern-per-PR.
Describe the bug
NativeFormatReader.readBlock()(client-v2) decodesArray(T)columns incorrectly when a result has more than one row and the rows have different array lengths.In the ClickHouse Native format an
Array(T)column in a block is encoded as N cumulativeUInt64offsets followed by the flattened elements. The per-row element count for rowjisoffsets[j] - offsets[j-1](withoffsets[-1] = 0). The reader instead reads the offsets into a local array and then usessizes[0](row 0's offset) as the element count for every row.Consequences:
IllegalArgumentException: Non-empty typeName is requiredwhile parsing the next column).It is accidentally correct only when every row has the same length (all equal to row 0's length), which is why existing single-row / uniform-length tests do not catch it.
There is also a latent problem on the same path for empty array rows:
BinaryStreamReader.readArrayItem(col, len)is called directly by the Native reader and, for a non-nullable primitive element type, unconditionally reads one element and writes index 0 — solen == 0reads a phantom value and throwsArrayIndexOutOfBoundsException. (readArrayavoids this by guardinglen == 0before callingreadArrayItem; the Native reader does not.)How to reproduce
main(0.11.0-rc1)Native bytes for the
arrcolumn: offsets =[1, 3, 6](cumulativeUInt64), data =10,20,21,30,31,32.Reading this via
client.query(..., new QuerySettings().setFormat(ClickHouseFormat.Native))+newBinaryFormatReader(response):[10],[20, 21],[30, 31, 32][10],[20],[21](each row truncated tosizes[0] == 1), and any column after the array in the block is corrupted.Expected behaviour
Each row's array should contain exactly
offsets[j] - offsets[j-1]elements, and columns following an array column in the same block should read correctly (including rows whose array is empty).Notes
Found during review of PR #2939 (QBit support); confirmed pre-existing and unrelated to that change (the
if (column.isArray())block dates to commita07534070, 2024-12-03). Filing separately so the fix stays one-concern-per-PR.