Skip to content

Commit 9c68e58

Browse files
dfa1claude
andcommitted
fix: oracle formats UINT_32/UINT_64 Parquet columns as unsigned (#253)
INT32/INT64 columns with a UINT_32/UINT_64 Parquet logical-type annotation were formatted using Integer.toString/Long.toString (signed), which diverges from the unsigned U32/U64 representation stored in the Vortex file. Values above Integer.MAX_VALUE (e.g. total_boosters in covid-world-vaccination-progress) produced the wrong sign. Fix: detect LogicalType.IntType with isSigned()=false and switch to Integer.toUnsignedString / Long.toUnsignedString in oracleCell. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 246122a commit 9c68e58

3 files changed

Lines changed: 10 additions & 3 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
1616
- `SparseEncodingDecoder` now asserts exactly two children (`patch_indices`, `patch_values`) and throws `VortexException` on any other count; previously a third child was silently accepted. ([#250](https://github.com/dfa1/vortex-java/issues/250))
1717
- `DateTimePartsArrays.readLong` now zero-extends unsigned children (`U8`/`U16`/`U32`) instead of sign-extending them; previously a `U16` seconds-within-day value ≥ 32 768 was widened as a negative short, producing a 65 536-second error in the reconstructed timestamp (#252, bi-euro2016 corpus).
1818
- `RaincloudConformanceIntegrationTest` now writes both the vortex-java and Parquet oracle CSVs to temp files and streams them line-by-line for comparison; previously it materialized both into heap `String` objects, causing OOM for large corpus files (≥157 MB) that were previously hidden by throwing before reaching the corrupted line. ([#254](https://github.com/dfa1/vortex-java/issues/254))
19+
- `RaincloudConformanceIntegrationTest` oracle now formats `INT32`/`INT64` Parquet columns with a `UINT_32`/`UINT_64` logical-type annotation using `Integer.toUnsignedString`/`Long.toUnsignedString`, matching the unsigned representation that Vortex stores as `U32`/`U64`; previously the Parquet signed interpretation diverged for values above `Integer.MAX_VALUE` (e.g. `total_boosters` in the covid-world-vaccination-progress corpus). ([#253](https://github.com/dfa1/vortex-java/issues/253))
1920

2021
## [0.12.1] — 2026-07-08
2122

integration/src/test/java/io/github/dfa1/vortex/integration/RaincloudConformanceIntegrationTest.java

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22

33
import de.siegmar.fastcsv.writer.CsvWriter;
44
import dev.hardwood.InputFile;
5+
import dev.hardwood.metadata.LogicalType;
56
import dev.hardwood.metadata.RepetitionType;
67
import dev.hardwood.reader.ParquetFileReader;
78
import dev.hardwood.reader.RowReader;
@@ -220,6 +221,10 @@ private static void writeOracleCsv(Path parquet, Writer out) throws IOException
220221
/// null rows export as an empty field, valid rows use the JDK canonical
221222
/// `toString` of the value.
222223
///
224+
/// INT32/INT64 columns with a `UINT_32`/`UINT_64` logical-type annotation are treated
225+
/// as unsigned so their string representation matches the U32/U64 Vortex columns that
226+
/// carry the same bits (#253).
227+
///
223228
/// @param col the column schema
224229
/// @param rows the row reader positioned at the current row
225230
/// @return the formatted cell string
@@ -228,9 +233,10 @@ private static String oracleCell(ColumnSchema col, RowReader rows) {
228233
if (col.repetitionType() == RepetitionType.OPTIONAL && rows.isNull(name)) {
229234
return "";
230235
}
236+
boolean unsignedInt = col.logicalType() instanceof LogicalType.IntType lt && !lt.isSigned();
231237
return switch (col.type()) {
232-
case INT32 -> Integer.toString(rows.getInt(name));
233-
case INT64 -> Long.toString(rows.getLong(name));
238+
case INT32 -> unsignedInt ? Integer.toUnsignedString(rows.getInt(name)) : Integer.toString(rows.getInt(name));
239+
case INT64 -> unsignedInt ? Long.toUnsignedString(rows.getLong(name)) : Long.toString(rows.getLong(name));
234240
case FLOAT -> Float.toString(rows.getFloat(name));
235241
case DOUBLE -> Double.toString(rows.getDouble(name));
236242
case BOOLEAN -> Boolean.toString(rows.getBoolean(name));

integration/src/test/resources/raincloud/expected-status.csv

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ coig,untriaged
8282
coronahack-chest-xraydataset,untriaged
8383
cosmopedia-stanford,untriaged
8484
countries-of-the-world,ok
85-
covid-world-vaccination-progress,gap:253
85+
covid-world-vaccination-progress,ok
8686
covid19-data-from-john-hopkins-university,untriaged
8787
crimes-in-boston,untriaged
8888
databricks-dolly-15k,untriaged

0 commit comments

Comments
 (0)