Skip to content

Commit 6058e9b

Browse files
dfa1claude
dfa1
and
claude
committed
refactor: type column names as ColumnName across read and write paths (#200)
A Map<String, Array> lies about its own domain: String admits any text, but the map holds columns of this schema. Typing the key — the keystone being DType.Struct.fieldNames() -> List<ColumnName> — makes the signature tell the truth and turns whole classes of key-mismatch bugs into compile errors (this change's own calcite zoneStats.get(col) NPE, a String-map looked up with a ColumnName, could no longer compile). Typed throughout vortex's own APIs: fieldNames(), VortexReader.columnStats() keys, RowFilter.Column.column, ScanIterator projection, the writer's internal column maps, and the Chunk.put(ColumnName) builder. The builder's former put(String) sugar was test-only, so it came off. String stays only at genuine external boundaries — user-input sugar (field(String), RowFilter.eq(String), ScanOptions.columns(String...), writeChunk(Map<String,Object>)) and other-system namespaces (Calcite RelDataType, CSV headers, JDBC/Parquet metadata) — converted via .value() at the seam. Wire output byte-identical; Rust-interop oracle green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 41ea5d8 commit 6058e9b

85 files changed

Lines changed: 423 additions & 358 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,8 @@ claim measured against the Rust (JNI) oracle; behavior divergences documented in
3131

3232
### Changed
3333

34+
- Column names are typed as `ColumnName` throughout vortex's own read and write APIs, not raw `String`. `DType.Struct.fieldNames()` returns `List<ColumnName>` (the builder already validated then discarded the type — now it keeps it), `VortexReader.columnStats()` is keyed by `ColumnName`, `RowFilter.Column.column` is a `ColumnName`, and the writer's internal column maps and typed `Chunk.put(ColumnName, …)` builder follow. User-facing sugar is unchanged — `structBuilder().field(String)`, `RowFilter.eq(String, …)`, `ScanOptions.columns(String...)`, and `writeChunk(Map<String,Object>)` still take strings and validate at the edge. A footgun field name (blank/control) is now rejected structurally at `ColumnName` construction — a footgun-named schema can no longer be built. Framework/IO boundaries (Calcite, CSV, JDBC, Parquet, display) keep `String`. Wire output unchanged (integration oracle green). ([84769863](https://github.com/dfa1/vortex-java/commit/84769863))
35+
3436
- `ScanOptions.columns()` returns `List<ColumnName>` instead of `List<String>`. The `columns(String...)` / `withColumns(String...)` factories still take strings at the API boundary but validate them into `ColumnName`, so a blank or control-character projection name fails fast rather than silently matching nothing. ([e478a3a7](https://github.com/dfa1/vortex-java/commit/e478a3a7))
3537

3638
- `Footer.arraySpecs()` / `Footer.layoutSpecs()` return typed `List<EncodingId>` / `List<LayoutId>` instead of `List<String>`. The wire strings are parsed to their typed ids once at the footer boundary (with the blank-id guard), so array and layout nodes index directly into typed dictionaries — completing "strings at the boundary, types inside": the two scattered per-node `parse` calls in `SerializedArrayDecoder` and the layout converter are gone. ([0dd677ef](https://github.com/dfa1/vortex-java/commit/0dd677ef))

‎calcite/src/main/java/io/github/dfa1/vortex/calcite/VortexAggregates.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
package io.github.dfa1.vortex.calcite;
22

3+
import io.github.dfa1.vortex.core.model.ColumnName;
34
import io.github.dfa1.vortex.reader.ArrayStats;
45
import io.github.dfa1.vortex.reader.Chunk;
56
import io.github.dfa1.vortex.reader.ScanIterator;
@@ -58,7 +59,7 @@ private VortexAggregates() {
5859
/// @param column the numeric column name
5960
/// @return the column's aggregate summary
6061
public static Summary of(VortexReader reader, String column) {
61-
ArrayStats stats = reader.columnStats().getOrDefault(column, ArrayStats.empty());
62+
ArrayStats stats = reader.columnStats().getOrDefault(ColumnName.of(column), ArrayStats.empty());
6263
long totalRows = totalRows(reader);
6364
long nullCount = stats.nullCount() == null ? 0L : stats.nullCount();
6465
long count = totalRows - nullCount;

‎calcite/src/main/java/io/github/dfa1/vortex/calcite/VortexTable.java‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
package io.github.dfa1.vortex.calcite;
22

3+
import io.github.dfa1.vortex.core.model.ColumnName;
34
import io.github.dfa1.vortex.core.model.DType;
45
import io.github.dfa1.vortex.reader.ArrayStats;
56
import io.github.dfa1.vortex.reader.Chunk;
@@ -103,7 +104,7 @@ public record ColumnStats(ArrayStats stats, long totalRows) {
103104
/// @return the column's aggregated statistics paired with the file's total row count
104105
public ColumnStats statsAndRows(String column) {
105106
try (VortexReader reader = VortexReader.open(file)) {
106-
ArrayStats stats = reader.columnStats().getOrDefault(column, ArrayStats.empty());
107+
ArrayStats stats = reader.columnStats().getOrDefault(ColumnName.of(column), ArrayStats.empty());
107108
return new ColumnStats(stats, countRows(reader));
108109
} catch (IOException e) {
109110
throw new UncheckedIOException("cannot read stats of " + file, e);
@@ -137,7 +138,7 @@ public ZoneSum zoneSum(String column) {
137138
try (VortexReader reader = VortexReader.open(file)) {
138139
Number sum = new ZoneReducer(reader).sum(column);
139140
Long nullCount = reader.columnStats()
140-
.getOrDefault(column, ArrayStats.empty()).nullCount();
141+
.getOrDefault(ColumnName.of(column), ArrayStats.empty()).nullCount();
141142
return new ZoneSum(sum, nullCount, countRows(reader));
142143
} catch (IOException e) {
143144
throw new UncheckedIOException("cannot read zone sum of " + file, e);
@@ -420,7 +421,7 @@ private FilteredFold result() {
420421
/// @return the conjoined filter, or empty when any predicate is not fully translatable
421422
public Optional<RowFilter> translatePushedFilters(List<RexNode> filters) {
422423
DType.Struct struct = struct();
423-
List<String> names = struct.fieldNames();
424+
List<String> names = struct.fieldNames().stream().map(ColumnName::value).toList();
424425
List<DType> types = struct.fieldTypes();
425426
List<RowFilter> translated = new ArrayList<>();
426427
for (RexNode node : filters) {
@@ -467,7 +468,7 @@ private static Match classify(RowFilter filter, int zone,
467468
yield allIn ? Match.IN : Match.BOUNDARY;
468469
}
469470
case RowFilter.Column(var col, var predicate) ->
470-
classifyColumn(predicate, zoneStats.get(col).get(zone), rowCount);
471+
classifyColumn(predicate, zoneStats.get(col.value()).get(zone), rowCount);
471472
};
472473
}
473474

@@ -567,15 +568,15 @@ private static Match compare(ArrayStats s, Object value,
567568
/// Whether `column` is an unsigned primitive, whose signed stat ordering [#compareStat] cannot
568569
/// safely classify (the fold abandons such columns to the scan).
569570
private static boolean isUnsigned(DType.Struct struct, String column) {
570-
int idx = struct.fieldNames().indexOf(column);
571+
int idx = struct.fieldNames().indexOf(ColumnName.of(column));
571572
return idx >= 0 && struct.fieldTypes().get(idx) instanceof DType.Primitive p && p.ptype().isUnsigned();
572573
}
573574

574575
/// Whether `column` is a floating-point primitive, whose compute-kernel compare
575576
/// ([Compare#values(Object, Object, DType)]) sorts NaN as the maximum and so cannot soundly fold a
576577
/// boundary partition (the fold abandons such a filter column to the NaN-correct scan).
577578
private static boolean isFloating(DType.Struct struct, String column) {
578-
int idx = struct.fieldNames().indexOf(column);
579+
int idx = struct.fieldNames().indexOf(ColumnName.of(column));
579580
return idx >= 0 && struct.fieldTypes().get(idx) instanceof DType.Primitive p && p.ptype().isFloating();
580581
}
581582

@@ -640,15 +641,15 @@ public RelDataType getRowType(RelDataTypeFactory typeFactory) {
640641
DType.Struct struct = struct();
641642
RelDataTypeFactory.Builder builder = typeFactory.builder();
642643
for (int i = 0; i < struct.fieldNames().size(); i++) {
643-
builder.add(struct.fieldNames().get(i), toSqlType(typeFactory, struct.fieldTypes().get(i)));
644+
builder.add(struct.fieldNames().get(i).value(), toSqlType(typeFactory, struct.fieldTypes().get(i)));
644645
}
645646
return builder.build();
646647
}
647648

648649
@Override
649650
public Enumerable<Object[]> scan(DataContext root, List<RexNode> filters, int[] projects) {
650651
DType.Struct struct = struct();
651-
List<String> allNames = struct.fieldNames();
652+
List<String> allNames = struct.fieldNames().stream().map(ColumnName::value).toList();
652653

653654
// Projection: the columns to decode and emit, in the order Calcite asked for. A null
654655
// projects array means "all columns".
@@ -847,7 +848,7 @@ private static Optional<RowFilter> toRowFilter(List<RexNode> filters, List<Strin
847848
private static void collectColumns(RowFilter filter, java.util.Set<String> out) {
848849
switch (filter) {
849850
case RowFilter.And(var parts) -> parts.forEach(f -> collectColumns(f, out));
850-
case RowFilter.Column(var col, var ignored) -> out.add(col);
851+
case RowFilter.Column(var col, var ignored) -> out.add(col.value());
851852
}
852853
}
853854

‎calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateSumNullTest.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
package io.github.dfa1.vortex.calcite;
22

3+
import io.github.dfa1.vortex.core.model.ColumnName;
34
import io.github.dfa1.vortex.core.model.DType;
45
import io.github.dfa1.vortex.core.model.PType;
56
import io.github.dfa1.vortex.writer.VortexWriter;
@@ -45,7 +46,7 @@ class AggregateSumNullTest {
4546
/// one per-zone SUM row. `values` holds the raw longs; `valid[i] == false` marks row `i` null.
4647
private SchemaPlus tableOf(long[] values, boolean[] valid) throws IOException {
4748
DType.Struct schema = new DType.Struct(
48-
List.of("v"), List.of(new DType.Primitive(PType.I64, true)), false);
49+
List.of(ColumnName.of("v")), List.of(new DType.Primitive(PType.I64, true)), false);
4950
Path file = tmp.resolve("sum-nulls.vortex");
5051
// Large chunk so the whole column is one chunk; zone maps on so the SUM stat is emitted.
5152
WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false);

‎calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereBoundaryTest.java‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
package io.github.dfa1.vortex.calcite;
22

3+
import io.github.dfa1.vortex.core.model.ColumnName;
34
import io.github.dfa1.vortex.core.model.DType;
45
import io.github.dfa1.vortex.core.model.PType;
56
import io.github.dfa1.vortex.writer.VortexWriter;
@@ -201,7 +202,7 @@ void nullableAggColumnAcrossBoundaryChunk() throws Exception {
201202
// predicate `id > 0 AND id < 100` cuts chunk 0 (drops id 0, keeps ids 1,2,3) — a boundary
202203
// zone — and selects chunk 1 whole (IN).
203204
DType.Struct schema = new DType.Struct(
204-
List.of("id", "val"),
205+
List.of(ColumnName.of("id"), ColumnName.of("val")),
205206
List.of(new DType.Primitive(PType.I64, false), new DType.Primitive(PType.I64, true)),
206207
false);
207208
Path f = tmp.resolve("nullable-boundary.vortex");
@@ -238,7 +239,7 @@ void nonNumericMinAcrossBoundaryAbandonsButScanIsCorrect() throws Exception {
238239
// rewrite must abandon so the scan returns the value. chunk 0 id 0..3 / name {"a".."d"},
239240
// chunk 1 id 10..13 / name {"e".."h"}; `id > 0 AND id < 100` cuts chunk 0 (keeps b,c,d).
240241
DType.Struct schema = new DType.Struct(
241-
List.of("id", "name"),
242+
List.of(ColumnName.of("id"), ColumnName.of("name")),
242243
List.of(new DType.Primitive(PType.I64, false), new DType.Utf8(false)),
243244
false);
244245
Path f = tmp.resolve("strmin-boundary.vortex");
@@ -267,7 +268,7 @@ void unsignedKeyAcrossBoundaryAbandonsButScanIsCorrect() throws Exception {
267268
// past the high bit, so the fold abandons any unsigned column outright — even on the boundary
268269
// path. chunk 0 id 0..3, chunk 1 id 10..13; `id > 1 AND id < 13` cuts both chunks.
269270
DType.Struct schema = new DType.Struct(
270-
List.of("id", "val"),
271+
List.of(ColumnName.of("id"), ColumnName.of("val")),
271272
List.of(new DType.Primitive(PType.U64, false), new DType.Primitive(PType.I64, false)),
272273
false);
273274
Path f = tmp.resolve("u64-boundary.vortex");
@@ -334,7 +335,7 @@ void neqWithNullInFilterColumnAcrossBoundaryExcludesNullAndValue() throws Except
334335
// null row AND the 30 row must both drop. This confirms the boundary mask's three-valued
335336
// logic for `<>` — a case the equality/range tests do not reach.
336337
DType.Struct schema = new DType.Struct(
337-
List.of("id", "amt"),
338+
List.of(ColumnName.of("id"), ColumnName.of("amt")),
338339
List.of(new DType.Primitive(PType.I64, false), new DType.Primitive(PType.I64, true)),
339340
false);
340341
Path f = tmp.resolve("neq-null-boundary.vortex");
@@ -385,7 +386,7 @@ void floatFilterColumnAcrossBoundaryAbandonsButScanIsNaNCorrect() throws Excepti
385386
// NaN-correct. chunk 0 f {1.0,NaN,2.0,3.0}, chunk 1 f {4.0,5.0,6.0,7.0}; `f >= 1.5` cuts
386387
// chunk 0 (a boundary) and keeps chunk 1 whole.
387388
DType.Struct schema = new DType.Struct(
388-
List.of("f", "val"),
389+
List.of(ColumnName.of("f"), ColumnName.of("val")),
389390
List.of(new DType.Primitive(PType.F64, false), new DType.Primitive(PType.I64, false)),
390391
false);
391392
Path f = tmp.resolve("float-nan-boundary.vortex");

‎calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateWhereCleanPartitionTest.java‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
package io.github.dfa1.vortex.calcite;
22

3+
import io.github.dfa1.vortex.core.model.ColumnName;
34
import io.github.dfa1.vortex.core.model.DType;
45
import io.github.dfa1.vortex.core.model.PType;
56
import io.github.dfa1.vortex.writer.VortexWriter;
@@ -155,7 +156,7 @@ void narrowIntegerKeyFoldsFromStats() throws Exception {
155156
// a same-type comparison would never fire — this guards that the widening compare folds
156157
// narrow columns. Two chunks: ids 0..3, then 10..13.
157158
DType.Struct schema = new DType.Struct(
158-
List.of("id", "val"),
159+
List.of(ColumnName.of("id"), ColumnName.of("val")),
159160
List.of(new DType.Primitive(PType.I32, false), new DType.Primitive(PType.I32, false)),
160161
false);
161162
Path f = tmp.resolve("i32.vortex");
@@ -185,7 +186,7 @@ void narrowIntegerKeyFoldsFromStats() throws Exception {
185186
void nullAwareSumAndCountOverKeptZones() throws Exception {
186187
// Given a non-null clustered key and a nullable value with two nulls in the kept chunk
187188
DType.Struct schema = new DType.Struct(
188-
List.of("id", "val"),
189+
List.of(ColumnName.of("id"), ColumnName.of("val")),
189190
List.of(new DType.Primitive(PType.I64, false), new DType.Primitive(PType.I64, true)),
190191
false);
191192
Path f = tmp.resolve("nullable.vortex");
@@ -217,7 +218,7 @@ void unsignedKeyColumnAbandonsToScan() throws Exception {
217218
// Given a clustered U64 key: a signed stat compare could fold the wrong zones past the high
218219
// bit, so the fold must abandon unsigned columns to the (unsigned-aware) scan
219220
DType.Struct schema = new DType.Struct(
220-
List.of("id", "val"),
221+
List.of(ColumnName.of("id"), ColumnName.of("val")),
221222
List.of(new DType.Primitive(PType.U64, false), new DType.Primitive(PType.I64, false)),
222223
false);
223224
Path f = tmp.resolve("u64.vortex");
@@ -301,7 +302,7 @@ void isNotNullBoundaryChunkFoldsViaDecode() throws Exception {
301302
// builds an IS NOT NULL mask, and reduces only its selected rows, so the fold answers
302303
// without a full scan.
303304
DType.Struct schema = new DType.Struct(
304-
List.of("id", "val"),
305+
List.of(ColumnName.of("id"), ColumnName.of("val")),
305306
List.of(new DType.Primitive(PType.I64, false), new DType.Primitive(PType.I64, true)),
306307
false);
307308
Path f = tmp.resolve("isnotnull-boundary.vortex");
@@ -413,7 +414,7 @@ void isNullOverExpressionAbandonsToScan() throws Exception {
413414
/// `IS NULL` / `IS NOT NULL` clean-partition tests.
414415
private static Path nullPartitionedFile(String name) throws Exception {
415416
DType.Struct schema = new DType.Struct(
416-
List.of("id", "val"),
417+
List.of(ColumnName.of("id"), ColumnName.of("val")),
417418
List.of(new DType.Primitive(PType.I64, false), new DType.Primitive(PType.I64, true)),
418419
false);
419420
Path f = tmp.resolve(name);

‎calcite/src/test/java/io/github/dfa1/vortex/calcite/OhlcSqlDemoTest.java‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
package io.github.dfa1.vortex.calcite;
22

3+
import io.github.dfa1.vortex.core.model.ColumnName;
34
import io.github.dfa1.vortex.reader.ScanIterator;
45
import io.github.dfa1.vortex.reader.ScanOptions;
56
import io.github.dfa1.vortex.reader.VortexReader;
@@ -224,7 +225,8 @@ private static Pushdown runPushdown(Path file) throws Exception {
224225
total += c;
225226
}
226227
}
227-
return new Pushdown(stats.get("low").min(), stats.get("high").max(), total);
228+
return new Pushdown(stats.get(ColumnName.of("low")).min(),
229+
stats.get(ColumnName.of("high")).max(), total);
228230
}
229231
}
230232

‎cli/src/main/java/io/github/dfa1/vortex/cli/FilterCommand.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -151,7 +151,7 @@ private static Comparable<?> parseValue(String raw) {
151151

152152
private static RowPredicate toRowPredicate(RowFilter filter) {
153153
return switch (filter) {
154-
case RowFilter.Column(var col, var predicate) -> columnPredicate(col, predicate);
154+
case RowFilter.Column(var col, var predicate) -> columnPredicate(col.value(), predicate);
155155
case RowFilter.And(var filters) -> {
156156
RowPredicate[] preds = filters.stream().map(FilterCommand::toRowPredicate).toArray(RowPredicate[]::new);
157157
yield (chunk, rowIdx) -> {

‎cli/src/main/java/io/github/dfa1/vortex/cli/SchemaCommand.java‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
package io.github.dfa1.vortex.cli;
22

3+
import io.github.dfa1.vortex.core.model.ColumnName;
34
import io.github.dfa1.vortex.core.model.DType;
45
import io.github.dfa1.vortex.reader.VortexReader;
56

@@ -41,8 +42,8 @@ private static void printSchema(PrintStream out, Path path, DType dtype, long ro
4142
out.printf("%s (%s rows, %d columns)%n%n",
4243
path.getFileName(), formatRows(rowCount), n);
4344
int idxWidth = Integer.toString(n).length();
44-
int nameWidth = s.fieldNames().stream().mapToInt(String::length).max().orElse(0);
45-
List<String> names = s.fieldNames();
45+
List<String> names = s.fieldNames().stream().map(ColumnName::value).toList();
46+
int nameWidth = names.stream().mapToInt(String::length).max().orElse(0);
4647
List<DType> types = s.fieldTypes();
4748
for (int i = 0; i < n; i++) {
4849
out.printf(" %" + idxWidth + "d %-" + nameWidth + "s %s%n",
@@ -67,7 +68,7 @@ static String formatDType(DType dtype) {
6768
if (i > 0) {
6869
sb.append(", ");
6970
}
70-
sb.append(s.fieldNames().get(i)).append(": ").append(formatDType(s.fieldTypes().get(i)));
71+
sb.append(s.fieldNames().get(i).value()).append(": ").append(formatDType(s.fieldTypes().get(i)));
7172
}
7273
sb.append('>');
7374
yield sb.toString();

‎cli/src/main/java/io/github/dfa1/vortex/cli/StatsCommand.java‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
package io.github.dfa1.vortex.cli;
22

33
import io.github.dfa1.vortex.reader.ArrayStats;
4+
import io.github.dfa1.vortex.core.model.ColumnName;
45
import io.github.dfa1.vortex.core.model.DType;
56
import io.github.dfa1.vortex.reader.VortexReader;
67

@@ -32,21 +33,21 @@ static int run(String[] args) {
3233
return ExitStatus.ERROR;
3334
}
3435
long totalRows = reader.layout().rowCount();
35-
Map<String, ArrayStats> stats = reader.columnStats();
36+
Map<ColumnName, ArrayStats> stats = reader.columnStats();
3637

3738
System.out.printf("rows: %,d%n%n", totalRows);
3839
System.out.printf("%-20s %-12s %15s %15s%n", "column", "type", "min", "max");
3940
System.out.println("-".repeat(67));
4041

41-
List<String> names = schema.fieldNames();
42+
List<ColumnName> names = schema.fieldNames();
4243
List<DType> types = schema.fieldTypes();
4344
for (int i = 0; i < names.size(); i++) {
44-
String name = names.get(i);
45+
ColumnName name = names.get(i);
4546
String type = formatDType(types.get(i));
4647
ArrayStats s = stats.getOrDefault(name, ArrayStats.empty());
4748
String min = s.min() != null ? s.min().toString() : "n/a";
4849
String max = s.max() != null ? s.max().toString() : "n/a";
49-
System.out.printf("%-20s %-12s %15s %15s%n", name, type, min, max);
50+
System.out.printf("%-20s %-12s %15s %15s%n", name.value(), type, min, max);
5051
}
5152
return ExitStatus.OK;
5253
} catch (IOException e) {

0 commit comments

Comments
 (0)