fix(calcite): answer SUM over an all-null column as SQL NULL, not 0

The writer records an all-null zone's SUM as a sum-neutral 0 (validity
placeholders are zero), so the zone-map fold cannot tell an all-null
column from a genuine zero. The push-down rule therefore answered
`SUM(col)` over an all-null column as 0, diverging from SQL (and the
scan path), which yield NULL.

Guard the SUM branch with the column's null count, mirroring the MIN/MAX
provably-no-values check: emit the SQL NULL literal when the column is
provably all-null (empty table, or nullCount == rows); keep a non-zero
fold; and abandon a zero fold whose null count is unknown for a nullable
column (a genuine 0 is then indistinguishable from all-null). A
non-nullable column has no nulls, so any fold is the true sum.

VortexTable.zoneSum now returns a ZoneSum(sum, nullCount, totalRows)
record read in a single reader pass, so the null-aware SUM costs one
file open rather than three.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: dfa1

CHANGELOG.md

@@ -9,7 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- The Calcite aggregate push-down rule now rewrites a whole-table `SUM(col)` to a single-row `Values` computed from the zone-map table (via `VortexTable.zoneSum`), so `SELECT SUM(col)` answers metadata-only with no data segment decoded — joining the existing `MIN`/`MAX`/`COUNT` push-down. It abandons to a normal scan when a zone carries no usable sum (no zone map, or an overflowed zone) or when a `WHERE` filters the aggregate. ([24b64b32](https://github.com/dfa1/vortex-java/commit/24b64b32))
+- The Calcite aggregate push-down rule now rewrites a whole-table `SUM(col)` to a single-row `Values` computed from the zone-map table (via `VortexTable.zoneSum`), so `SELECT SUM(col)` answers metadata-only with no data segment decoded — joining the existing `MIN`/`MAX`/`COUNT` push-down. `SUM` over an all-null (or empty) column answers the SQL `NULL`, and the rule abandons to a normal scan when a zone carries no usable sum (no zone map, or an overflowed zone) or when a `WHERE` filters the aggregate. ([24b64b32](https://github.com/dfa1/vortex-java/commit/24b64b32))
 - `ScanIterator.columnZoneStats(column)` surfaces per-zone min/max/sum/null-count from a column's `vortex.stats` zone-map table without decoding any data segment — the read side of aggregate push-down (ADR 0013 §6). `ArrayStats` gains a `sum` component, decoded from the zone-map table (where the Rust reference stores it too), so the Calcite adapter now answers `SUM`/`AVG` metadata-only when every zone carries a sum, falling back to a streaming scan only for columns without a zone map. ([05dd9204](https://github.com/dfa1/vortex-java/commit/05dd9204))
 
 ### Changed

calcite/src/main/java/io/github/dfa1/vortex/calcite/VortexAggregatePushDownRule.java

@@ -31,10 +31,12 @@
 ///
 /// Fires only when it can answer *every* aggregate from statistics: no `GROUP BY`, and each call
 /// is `COUNT(*)`, `COUNT(col)`, `MIN(col)`, `MAX(col)`, or `SUM(col)` over a numeric column. `SUM`
-/// folds the per-zone `SUM` rows via [VortexTable#zoneSum(String)]; it abandons (falling back to
-/// the scan) for a column whose zone-map table cannot answer it — no zone map, or an overflowed
-/// zone. Anything else (a grouped aggregate, `MIN` on a non-numeric column, `AVG` that was not
-/// reduced to `SUM`/`COUNT`) leaves the plan untouched for the normal scan path.
+/// folds the per-zone `SUM` rows via [VortexTable#zoneSum(String)]; it emits the SQL `NULL` of an
+/// all-null (or empty) column, and abandons (falling back to the scan) for a column whose zone-map
+/// table cannot answer it — no zone map, an overflowed zone, or a zero fold whose null count is
+/// unknown (a genuine zero is indistinguishable from all-null). Anything else (a grouped aggregate,
+/// `MIN` on a non-numeric column, `AVG` that was not reduced to `SUM`/`COUNT`) leaves the plan
+/// untouched for the normal scan path.
 // Calcite 1.40 removed RelRule.Config.EMPTY; the modern RelRule.Config path requires the
 // Immutables annotation processor. The classic operand() constructor is deprecated but fully
 // supported and far lighter for a single adapter rule — suppression is localized and justified.
@@ -163,14 +165,30 @@ private static RexLiteral evaluate(AggregateCall agg, RelDataType outType, Vorte
                 if (col == null) {
                     yield null;
                 }
-                // Fold the per-zone SUM rows (metadata-only). A null fold means a zone carries no
-                // usable sum (no zone map, or an overflowed zone) — abandon so the scan computes it.
-                // It also covers SQL's empty/all-null SUM = NULL: zero zones fold to null and the
-                // scan path produces the NULL literal, so the rule need not special-case it.
-                Number sum = table.zoneSum(col);
+                // Fold the per-zone SUM rows (metadata-only), with the null and row counts read in
+                // the same pass. A null fold means a zone carries no usable sum (no zone map, or an
+                // overflowed zone) — abandon so the scan computes it.
+                VortexTable.ZoneSum zoneSum = table.zoneSum(col);
+                Number sum = zoneSum.sum();
                 if (sum == null) {
                     yield null;
                 }
+                // SQL SUM over zero non-null rows is NULL, not 0 — but the writer records an all-null
+                // zone's sum as a sum-neutral 0 (validity placeholders are zero), so the fold cannot
+                // tell an all-null column from a genuine 0. Resolve it from the null count:
+                //   - provably all-null (empty table, or nullCount == rows) → emit the NULL literal;
+                //   - fold is a non-zero value → at least one non-null row, the number is correct;
+                //   - fold is 0 but the null count is unknown for a nullable column → ambiguous
+                //     (real 0 vs all-null), so abandon and let the scan decide.
+                // A non-nullable column has no nulls, so any fold is the true sum.
+                Long nullCount = zoneSum.nullCount();
+                long total = zoneSum.totalRows();
+                if (total == 0 || (nullCount != null && nullCount == total)) {
+                    yield rexBuilder.makeNullLiteral(outType);
+                }
+                if (isZero(sum) && nullCount == null && isNullable(scanRowType, col)) {
+                    yield null;
+                }
                 yield numericLiteral(rexBuilder, sum, outType);
             }
             default -> null;
@@ -196,6 +214,16 @@ private static String resolveColumn(int aggInput, List<String> scanColumns, Proj
         return null;
     }
 
+    /// Returns whether a folded sum is numerically zero (`0L` or `0.0`) — the value an all-null zone
+    /// records, indistinguishable from a genuine zero sum without a null count.
+    private static boolean isZero(Number sum) {
+        return switch (sum) {
+            case Long l -> l == 0L;
+            case Double d -> d == 0.0;
+            default -> sum.doubleValue() == 0.0;
+        };
+    }
+
     private static RexLiteral exact(RexBuilder rexBuilder, long value, RelDataType type) {
         return rexBuilder.makeExactLiteral(BigDecimal.valueOf(value), type);
     }

calcite/src/main/java/io/github/dfa1/vortex/calcite/VortexTable.java

@@ -87,18 +87,41 @@ public io.github.dfa1.vortex.reader.ArrayStats statsOf(String column) {
         }
     }
 
-    /// Folds the per-zone `SUM` statistics for `column` without decoding any data segment, or
-    /// returns `null` when the zone-map table cannot answer the reduction — a column with no zone
-    /// map, or a zone whose sum was not retained (e.g. an overflowed zone) — so the caller scans.
+    /// The folded `SUM` of a column together with the two facts needed to interpret a zero fold:
+    /// SQL `SUM` over zero non-null rows is `NULL`, but an all-null zone records a sum-neutral `0`,
+    /// so `sum == 0` alone cannot tell an all-null column from a genuine zero. The `nullCount` and
+    /// `totalRows` resolve it.
     ///
-    /// Integer columns fold into a [Long] (exact); floating columns into a [Double]. Used by the
-    /// aggregate push-down rule to answer `SUM` (ADR 0013 §6).
+    /// @param sum       the folded column sum ([Long] for integer columns, [Double] for floating),
+    ///                  or `null` when no zone carries a usable sum (no zone map, or an overflowed
+    ///                  zone)
+    /// @param nullCount the column's total null count from the zone map, or `null` if not recorded
+    /// @param totalRows the total row count across all chunks
+    public record ZoneSum(Number sum, Long nullCount, long totalRows) {
+    }
+
+    /// Folds the per-zone `SUM` statistics for `column` without decoding any data segment, reading
+    /// the fold, null count, and row count in a single pass over the open reader so the caller can
+    /// answer `SUM` null-correctly. Used by the aggregate push-down rule (ADR 0013 §6).
+    ///
+    /// Integer columns fold into a [Long] (exact); floating columns into a [Double]. The
+    /// [ZoneSum#sum()] is `null` when the zone-map table cannot answer the reduction — a column
+    /// with no zone map, or a zone whose sum was not retained (e.g. an overflowed zone).
     ///
     /// @param column the numeric column name
-    /// @return the column sum as a [Long] or [Double], or `null` if no zone carries a usable sum
-    public Number zoneSum(String column) {
+    /// @return the folded sum with the null and row counts needed to interpret a zero
+    public ZoneSum zoneSum(String column) {
         try (VortexReader reader = VortexReader.open(file)) {
-            return new ZoneReducer(reader).sum(column);
+            Number sum = new ZoneReducer(reader).sum(column);
+            Long nullCount = reader.columnStats()
+                    .getOrDefault(column, io.github.dfa1.vortex.reader.ArrayStats.empty()).nullCount();
+            long total = 0;
+            try (ScanIterator scan = reader.scan(ScanOptions.all())) {
+                for (long c : scan.chunkRowCounts()) {
+                    total += c;
+                }
+            }
+            return new ZoneSum(sum, nullCount, total);
         } catch (IOException e) {
             throw new UncheckedIOException("cannot read zone sum of " + file, e);
         }

calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregateSumNullTest.java

@@ -0,0 +1,125 @@
+package io.github.dfa1.vortex.calcite;
+
+import io.github.dfa1.vortex.core.model.DType;
+import io.github.dfa1.vortex.core.model.PType;
+import io.github.dfa1.vortex.writer.VortexWriter;
+import io.github.dfa1.vortex.writer.WriteOptions;
+import io.github.dfa1.vortex.writer.encode.NullableData;
+
+import org.apache.calcite.avatica.util.Casing;
+import org.apache.calcite.plan.RelOptUtil;
+import org.apache.calcite.plan.hep.HepPlanner;
+import org.apache.calcite.plan.hep.HepProgram;
+import org.apache.calcite.plan.hep.HepProgramBuilder;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.Values;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.schema.SchemaPlus;
+import org.apache.calcite.sql.SqlNode;
+import org.apache.calcite.sql.parser.SqlParser;
+import org.apache.calcite.tools.FrameworkConfig;
+import org.apache.calcite.tools.Frameworks;
+import org.apache.calcite.tools.Planner;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.IOException;
+import java.nio.channels.FileChannel;
+import java.nio.file.Path;
+import java.nio.file.StandardOpenOption;
+import java.util.List;
+import java.util.Map;
+
+import static org.assertj.core.api.Assertions.assertThat;
+
+/// Null-aware `SUM` push-down for [VortexAggregatePushDownRule]: the per-zone SUM of an all-null
+/// zone is recorded as a sum-neutral 0, so a naive fold would answer `SUM` over an all-null column
+/// as `0` instead of the SQL `NULL`. These tests pin the null-count guard that distinguishes the
+/// two — and prove a *genuine* zero sum (no nulls) still pushes down to the literal `0`.
+class AggregateSumNullTest {
+
+    @TempDir
+    Path tmp;
+
+    /// Single nullable `I64` column `v`, written as one chunk (one zone) so the file carries exactly
+    /// one per-zone SUM row. `values` holds the raw longs; `valid[i] == false` marks row `i` null.
+    private SchemaPlus tableOf(long[] values, boolean[] valid) throws IOException {
+        DType.Struct schema = new DType.Struct(
+                List.of("v"), List.of(new DType.Primitive(PType.I64, true)), false);
+        Path file = tmp.resolve("sum-nulls.vortex");
+        // Large chunk so the whole column is one chunk; zone maps on so the SUM stat is emitted.
+        WriteOptions opts = new WriteOptions(1024, true, 0.90, 0, false, false);
+        try (var ch = FileChannel.open(file, StandardOpenOption.CREATE, StandardOpenOption.WRITE);
+             var writer = VortexWriter.create(ch, schema, opts)) {
+            writer.writeChunk(Map.of("v", new NullableData(values, valid)));
+        }
+        SchemaPlus root = Frameworks.createRootSchema(true);
+        return root.add("vtx", new VortexSchema(Map.of("t", file)));
+    }
+
+    @Test
+    void sumOverAllNullColumn_rewritesToNullNotZero() throws Exception {
+        // Given a column whose every row is null — SQL SUM is NULL, yet the writer records the zone
+        // sum as a sum-neutral 0; the null-count guard must override the 0 fold
+        SchemaPlus schema = tableOf(new long[]{0, 0, 0}, new boolean[]{false, false, false});
+
+        // When the rule runs over SUM(v)
+        Values values = pushDown(schema, "select sum(v) from t");
+
+        // Then it answers from stats (a single-row Values, no scan) but the value is SQL NULL, not 0
+        assertThat(values).isNotNull();
+        RexLiteral sum = values.getTuples().getFirst().getFirst();
+        assertThat(sum.isNull()).isTrue();
+    }
+
+    @Test
+    void sumOverGenuineZero_rewritesToZeroLiteral() throws Exception {
+        // Given a column with no nulls whose values cancel to 0 (1 + -1) — the fold is 0 but the
+        // null count is 0, so the guard must NOT mistake it for all-null; SUM is the literal 0
+        SchemaPlus schema = tableOf(new long[]{1, -1}, new boolean[]{true, true});
+
+        // When the rule runs over SUM(v)
+        Values values = pushDown(schema, "select sum(v) from t");
+
+        // Then the rewrite succeeds with an exact 0, not NULL and not abandoned
+        assertThat(values).isNotNull();
+        RexLiteral sum = values.getTuples().getFirst().getFirst();
+        assertThat(sum.isNull()).isFalse();
+        assertThat(sum.getValueAs(Long.class)).isZero();
+    }
+
+    /// Optimizes `sql` with the push-down rules and returns the single-row [Values] the rewrite
+    /// produced, asserting no scan or aggregate survived. Fails the test if the rule abandoned.
+    private static Values pushDown(SchemaPlus schema, String sql) throws Exception {
+        FrameworkConfig config = Frameworks.newConfigBuilder()
+                .defaultSchema(schema)
+                .parserConfig(SqlParser.config().withUnquotedCasing(Casing.UNCHANGED))
+                .build();
+        Planner planner = Frameworks.getPlanner(config);
+        SqlNode parsed = planner.parse(sql);
+        RelNode logical = planner.rel(planner.validate(parsed)).rel;
+        HepProgram program = new HepProgramBuilder()
+                .addRuleCollection(VortexAggregatePushDownRule.RULES)
+                .build();
+        HepPlanner hep = new HepPlanner(program);
+        hep.setRoot(logical);
+        RelNode optimized = hep.findBestExp();
+
+        String plan = RelOptUtil.toString(optimized);
+        assertThat(plan).contains("LogicalValues").doesNotContain("TableScan").doesNotContain("Aggregate");
+        return findValues(optimized);
+    }
+
+    private static Values findValues(RelNode node) {
+        if (node instanceof Values values) {
+            return values;
+        }
+        for (RelNode input : node.getInputs()) {
+            Values found = findValues(input);
+            if (found != null) {
+                return found;
+            }
+        }
+        return null;
+    }
+}