feat(calcite): auto-register aggregate push-down over bare jdbc, push AVG

Phase-2 productionisation (ADR 0018). Until now the aggregate push-down
rule only fired when a test wired it onto the planner by hand via
Hook.PLANNER; a plain `jdbc:calcite:` connection ran every aggregate as a
full scan.

VortexTable becomes a TranslatableTable that translates to a custom
VortexTableScan whose register() installs the rules the first time the
planner sees the node — so SELECT MIN/MAX/COUNT/SUM over a VortexSchema
is now answered from zone-map statistics with no caller wiring. The scan
immediately expands to a stock LogicalTableScan (VortexExpandTableScanRule),
leaving Calcite's Bindable column-projection and WHERE chunk-skip
push-down untouched (LogicalTableScan is final and its register() is a
no-op, so the custom-scan + expand-rule pair is the seam that adds the
registration hook without disturbing the scan/execute path).

AVG joins the pushed aggregates: register() also adds Calcite's
AggregateReduceFunctionsRule, which rewrites AVG to SUM/COUNT — both of
which the rule answers from stats — so a whole-table AVG also decodes no
data segment.

The two JDBC end-to-end tests drop their Hook.PLANNER install to prove
the rule auto-registers; OhlcSqlDemoTest's full-scan baseline switches
off `count(*)` (now itself pushed down to zero chunks) to a prune-nothing
filtered scan.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: dfa1

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- The Calcite aggregate push-down rule now **auto-registers** over a bare `jdbc:calcite:` connection: a `VortexTable` translates to a `VortexTableScan` whose `register()` installs the rules when the planner first sees it, so `SELECT MIN/MAX/COUNT/SUM` over a `VortexSchema` is answered from zone-map statistics with no caller wiring (previously the rule had to be attached to the planner by hand). `AVG` joins them — `AggregateReduceFunctionsRule` reduces it to `SUM`/`COUNT`, both of which push down, so a whole-table `AVG` also decodes no data segment. Column projection and `WHERE` chunk-skip push-down are unchanged. ([24b64b32](https://github.com/dfa1/vortex-java/commit/24b64b32))
 - The Calcite aggregate push-down rule now rewrites a whole-table `SUM(col)` to a single-row `Values` computed from the zone-map table (via `VortexTable.zoneSum`), so `SELECT SUM(col)` answers metadata-only with no data segment decoded — joining the existing `MIN`/`MAX`/`COUNT` push-down. `SUM` over an all-null (or empty) column answers the SQL `NULL`, and the rule abandons to a normal scan when a zone carries no usable sum (no zone map, or an overflowed zone) or when a `WHERE` filters the aggregate. ([24b64b32](https://github.com/dfa1/vortex-java/commit/24b64b32))
 - `ScanIterator.columnZoneStats(column)` surfaces per-zone min/max/sum/null-count from a column's `vortex.stats` zone-map table without decoding any data segment — the read side of aggregate push-down (ADR 0013 §6). `ArrayStats` gains a `sum` component, decoded from the zone-map table (where the Rust reference stores it too), so the Calcite adapter now answers `SUM`/`AVG` metadata-only when every zone carries a sum, falling back to a streaming scan only for columns without a zone map. ([05dd9204](https://github.com/dfa1/vortex-java/commit/05dd9204))
 

TODO.md

@@ -101,10 +101,12 @@ Per-encoding gotchas:
   Rust, whose flat writer omits per-flat sum). Calcite `VortexAggregates.SUM`/`AVG` now fold those
   per-zone sums (metadata-only), falling back to a full scan only when a column has no zone map.
   The fold is a reusable `reader.compute.ZoneReducer.sum(col)` (the seam a future `vortex-compute`
-  extracts), now consumed by the planner: `VortexAggregatePushDownRule` rewrites a whole-table
-  `SUM(col)` to a single-row `Values` via `VortexTable.zoneSum`, abandoning to the scan only when a
-  zone carries no usable sum. A `SUM` with a `WHERE` still abandons (whole-zone stats can't answer a
-  filtered aggregate) — that is the residual tier below.
+  extracts), consumed by the planner: `VortexAggregatePushDownRule` rewrites a whole-table
+  `MIN`/`MAX`/`COUNT`/`SUM`/`AVG` to a single-row `Values`, abandoning to the scan only when a zone
+  carries no usable sum (an all-null column answers SQL `NULL`; `AVG` reduces to `SUM`/`COUNT`). The
+  rule auto-registers over a bare `jdbc:calcite:` connection via `VortexTableScan.register()`, so
+  SQL over JDBC is rewritten with no caller wiring. A `SUM` with a `WHERE` still abandons (whole-zone
+  stats can't answer a filtered aggregate) — that is the residual tier below.
   Next: the residual tier — give `ZoneReducer` predicate support (whole-zone fold for fully-selected
   zones + boundary-zone streaming for partially-selected ones), then let the rule push `SUM` with a
   `WHERE`. `Mask`/`Predicate`/kernel vocab on top.

calcite/src/main/java/io/github/dfa1/vortex/calcite/VortexTable.java

@@ -21,6 +21,8 @@
 import org.apache.calcite.linq4j.AbstractEnumerable;
 import org.apache.calcite.linq4j.Enumerable;
 import org.apache.calcite.linq4j.Enumerator;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.rel.RelNode;
 import org.apache.calcite.rel.type.RelDataType;
 import org.apache.calcite.rel.type.RelDataTypeFactory;
 import org.apache.calcite.rex.RexBuilder;
@@ -30,6 +32,7 @@
 import org.apache.calcite.rex.RexNode;
 import org.apache.calcite.rex.RexUtil;
 import org.apache.calcite.schema.ProjectableFilterableTable;
+import org.apache.calcite.schema.TranslatableTable;
 import org.apache.calcite.schema.impl.AbstractTable;
 import org.apache.calcite.sql.type.SqlTypeName;
 
@@ -50,7 +53,7 @@
 /// consumed: zone-map pruning is approximate (it drops whole chunks that cannot match, not
 /// individual rows), so Calcite must still apply the predicate row-by-row for exactness. The
 /// win is decoding far fewer chunks when the filter is selective on a clustered column.
-public final class VortexTable extends AbstractTable implements ProjectableFilterableTable {
+public final class VortexTable extends AbstractTable implements ProjectableFilterableTable, TranslatableTable {
 
     /// Used only to expand `SEARCH`/`Sarg` predicates (e.g. `BETWEEN`, `IN`) back into ordinary
     /// comparison trees the [RowFilter] translation understands.
@@ -143,6 +146,19 @@ public long totalRows() {
         }
     }
 
+    /// Translates a reference to this table into a [VortexTableScan] — the seam that auto-registers
+    /// the aggregate push-down rules on the planner (see [VortexTableScan#register]). The scan
+    /// immediately expands to a stock `LogicalTableScan`, so projection and filter push-down run
+    /// through Calcite's `Bindables` convention unchanged.
+    ///
+    /// @param context     the planning context supplying the cluster
+    /// @param relOptTable the planner's handle to this table
+    /// @return a `VortexTableScan` over this table
+    @Override
+    public RelNode toRel(RelOptTable.ToRelContext context, RelOptTable relOptTable) {
+        return VortexTableScan.create(context.getCluster(), relOptTable);
+    }
+
     @Override
     public RelDataType getRowType(RelDataTypeFactory typeFactory) {
         DType.Struct struct = struct();

calcite/src/main/java/io/github/dfa1/vortex/calcite/VortexTableScan.java

@@ -0,0 +1,78 @@
+package io.github.dfa1.vortex.calcite;
+
+import org.apache.calcite.plan.Convention;
+import org.apache.calcite.plan.RelOptCluster;
+import org.apache.calcite.plan.RelOptPlanner;
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.plan.RelTraitSet;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.core.TableScan;
+import org.apache.calcite.rel.logical.LogicalTableScan;
+import org.apache.calcite.rel.rules.CoreRules;
+
+import java.util.List;
+
+/// The logical scan a [VortexTable] translates to. Its sole purpose over the stock
+/// [LogicalTableScan] is the [#register(RelOptPlanner)] hook: when the planner first sees this
+/// node class it installs the Vortex aggregate push-down rules, so a plain `jdbc:calcite:`
+/// connection rewrites whole-table aggregates from zone-map statistics **without the caller wiring
+/// the rule onto the planner** (ADR 0018 Phase 2 productionisation).
+///
+/// The node carries no projection or filter of its own. A `VortexExpandTableScanRule` immediately
+/// rewrites it to a [LogicalTableScan], so Calcite's stock `Bindables` rules drive column
+/// projection and `RowFilter` chunk-skip push-down exactly as before — this class only adds the
+/// rule-registration seam, it does not change the scan/execute path.
+final class VortexTableScan extends TableScan {
+
+    VortexTableScan(RelOptCluster cluster, RelTraitSet traitSet, RelOptTable table) {
+        super(cluster, traitSet, List.of(), table);
+    }
+
+    @Override
+    public RelNode copy(RelTraitSet traitSet, List<RelNode> inputs) {
+        assert inputs.isEmpty() : "VortexTableScan has no inputs";
+        return new VortexTableScan(getCluster(), traitSet, table);
+    }
+
+    /// Called once by the planner the first time a node of this class is registered. Installs the
+    /// whole-table aggregate push-down rules, the `AVG → SUM/COUNT` reduction that lets `AVG` ride
+    /// that path, and the rule that expands this node to a stock [LogicalTableScan].
+    ///
+    /// @param planner the planner registering this node class
+    @Override
+    public void register(RelOptPlanner planner) {
+        VortexAggregatePushDownRule.RULES.forEach(planner::addRule);
+        // Reduce AVG (and STDDEV/VAR) to SUM/COUNT so the aggregate rule can answer AVG from the
+        // per-zone SUM + row/null counts instead of streaming the column.
+        planner.addRule(CoreRules.AGGREGATE_REDUCE_FUNCTIONS);
+        planner.addRule(VortexExpandTableScanRule.INSTANCE);
+    }
+
+    /// Rewrites a [VortexTableScan] to a stock [LogicalTableScan], handing the scan back to
+    /// Calcite's `Bindables` convention for projection / filter push-down and execution. Without
+    /// this expansion the `Convention.NONE` [VortexTableScan] has no physical implementation.
+    // Same justification as VortexAggregatePushDownRule: the classic operand() constructor is
+    // deprecated but far lighter than the Immutables-based RelRule.Config for a one-line rule.
+    @SuppressWarnings("deprecation")
+    static final class VortexExpandTableScanRule extends RelOptRule {
+
+        static final VortexExpandTableScanRule INSTANCE = new VortexExpandTableScanRule();
+
+        private VortexExpandTableScanRule() {
+            super(operand(VortexTableScan.class, any()), "VortexExpandTableScanRule");
+        }
+
+        @Override
+        public void onMatch(RelOptRuleCall call) {
+            VortexTableScan scan = call.rel(0);
+            call.transformTo(LogicalTableScan.create(
+                    scan.getCluster(), scan.getTable(), scan.getHints()));
+        }
+    }
+
+    static VortexTableScan create(RelOptCluster cluster, RelOptTable table) {
+        return new VortexTableScan(cluster, cluster.traitSetOf(Convention.NONE), table);
+    }
+}

calcite/src/test/java/io/github/dfa1/vortex/calcite/AggregatePushDownTest.java

@@ -3,15 +3,13 @@
 import io.github.dfa1.vortex.reader.VortexReader;
 
 import org.apache.calcite.jdbc.CalciteConnection;
-import org.apache.calcite.plan.RelOptPlanner;
 import org.apache.calcite.plan.RelOptUtil;
 import org.apache.calcite.plan.hep.HepPlanner;
 import org.apache.calcite.plan.hep.HepProgram;
 import org.apache.calcite.plan.hep.HepProgramBuilder;
 import org.apache.calcite.rel.RelNode;
 import org.apache.calcite.rel.core.Values;
 import org.apache.calcite.rex.RexLiteral;
-import org.apache.calcite.runtime.Hook;
 import org.apache.calcite.schema.SchemaPlus;
 import org.apache.calcite.sql.SqlNode;
 import org.apache.calcite.tools.FrameworkConfig;
@@ -29,7 +27,6 @@
 import java.util.List;
 import java.util.Map;
 import java.util.Properties;
-import java.util.function.Consumer;
 
 import static org.assertj.core.api.Assertions.assertThat;
 
@@ -138,17 +135,14 @@ void sumRewritesToValuesFromZoneStats() throws Exception {
     }
 
     @Test
-    @SuppressWarnings("try") // the Hook.Closeable is used only for its scope (deregister on close)
     void pushDownRunsEndToEndThroughJdbcPlanner() throws Exception {
-        // Given a JDBC Calcite connection over the OHLC schema with the rule registered on the
-        // (Volcano) planner via Hook.PLANNER
+        // Given a plain JDBC Calcite connection over the OHLC schema — no rule is wired by hand; the
+        // VortexTableScan auto-registers the push-down rules when the planner first sees it
         Properties info = new Properties();
         info.setProperty("lex", "JAVA");
         String sql = "select min(low) lo, max(high) hi, count(*) c from vtx.ohlc";
 
-        try (Hook.Closeable ignored = Hook.PLANNER.addThread(
-                (Consumer<RelOptPlanner>) planner -> VortexAggregatePushDownRule.RULES.forEach(planner::addRule));
-             Connection conn = DriverManager.getConnection("jdbc:calcite:", info)) {
+        try (Connection conn = DriverManager.getConnection("jdbc:calcite:", info)) {
             conn.unwrap(CalciteConnection.class).getRootSchema()
                     .add("vtx", new VortexSchema(Map.of("ohlc", file)));
 
@@ -186,15 +180,12 @@ void pushDownRunsEndToEndThroughJdbcPlanner() throws Exception {
     }
 
     @Test
-    @SuppressWarnings("try") // the Hook.Closeable is used only for its scope (deregister on close)
     void filteredAggregateIsNotAnsweredFromWholeTableStats() throws Exception {
-        // Given the rule on the JDBC (Volcano) planner and a WHERE that pushes a predicate into the
-        // scan — whole-table stats must not be used to answer a filtered aggregate
+        // Given the auto-registered rule and a WHERE that pushes a predicate into the scan — whole-
+        // table stats must not be used to answer a filtered aggregate
         Properties info = new Properties();
         info.setProperty("lex", "JAVA");
-        try (Hook.Closeable ignored = Hook.PLANNER.addThread(
-                (Consumer<RelOptPlanner>) planner -> VortexAggregatePushDownRule.RULES.forEach(planner::addRule));
-             Connection conn = DriverManager.getConnection("jdbc:calcite:", info)) {
+        try (Connection conn = DriverManager.getConnection("jdbc:calcite:", info)) {
             conn.unwrap(CalciteConnection.class).getRootSchema()
                     .add("vtx", new VortexSchema(Map.of("ohlc", file)));
 
@@ -212,6 +203,42 @@ void filteredAggregateIsNotAnsweredFromWholeTableStats() throws Exception {
         }
     }
 
+    @Test
+    void avgRewritesViaSumAndCountWithNoScan() throws Exception {
+        // Given a plain JDBC connection — AVG must be reduced to SUM/COUNT by the auto-registered
+        // AggregateReduceFunctionsRule so both halves ride the zone-map push-down, leaving no scan
+        Properties info = new Properties();
+        info.setProperty("lex", "JAVA");
+        try (Connection conn = DriverManager.getConnection("jdbc:calcite:", info)) {
+            conn.unwrap(CalciteConnection.class).getRootSchema()
+                    .add("vtx", new VortexSchema(Map.of("ohlc", file)));
+
+            // When EXPLAIN runs over AVG of a non-null DOUBLE column
+            String plan;
+            try (Statement st = conn.createStatement();
+                 ResultSet rs = st.executeQuery("explain plan for select avg(low) a from vtx.ohlc")) {
+                rs.next();
+                plan = rs.getString(1);
+            }
+
+            // Then the plan decodes no data segment — the SUM and COUNT were both answered from stats
+            assertThat(plan).doesNotContain("TableScan");
+
+            // And the value equals sum/count folded from the zone-map stats (low is non-null, so
+            // COUNT = ROWS); AVG over a DOUBLE column is exact double division, no integer truncation
+            double avg;
+            try (Statement st = conn.createStatement();
+                 ResultSet rs = st.executeQuery("select avg(low) a from vtx.ohlc")) {
+                rs.next();
+                avg = rs.getDouble("a");
+            }
+            try (VortexReader reader = VortexReader.open(file)) {
+                double sum = VortexAggregates.of(reader, "low").sum().doubleValue();
+                assertThat(avg).isEqualTo(sum / ROWS);
+            }
+        }
+    }
+
     // Note: the absent-MIN/MAX-stat and absent-NULL_COUNT guards in VortexAggregatePushDownRule are
     // defensive for files whose writer omits per-column stats (e.g. some Rust-written columns or
     // future stat-less encodings). The in-house Java writer always emits min/max/null_count for

calcite/src/test/java/io/github/dfa1/vortex/calcite/OhlcSqlDemoTest.java

@@ -148,9 +148,12 @@ void filterPushDownPrunesChunksAndShowsInExplain() throws Exception {
             String windowed = "select count(*) c, max(high) h from vtx.ohlc "
                     + "where `date` between " + lo + " and " + hi;
 
-            // When the unfiltered query runs, every chunk is decoded
+            // When a full-scan baseline runs, every chunk is decoded. A `WHERE high > -1` matches
+            // every row (high is always positive) so zone maps prune nothing — and the predicate
+            // also stops the aggregate rule from answering count(*) from stats, which would
+            // otherwise decode zero chunks.
             try (Statement st = conn.createStatement();
-                 ResultSet rs = st.executeQuery("select count(*) c from vtx.ohlc")) {
+                 ResultSet rs = st.executeQuery("select count(*) c from vtx.ohlc where high > -1")) {
                 rs.next();
             }
             long chunksFull = schema.table("ohlc").chunksScannedLastQuery();