Skip to content

Commit d940ca6

Browse files
dfa1claude
andcommitted
feat(calcite): add VortexCalcite.connect with Babel parser for reserved-word columns
The adapter emits no SQL of its own, so it has nothing to quote — the reserved-word problem (a column named `close`/`open`/`value` breaking unquoted queries) is owned by the connection's parser, not the adapter. VortexCalcite.connect(schemaName, tables) is a one-call JDBC entry point that registers the VortexSchema and wires Calcite's Babel parser, which accepts SQL reserved words as identifiers. `select close, open from vtx.ohlc` now parses unquoted — the common case for columnar files. The typed-literal keywords (date/time/timestamp/interval) stay reserved even under Babel (they begin a literal, e.g. DATE '2020-01-01') and still need back-tick quoting; that residual is documented and tested. calcite-babel adds no transitive dependencies beyond calcite-core. OhlcSqlDemoTest.connect() now delegates to the helper, dogfooding it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent d7ecd98 commit d940ca6

6 files changed

Lines changed: 189 additions & 8 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
99

1010
### Added
1111

12+
- `VortexCalcite.connect(schemaName, tables)` opens a Calcite JDBC connection with the `VortexSchema` registered in one call, folding away the `DriverManager` / `unwrap` / `getRootSchema().add(...)` boilerplate. It wires the **Babel** SQL parser, so columns whose names are reserved words (`close`, `open`, `value`, `year`, …) are queryable **unquoted**`select close, open from vtx.ohlc` — which columnar files routinely need; only the typed-literal keywords (`date`, `time`, `timestamp`, `interval`) still require back-tick quoting. ([24b64b32](https://github.com/dfa1/vortex-java/commit/24b64b32))
1213
- The Calcite aggregate push-down rule now **auto-registers** over a bare `jdbc:calcite:` connection: a `VortexTable` translates to a `VortexTableScan` whose `register()` installs the rules when the planner first sees it, so `SELECT MIN/MAX/COUNT/SUM` over a `VortexSchema` is answered from zone-map statistics with no caller wiring (previously the rule had to be attached to the planner by hand). `AVG` joins them — `AggregateReduceFunctionsRule` reduces it to `SUM`/`COUNT`, both of which push down, so a whole-table `AVG` also decodes no data segment. Column projection and `WHERE` chunk-skip push-down are unchanged. ([24b64b32](https://github.com/dfa1/vortex-java/commit/24b64b32))
1314
- The Calcite aggregate push-down rule now rewrites a whole-table `SUM(col)` to a single-row `Values` computed from the zone-map table (via `VortexTable.zoneSum`), so `SELECT SUM(col)` answers metadata-only with no data segment decoded — joining the existing `MIN`/`MAX`/`COUNT` push-down. `SUM` over an all-null (or empty) column answers the SQL `NULL`, and the rule abandons to a normal scan when a zone carries no usable sum (no zone map, or an overflowed zone) or when a `WHERE` filters the aggregate. ([24b64b32](https://github.com/dfa1/vortex-java/commit/24b64b32))
1415
- `ScanIterator.columnZoneStats(column)` surfaces per-zone min/max/sum/null-count from a column's `vortex.stats` zone-map table without decoding any data segment — the read side of aggregate push-down (ADR 0013 §6). `ArrayStats` gains a `sum` component, decoded from the zone-map table (where the Rust reference stores it too), so the Calcite adapter now answers `SUM`/`AVG` metadata-only when every zone carries a sum, falling back to a streaming scan only for columns without a zone map. ([05dd9204](https://github.com/dfa1/vortex-java/commit/05dd9204))

calcite/pom.xml

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,13 @@
2727
<artifactId>calcite-core</artifactId>
2828
<version>${calcite.version}</version>
2929
</dependency>
30+
<!-- Babel SQL parser: lets reserved-word column names (date, close, open, …) be queried
31+
unquoted through VortexCalcite.connect. Adds no transitive deps beyond calcite-core. -->
32+
<dependency>
33+
<groupId>org.apache.calcite</groupId>
34+
<artifactId>calcite-babel</artifactId>
35+
<version>${calcite.version}</version>
36+
</dependency>
3037
<!-- testing -->
3138
<dependency>
3239
<groupId>io.github.dfa1.vortex</groupId>
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
package io.github.dfa1.vortex.calcite;
2+
3+
import org.apache.calcite.jdbc.CalciteConnection;
4+
import org.apache.calcite.schema.SchemaPlus;
5+
import org.apache.calcite.sql.parser.babel.SqlBabelParserImpl;
6+
7+
import java.nio.file.Path;
8+
import java.sql.Connection;
9+
import java.sql.DriverManager;
10+
import java.sql.SQLException;
11+
import java.util.Map;
12+
import java.util.Objects;
13+
import java.util.Properties;
14+
15+
/// Entry point for querying Vortex files with SQL through Apache Calcite.
16+
///
17+
/// [#connect(String, Map)] opens a Calcite JDBC connection with a [VortexSchema] already
18+
/// registered, folding the `DriverManager` + `unwrap` + `getRootSchema().add(...)` boilerplate into
19+
/// one call:
20+
///
21+
/// ```java
22+
/// try (Connection conn = VortexCalcite.connect("vtx", Map.of("ohlc", path));
23+
/// Statement st = conn.createStatement();
24+
/// ResultSet rs = st.executeQuery("select min(low), max(high), sum(volume) from vtx.ohlc")) {
25+
/// // whole-table aggregates are answered from zone-map statistics, no data segment decoded
26+
/// }
27+
/// ```
28+
///
29+
/// The connection uses Calcite's `JAVA` lexical policy (case-sensitive identifiers, back-tick
30+
/// quoting, unquoted case preserved) together with the **Babel** SQL parser, which accepts most SQL
31+
/// reserved words as identifiers. A column whose name is a reserved word — `close`, `open`, `value`,
32+
/// `year`, … — can therefore be queried **unquoted**: `select close, open from vtx.ohlc`, which
33+
/// columnar files routinely need.
34+
///
35+
/// The exception is the handful of keywords that begin a typed literal — `date`, `time`,
36+
/// `timestamp`, `interval` — which stay reserved even under Babel (`date` opens `DATE '2020-01-01'`).
37+
/// A column with one of those names must still be back-tick quoted: `select \`date\` from vtx.ohlc`.
38+
public final class VortexCalcite {
39+
40+
private VortexCalcite() {
41+
}
42+
43+
/// Opens a Calcite JDBC connection exposing `tables` under a schema named `schemaName`.
44+
///
45+
/// Each map entry binds a SQL table name to the `.vortex` file backing it; the tables are
46+
/// reachable as `schemaName.tableName`. The returned connection is the caller's to close, which
47+
/// releases the schema's readers. If schema registration fails the connection is closed before
48+
/// the failure propagates, so no connection leaks.
49+
///
50+
/// @param schemaName the schema name the tables are registered under
51+
/// @param tables map from SQL table name to the `.vortex` file path
52+
/// @return an open Calcite JDBC connection with the Vortex schema registered
53+
/// @throws SQLException if the JDBC connection cannot be opened or the schema cannot be registered
54+
public static Connection connect(String schemaName, Map<String, Path> tables) throws SQLException {
55+
Objects.requireNonNull(schemaName, "schemaName");
56+
Objects.requireNonNull(tables, "tables");
57+
Properties info = new Properties();
58+
// JAVA lex: case-sensitive identifiers, back-tick quoting, unquoted case preserved — so an
59+
// unquoted `ohlc` matches the registered table name exactly.
60+
info.setProperty("lex", "JAVA");
61+
// Babel parser: accept SQL reserved words (date, close, open, value, …) as identifiers, so
62+
// columnar files whose column names collide with keywords are queryable unquoted.
63+
info.setProperty("parserFactory", SqlBabelParserImpl.class.getName() + "#FACTORY");
64+
Connection connection = DriverManager.getConnection("jdbc:calcite:", info);
65+
try {
66+
SchemaPlus root = connection.unwrap(CalciteConnection.class).getRootSchema();
67+
root.add(schemaName, new VortexSchema(tables));
68+
return connection;
69+
} catch (SQLException | RuntimeException e) {
70+
try {
71+
connection.close();
72+
} catch (SQLException closeFailure) {
73+
e.addSuppressed(closeFailure);
74+
}
75+
throw e;
76+
}
77+
}
78+
}

calcite/src/test/java/io/github/dfa1/vortex/calcite/OhlcSqlDemoTest.java

Lines changed: 1 addition & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -195,12 +195,7 @@ void filterPushDownPrunesChunksAndShowsInExplain() throws Exception {
195195
}
196196

197197
private static Connection connect() throws Exception {
198-
Properties info = new Properties();
199-
info.setProperty("lex", "JAVA");
200-
Connection conn = DriverManager.getConnection("jdbc:calcite:", info);
201-
conn.unwrap(CalciteConnection.class).getRootSchema()
202-
.add("vtx", new VortexSchema(Map.of("ohlc", file)));
203-
return conn;
198+
return VortexCalcite.connect("vtx", Map.of("ohlc", file));
204199
}
205200

206201
private static SqlAggs runSql(Connection conn) throws Exception {
Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
package io.github.dfa1.vortex.calcite;
2+
3+
import org.junit.jupiter.api.BeforeAll;
4+
import org.junit.jupiter.api.Test;
5+
import org.junit.jupiter.api.io.TempDir;
6+
7+
import java.nio.file.Path;
8+
import java.sql.Connection;
9+
import java.sql.ResultSet;
10+
import java.sql.Statement;
11+
import java.util.Map;
12+
13+
import static org.assertj.core.api.Assertions.assertThat;
14+
import static org.assertj.core.api.Assertions.assertThatNullPointerException;
15+
16+
/// Covers [VortexCalcite#connect(String, Map)] — the one-call JDBC entry point: it registers the
17+
/// schema, the aggregate push-down rule auto-fires through it, and a reserved-word column is
18+
/// queryable when back-tick quoted.
19+
class VortexCalciteTest {
20+
21+
@TempDir
22+
static Path tmp;
23+
private static Path file;
24+
25+
@BeforeAll
26+
static void writeFile() throws Exception {
27+
file = tmp.resolve("ohlc.vortex");
28+
// Small two-zone file; the OHLC schema includes reserved-word columns (date, open, close).
29+
OhlcGenerator.write(file, 20_000, 10_000);
30+
}
31+
32+
@Test
33+
void connect_registersSchemaAndPushesAggregateWithNoScan() throws Exception {
34+
// Given a connection opened through the helper (no manual DriverManager/unwrap/add)
35+
try (Connection conn = VortexCalcite.connect("vtx", Map.of("ohlc", file))) {
36+
37+
// When a whole-table SUM is explained — the rule must auto-register via VortexTableScan
38+
String plan;
39+
try (Statement st = conn.createStatement();
40+
ResultSet rs = st.executeQuery("explain plan for select sum(volume) from vtx.ohlc")) {
41+
rs.next();
42+
plan = rs.getString(1);
43+
}
44+
45+
// Then it is answered from zone-map stats — no data segment decoded
46+
assertThat(plan).doesNotContain("TableScan");
47+
}
48+
}
49+
50+
@Test
51+
void connect_reservedWordColumnsAreQueryableUnquoted() throws Exception {
52+
// Given the OHLC table has columns named close and open — both SQL reserved words (CLOSE /
53+
// OPEN cursor) that the stock parser rejects unquoted
54+
try (Connection conn = VortexCalcite.connect("vtx", Map.of("ohlc", file))) {
55+
56+
// When they are selected UNQUOTED — the Babel parser accepts reserved words as identifiers
57+
long rows = 0;
58+
try (Statement st = conn.createStatement();
59+
ResultSet rs = st.executeQuery("select close, open from vtx.ohlc")) {
60+
while (rs.next()) {
61+
rows++;
62+
}
63+
}
64+
65+
// Then the query runs and returns every row, no back-ticks needed
66+
assertThat(rows).isEqualTo(20_000);
67+
}
68+
}
69+
70+
@Test
71+
void connect_typeKeywordColumnStillNeedsQuoting() throws Exception {
72+
// Given the `date` column: even Babel keeps DATE reserved (it begins a date literal,
73+
// DATE '2020-01-01'), so it is the residual case that still needs back-ticks
74+
try (Connection conn = VortexCalcite.connect("vtx", Map.of("ohlc", file))) {
75+
76+
// When `date` is back-tick quoted
77+
long rows = 0;
78+
try (Statement st = conn.createStatement();
79+
ResultSet rs = st.executeQuery("select `date` from vtx.ohlc")) {
80+
while (rs.next()) {
81+
rows++;
82+
}
83+
}
84+
85+
// Then it resolves to the column and returns every row
86+
assertThat(rows).isEqualTo(20_000);
87+
}
88+
}
89+
90+
@Test
91+
void connect_nullArguments_throw() {
92+
// Given / When / Then — both arguments are required
93+
assertThatNullPointerException().isThrownBy(() -> VortexCalcite.connect(null, Map.of()));
94+
assertThatNullPointerException().isThrownBy(() -> VortexCalcite.connect("vtx", null));
95+
}
96+
}

docs/adr/0018-calcite-sql-adapter.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -122,8 +122,12 @@ Phases 0–2 are implemented and tested:
122122

123123
Gotchas found and recorded for the production adapter:
124124

125-
- **Reserved words.** Beyond `date`, the OHLC columns `close`/`open` (CLOSE/OPEN cursor) must be
126-
quoted. A production adapter should quote all identifiers it emits.
125+
- **Reserved words.** Column names that are SQL keywords (`close`/`open`, `value`, `year`, …)
126+
break unquoted queries under the stock parser. Resolved by `VortexCalcite.connect`, which wires
127+
the Babel parser (`calcite-babel`) so reserved words parse as identifiers unquoted — the adapter
128+
emits no SQL of its own, so there is nothing for it to quote; the lever is the parser, owned by
129+
the connection. The residual exceptions are the typed-literal keywords (`date`, `time`,
130+
`timestamp`, `interval`), which stay reserved even under Babel and still need back-tick quoting.
127131
- **Integer stats are `Long`.** Zone-map integer stats decode as `Long`, floats as `Double`,
128132
regardless of column width. A filter literal boxed at the column's natural width (`Integer` for
129133
`I32`) silently disables pruning because `ScanIterator.compareValues` swallows the resulting

0 commit comments

Comments
 (0)