Skip to content

Commit 477a25e

Browse files
dfa1claude
dfa1
and
claude
committed
docs: add from-sql JDBC reader implementation plan
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent da16371 commit 477a25e

1 file changed

Lines changed: 174 additions & 0 deletions

File tree

‎docs/from-sql-plan.md‎

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
# Plan: `from-sql` command (JDBC reader)
2+
3+
## Core constraint: JDBC drivers cannot be in the fat jar
4+
5+
Fat jar is currently 1.7 MB. Bundling even one driver (PostgreSQL ~1.1 MB, H2 ~2.5 MB) bloats it.
6+
More importantly: users need different drivers, and MySQL Connector/J is GPL — cannot redistribute in MIT project.
7+
8+
## Driver distribution: lazy download from Maven Central
9+
10+
On first use, `from-sql` parses the URL prefix, looks up Maven coordinates in a built-in registry,
11+
downloads the JAR from Maven Central, verifies the SHA-1 checksum, and caches it in `~/.hosh/jdbc/`.
12+
Subsequent runs use the cached JAR — no download.
13+
14+
Flow:
15+
```
16+
from-sql "jdbc:mysql://localhost/db?user=root&password=x" "SELECT * FROM t"
17+
→ parse prefix "jdbc:mysql:"
18+
→ check ~/.hosh/jdbc/ for mysql driver JAR
19+
→ not found → look up registry → com.mysql:mysql-connector-j:9.3.0
20+
→ GET https://repo1.maven.org/maven2/com/mysql/mysql-connector-j/9.3.0/mysql-connector-j-9.3.0.jar
21+
→ verify SHA-1
22+
→ save to ~/.hosh/jdbc/mysql-connector-j-9.3.0.jar
23+
→ load driver, connect, run query
24+
→ next run: JAR cached, skip download
25+
```
26+
27+
Download uses `java.net.http.HttpClient` (JDK 11+) — no new prod dependencies.
28+
29+
**Override:** check `State.getVariables()` for `JDBC_DRIVERS_DIR` to use a non-default cache dir.
30+
31+
**Version override:** hosh variable `JDBC_DRIVER_MYSQL=com.mysql:mysql-connector-j:8.0.33` pins a
32+
specific version instead of the registry default.
33+
34+
**Manual JAR:** if a matching JAR already exists in `~/.hosh/jdbc/`, skip download entirely.
35+
36+
**Unknown prefix:** error — "unknown JDBC URL prefix; place driver JAR in ~/.hosh/jdbc/"
37+
38+
### Built-in registry
39+
40+
| URL prefix | Maven coordinates |
41+
|---|---|
42+
| `jdbc:postgresql:` | `org.postgresql:postgresql:42.7.3` |
43+
| `jdbc:mysql:` | `com.mysql:mysql-connector-j:9.3.0` |
44+
| `jdbc:mariadb:` | `org.mariadb.jdbc:mariadb-java-client:3.5.2` |
45+
| `jdbc:sqlite:` | `org.xerial:sqlite-jdbc:3.45.3.0` |
46+
| `jdbc:h2:` | `com.h2database:h2:2.3.232` |
47+
| `jdbc:sqlserver:` | `com.microsoft.sqlserver:mssql-jdbc:12.10.0.jre11` |
48+
| `jdbc:oracle:thin:` | `com.oracle.database.jdbc:ojdbc11:23.7.0.25.01` |
49+
50+
Oracle note: ojdbc is on Maven Central today; was historically behind a paywall.
51+
52+
## Multiple versions in cache
53+
54+
If `~/.hosh/jdbc/` contains multiple JARs matching the same vendor prefix (e.g. both
55+
`mysql-connector-j-8.0.33.jar` and `mysql-connector-j-9.3.0.jar`), two options are under
56+
consideration:
57+
58+
### Option A: Fail with helpful error (preferred)
59+
60+
```
61+
error: multiple MySQL drivers found in ~/.hosh/jdbc/:
62+
mysql-connector-j-8.0.33.jar
63+
mysql-connector-j-9.3.0.jar
64+
set JDBC_DRIVER_MYSQL=com.mysql:mysql-connector-j:9.3.0 to pin a version
65+
```
66+
67+
Forces the user to decide explicitly. Avoids silent surprises — schema differences, SSL behavior,
68+
and protocol changes between driver major versions make silent version selection dangerous.
69+
70+
### Option B: Use latest by version sort
71+
72+
Scan for matching JARs, pick highest version via semver comparison. Simple, but the user may not
73+
know which version is actually running.
74+
75+
Option A is preferred for v1. Option B can be reconsidered if the explicit error proves too noisy
76+
in practice.
77+
78+
## Why not `DriverManager.getConnection()`
79+
80+
`DriverManager` checks that the driver was loaded by the calling classloader or an ancestor.
81+
A `URLClassLoader` child fails this check. Must use `driver.connect(url, props)` directly —
82+
the JDBC spec allows this and bypasses the classloader restriction.
83+
84+
## Command API
85+
86+
```
87+
from-sql <jdbc-url> <sql-query>
88+
```
89+
90+
Examples:
91+
92+
```
93+
from-sql "jdbc:postgresql://localhost/mydb?user=alice&password=s3cr3t" "SELECT id, name FROM users"
94+
from-sql "jdbc:sqlite:/tmp/data.db" "SELECT * FROM log WHERE ts > '2026-01-01'"
95+
from-sql "jdbc:h2:mem:test" "SELECT 1 AS n"
96+
```
97+
98+
Credentials live in the URL — consistent with every JDBC CLI tool. Users use hosh variables to
99+
avoid plaintext in scripts:
100+
101+
```
102+
from-sql $DB_URL "SELECT * FROM users"
103+
```
104+
105+
## Module: `modules/jdbc`
106+
107+
**Prod deps:** zero — only JDK JDBC (`java.sql`) and `java.net.http` for downloads.
108+
109+
**Test deps:** H2 database (`com.h2database:h2`, EPL-2.0/MPL-2.0) as `test` scope. Runs fully
110+
in-memory — no external infrastructure needed for tests.
111+
112+
`module-info.java`:
113+
114+
```java
115+
module hosh.modules.jdbc {
116+
requires hosh.spi;
117+
requires java.sql;
118+
requires java.net.http;
119+
}
120+
```
121+
122+
## Driver loading sequence in `run()`
123+
124+
```
125+
1. resolve driversDir:
126+
- check state.getVariables().get(VariableName.of("JDBC_DRIVERS_DIR"))
127+
- fallback: Path.of(System.getProperty("user.home"), ".hosh", "jdbc")
128+
2. parse URL prefix → look up registry
129+
3. if unknown prefix → err "unknown JDBC URL prefix; place driver JAR in <driversDir>"
130+
4. scan driversDir for JARs matching vendor
131+
5. if multiple found → err with list + pin instruction (Option A)
132+
6. if none found → download from Maven Central + verify SHA-1 → save to driversDir
133+
7. URLClassLoader driverLoader = new URLClassLoader(new URL[]{jarPath}, currentClassLoader)
134+
8. ServiceLoader.load(Driver.class, driverLoader) → find driver accepting url
135+
9. driver.connect(url, new Properties()) → Connection
136+
10. conn.prepareStatement(query).executeQuery() → ResultSet
137+
11. ResultSetMetaData → Keys, rows → Records
138+
```
139+
140+
## ResultSet → Records type mapping
141+
142+
| SQL type | hosh Value |
143+
|---|---|
144+
| `VARCHAR`, `CHAR`, `CLOB` | `Values.ofText()` |
145+
| `INTEGER`, `SMALLINT`, `TINYINT` | `Values.ofNumeric()` |
146+
| `BIGINT` | `Values.ofNumeric()` |
147+
| `FLOAT`, `REAL`, `DOUBLE` | `Values.ofText(Double.toString())` |
148+
| `NUMERIC`, `DECIMAL` | `Values.ofText(bd.toPlainString())` |
149+
| `BOOLEAN`, `BIT` | `Values.ofText("true"/"false")` |
150+
| `DATE`, `TIMESTAMP` | `Values.ofInstant()` |
151+
| `BINARY`, `VARBINARY`, `BLOB` | `Values.ofBytes()` |
152+
| `NULL` / `rs.wasNull()` | `Values.none()` |
153+
154+
## Tests
155+
156+
**Unit tests** (Mockito, no DB): missing args, too many args, unknown URL prefix, multiple JARs
157+
in cache, download failure (mocked HTTP).
158+
159+
**Integration tests** (H2 in test scope): full round-trip — create in-memory H2 DB, insert rows,
160+
`from-sql` reads them, verify Records match. Tests all type mappings.
161+
162+
## Open questions
163+
164+
1. **`to-sql`?** Insert records pipeline into a table. Needs `PreparedStatement` with dynamic
165+
column binding. Plan for later.
166+
167+
2. **Connection per query vs pool?** `from-sql` is a one-shot CLI command. Single connection,
168+
open/close per invocation is correct.
169+
170+
3. **Credentials security:** URL approach means credentials appear in `ps aux` and hosh history.
171+
Future: support password from a hosh variable or `~/.hosh/jdbc.properties`. Out of scope for v1.
172+
173+
4. **Streaming vs collect-all?** `ResultSet.next()` is lazy — stream directly to `out.send()`
174+
without buffering. Correct for large result sets.

0 commit comments

Comments
 (0)