Skip to content

Commit 350e956

Browse files
anandgupta42claude
anandgupta42
and
claude
committed
feat: add ClickHouse warehouse driver with full integration
Add first-class ClickHouse support as the 12th database driver: **Driver (`packages/drivers/src/clickhouse.ts`):** - Official `@clickhouse/client` over HTTP(S) - Supports ClickHouse server 23.3+ (all non-EOL versions) - Password, connection string, and TLS/mTLS auth - ClickHouse Cloud and self-hosted compatible - Parameterized queries for SQL injection prevention - DML-aware LIMIT injection (won't break `WITH...INSERT`) **Integration (23 touchpoints):** - Registry: `DRIVER_MAP`, import switch, `PASSWORD_DRIVERS` - Discovery: Docker containers, env vars (`CLICKHOUSE_HOST`/`CLICKHOUSE_URL`), dbt profiles (`ADAPTER_TYPE_MAP`), dbt lineage dialect - FinOps: `system.query_log` query history template - Normalization: aliases for `connectionString`, `requestTimeout`, TLS fields - Publish: `@clickhouse/client` in `peerDependencies` **Tests:** - 30+ E2E tests across 5 suites (latest, LTS 23.8, 24.3, 24.8, connection string) - 14 config normalization tests for all ClickHouse aliases - MergeTree variants, materialized views, Nullable columns, Array/Map/IPv4 types **Documentation:** - Full config section in `warehouses.md` (standard, Cloud, connection string) - Support matrix entry in `drivers.md` with auth methods - Dedicated guide (`guides/clickhouse.md`): MergeTree optimization, materialized view pipelines, dialect translation, LowCardinality tips, dbt integration - Updated README, getting-started, warehouse-tools docs **Engineering:** - `packages/drivers/ADDING_A_DRIVER.md` — 23-point checklist for adding future drivers - `.claude/commands/add-database-driver.md` — Claude skill to automate the process Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent d66354c commit 350e956

26 files changed

Lines changed: 1826 additions & 82 deletions

File tree

‎.claude/commands/add-database-driver.md‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
---
2+
description: Add a new database driver to Altimate Code. Scaffolds the driver, registers it across all 23 integration points, writes E2E tests, and updates docs. Usage - /add-database-driver <database-name>
3+
---
4+
5+
# Add Database Driver
6+
7+
Scaffold and fully integrate a new database/warehouse driver into Altimate Code. This command handles all 23 integration points — driver code, registry, discovery, finops, tests, and documentation.
8+
9+
## Input
10+
11+
`$ARGUMENTS` = the database name (e.g., `cockroachdb`, `timescaledb`, `cassandra`, `neo4j`).
12+
13+
If empty, ask: "Which database should I add support for?"
14+
15+
## Step 0: Research
16+
17+
Before writing any code, research the database:
18+
19+
1. **Find the official Node.js/TypeScript client package** on npm. Search for `@{database}/client`, `{database}-js`, or similar.
20+
2. **Check supported server versions** — which versions are not EOL?
21+
3. **Identify auth methods** — password, token, TLS/certificate, connection string, cloud-specific?
22+
4. **Check SQL dialect** — standard SQL? Custom syntax? LIMIT vs TOP vs FETCH FIRST? System tables for schemas/tables/columns?
23+
5. **Find Docker image** — official image on Docker Hub for E2E testing?
24+
6. **Check if dbt adapter exists** — search for `dbt-{database}` on PyPI.
25+
26+
Present findings to the user before proceeding:
27+
```
28+
## Research: {Database}
29+
30+
- **npm package**: `{package}` (v{version})
31+
- **Server versions**: {non-EOL versions}
32+
- **Auth methods**: {list}
33+
- **SQL dialect**: {notes on LIMIT, system tables, parameterized queries}
34+
- **Docker image**: `{image}:{tag}`
35+
- **dbt adapter**: {exists/not found}
36+
37+
Proceed with implementation?
38+
```
39+
40+
## Step 1: Read Reference Document
41+
42+
Read the comprehensive checklist:
43+
```bash
44+
cat packages/drivers/ADDING_A_DRIVER.md
45+
```
46+
47+
This document has all 23 integration points with exact file paths and code patterns.
48+
49+
## Step 2: Read Existing Driver for Pattern
50+
51+
Read a similar existing driver as a template. Choose based on database type:
52+
53+
- **SQL database with password auth** → read `packages/drivers/src/mysql.ts`
54+
- **Cloud warehouse with token auth** → read `packages/drivers/src/databricks.ts`
55+
- **Database with connection string support** → read `packages/drivers/src/postgres.ts`
56+
- **HTTP-based client** → read `packages/drivers/src/clickhouse.ts`
57+
- **Document database (non-SQL)** → read `packages/drivers/src/mongodb.ts`
58+
59+
Also read:
60+
- `packages/drivers/src/normalize.ts` — for alias pattern
61+
- `packages/opencode/src/altimate/native/connections/registry.ts` — for registration pattern
62+
- `packages/opencode/test/altimate/drivers-docker-e2e.test.ts` — for E2E test pattern
63+
64+
## Step 3: Implement (23 integration points)
65+
66+
Work through all 9 phases from the checklist. Use parallel edits where possible.
67+
68+
### Phase 1: Core Driver (4 files)
69+
70+
1. **Create `packages/drivers/src/{database}.ts`**
71+
- Follow the Connector interface: `connect()`, `execute()`, `listSchemas()`, `listTables()`, `describeTable()`, `close()`
72+
- Lazy-import the npm package
73+
- Use parameterized queries for schema introspection
74+
- Handle LIMIT injection with DML guard: `!hasDML` check before appending LIMIT
75+
- Handle TLS detection from connection strings
76+
77+
2. **Add export to `packages/drivers/src/index.ts`**
78+
79+
3. **Add optionalDependency to `packages/drivers/package.json`**
80+
81+
4. **Add aliases to `packages/drivers/src/normalize.ts`**
82+
83+
### Phase 2: Registry (4 files in registry.ts)
84+
85+
5. Add to `DRIVER_MAP`
86+
6. Add to import switch statement
87+
7. Add to `PASSWORD_DRIVERS` (if applicable)
88+
8. Remove from `KNOWN_UNSUPPORTED` (if listed)
89+
90+
### Phase 3: Discovery (4 files)
91+
92+
9. Docker discovery — `docker-discovery.ts` (IMAGE_MAP, ENV_MAP, DEFAULT_PORTS, DEFAULT_USERS)
93+
10. Env var detection — `project-scan.ts` (detectEnvVars warehouses array)
94+
11. dbt adapter — `dbt-profiles.ts` (ADAPTER_TYPE_MAP)
95+
12. dbt lineage — `dbt/lineage.ts` (detectDialect dialectMap)
96+
97+
### Phase 4: FinOps (1 file)
98+
99+
13. Query history — `finops/query-history.ts` (SQL template + handler if database has system query log)
100+
101+
### Phase 5: Build (1 file)
102+
103+
14. Peer deps — `script/publish.ts` (driverPeerDependencies)
104+
105+
### Phase 6: Tool Descriptions (1 file)
106+
107+
15. warehouse_add — `tools/warehouse-add.ts` (config description + error message)
108+
109+
### Phase 7: Tests (2 new files + 1 edit)
110+
111+
16. E2E tests — `test/altimate/drivers-{database}-e2e.test.ts`
112+
17. Normalization tests — add to `test/altimate/driver-normalize.test.ts`
113+
18. Verify existing tests pass
114+
115+
### Phase 8: Documentation (5 files)
116+
117+
19. `docs/docs/configure/warehouses.md` — config section + update count
118+
20. `docs/docs/drivers.md` — support matrix + installation + auth + update count
119+
21. `docs/docs/data-engineering/tools/warehouse-tools.md` — env vars + Docker
120+
22. `README.md` — warehouse list
121+
23. `docs/docs/getting-started/index.md` — homepage list
122+
123+
### Phase 9: Optional
124+
125+
- Guide page at `docs/docs/data-engineering/guides/{database}.md`
126+
- Update `mkdocs.yml` nav and `guides/index.md`
127+
- Check fingerprint regex in `fingerprint/index.ts`
128+
129+
## Step 4: Run Quality Gates
130+
131+
```bash
132+
# Tests (from packages/opencode/)
133+
cd packages/opencode && bun test test/altimate/driver-normalize.test.ts test/altimate/connections.test.ts
134+
135+
# Typecheck
136+
bun turbo typecheck
137+
138+
# Marker check
139+
bun run script/upstream/analyze.ts --markers --base main --strict
140+
```
141+
142+
All three must pass before proceeding.
143+
144+
## Step 5: Run Code Review
145+
146+
Run `/consensus:code-review` to get the implementation reviewed by multiple models before committing.
147+
148+
## Step 6: Summary
149+
150+
Present final summary:
151+
```
152+
## {Database} Driver Added
153+
154+
### Files Created
155+
- packages/drivers/src/{database}.ts
156+
- packages/opencode/test/altimate/drivers-{database}-e2e.test.ts
157+
- docs/docs/data-engineering/guides/{database}.md (if created)
158+
159+
### Files Modified
160+
- {list all modified files}
161+
162+
### Test Results
163+
- {N} normalization tests pass
164+
- {N} connection tests pass
165+
- Typecheck: pass
166+
- Marker check: pass
167+
168+
### E2E Test Coverage
169+
- {list of test suites and server versions}
170+
171+
Ready to commit.
172+
```
173+
174+
## Rules
175+
176+
1. **Read before writing.** Always read existing drivers and the reference doc before creating new code.
177+
2. **Don't skip integration points.** All 23 points exist for a reason — missing one causes inconsistencies users will hit.
178+
3. **Use parameterized queries** for `listTables` and `describeTable` — never interpolate user input into SQL.
179+
4. **Test multiple server versions** — at minimum: latest stable + oldest non-EOL LTS.
180+
5. **Run all quality gates** before presenting the summary.
181+
6. **Don't modify finops tools** (credit-analyzer, warehouse-advisor, unused-resources) unless the database has equivalent cost/credit APIs.

‎.github/meta/commit.txt‎

Lines changed: 32 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,37 @@
1-
ci: add Verdaccio sanity suite to CI and release workflows
1+
feat: add ClickHouse warehouse driver with full integration
22

3-
Adds the Verdaccio-based sanity suite (real `npm install -g` flow)
4-
to both CI and release pipelines:
3+
Add first-class ClickHouse support as the 12th database driver:
54

6-
**CI (`ci.yml`):**
7-
- New `sanity-verdaccio` job on push to main
8-
- Builds linux-x64 binary + dbt-tools, runs full Docker Compose suite
9-
- Independent of other jobs (doesn't block PRs)
5+
**Driver (`packages/drivers/src/clickhouse.ts`):**
6+
- Official `@clickhouse/client` over HTTP(S)
7+
- Supports ClickHouse server 23.3+ (all non-EOL versions)
8+
- Password, connection string, and TLS/mTLS auth
9+
- ClickHouse Cloud and self-hosted compatible
10+
- Parameterized queries for SQL injection prevention
11+
- DML-aware LIMIT injection (won't break `WITH...INSERT`)
1012

11-
**Release (`release.yml`):**
12-
- New `sanity-verdaccio` job between build and npm publish
13-
- Downloads linux-x64 artifact from build matrix
14-
- **Blocks `publish-npm`** — broken install flow prevents release
15-
- Dependency chain: build → sanity-verdaccio → publish-npm → github-release
13+
**Integration (23 touchpoints):**
14+
- Registry: `DRIVER_MAP`, import switch, `PASSWORD_DRIVERS`
15+
- Discovery: Docker containers, env vars (`CLICKHOUSE_HOST`/`CLICKHOUSE_URL`),
16+
dbt profiles (`ADAPTER_TYPE_MAP`), dbt lineage dialect
17+
- FinOps: `system.query_log` query history template
18+
- Normalization: aliases for `connectionString`, `requestTimeout`, TLS fields
19+
- Publish: `@clickhouse/client` in `peerDependencies`
20+
21+
**Tests:**
22+
- 30+ E2E tests across 5 suites (latest, LTS 23.8, 24.3, 24.8, connection string)
23+
- 14 config normalization tests for all ClickHouse aliases
24+
- MergeTree variants, materialized views, Nullable columns, Array/Map/IPv4 types
25+
26+
**Documentation:**
27+
- Full config section in `warehouses.md` (standard, Cloud, connection string)
28+
- Support matrix entry in `drivers.md` with auth methods
29+
- Dedicated guide (`guides/clickhouse.md`): MergeTree optimization, materialized
30+
view pipelines, dialect translation, LowCardinality tips, dbt integration
31+
- Updated README, getting-started, warehouse-tools docs
32+
33+
**Engineering:**
34+
- `packages/drivers/ADDING_A_DRIVER.md` — 23-point checklist for adding future drivers
35+
- `.claude/commands/add-database-driver.md` — Claude skill to automate the process
1636

1737
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -151,7 +151,7 @@ Each mode has scoped permissions, tool access, and SQL write-access control.
151151
152152
## Supported Warehouses
153153
154-
Snowflake · BigQuery · Databricks · PostgreSQL · Redshift · DuckDB · MySQL · SQL Server · Oracle · SQLite
154+
Snowflake · BigQuery · Databricks · PostgreSQL · Redshift · ClickHouse · DuckDB · MySQL · SQL Server · Oracle · SQLite · MongoDB
155155
156156
First-class support with schema indexing, query execution, and metadata introspection. SSH tunneling available for secure connections.
157157

‎bun.lock‎

Lines changed: 5 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/docs/configure/warehouses.md‎

Lines changed: 61 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Warehouses
22

3-
Altimate Code connects to 9 warehouse types. Configure them in `.altimate-code/connections.json` (project-local) or `~/.altimate-code/connections.json` (global).
3+
Altimate Code connects to 10 warehouse types. Configure them in `.altimate-code/connections.json` (project-local) or `~/.altimate-code/connections.json` (global).
44

55
## Configuration
66

@@ -282,6 +282,66 @@ If you're already authenticated via `gcloud`, omit `credentials_path`:
282282
!!! info "Server compatibility"
283283
The MongoDB driver (v6.x) supports MongoDB server versions 3.6 through 8.0, covering all releases from the last 3+ years.
284284

285+
## ClickHouse
286+
287+
```json
288+
{
289+
"clickhouse-prod": {
290+
"type": "clickhouse",
291+
"host": "localhost",
292+
"port": 8123,
293+
"database": "analytics",
294+
"user": "default",
295+
"password": "{env:CLICKHOUSE_PASSWORD}"
296+
}
297+
}
298+
```
299+
300+
| Field | Required | Description |
301+
|-------|----------|-------------|
302+
| `connection_string` | No | Full URL (alternative to individual fields, e.g. `http://user:pass@host:8123`) |
303+
| `host` | No | Hostname (default: `localhost`) |
304+
| `port` | No | HTTP port (default: `8123`) |
305+
| `database` | No | Database name (default: `default`) |
306+
| `user` | No | Username (default: `default`) |
307+
| `password` | No | Password |
308+
| `protocol` | No | `http` or `https` (default: `http`) |
309+
| `request_timeout` | No | Request timeout in ms (default: `30000`) |
310+
| `tls_ca_cert` | No | Path to CA certificate for TLS |
311+
| `tls_cert` | No | Path to client certificate for mutual TLS |
312+
| `tls_key` | No | Path to client key for mutual TLS |
313+
| `clickhouse_settings` | No | Object of ClickHouse server settings |
314+
315+
### ClickHouse Cloud
316+
317+
```json
318+
{
319+
"clickhouse-cloud": {
320+
"type": "clickhouse",
321+
"host": "abc123.us-east-1.aws.clickhouse.cloud",
322+
"port": 8443,
323+
"protocol": "https",
324+
"user": "default",
325+
"password": "{env:CLICKHOUSE_CLOUD_PASSWORD}",
326+
"database": "default"
327+
}
328+
}
329+
```
330+
331+
### Using a connection string
332+
333+
```json
334+
{
335+
"clickhouse-prod": {
336+
"type": "clickhouse",
337+
"connection_string": "https://default:secret@my-ch.cloud:8443"
338+
}
339+
}
340+
```
341+
342+
!!! info "Server compatibility"
343+
The ClickHouse driver supports ClickHouse server versions 23.3 and later, covering all non-EOL releases. This includes LTS releases 23.8, 24.3, 24.8, and all stable releases through the current version.
344+
285345
## SQL Server
286346

287347
```json

0 commit comments

Comments
 (0)