Skip to content

Commit ba16490

Browse files
committed
feat(ClickHouse): Add ClickHouse dialect
Implements the `Dialect` protocol against ClickHouse — `MergeTree` schema with `Nullable(String)` traits read via `JSONExtract*`, RE2 anchored regex via `match`, hex-chunk parse via `reinterpretAsUInt32(reverse(unhex(...)))`, type dispatch via `JSONType`. Adds a `ClickHouseHarness` for the engine-parity suite (clickhouse- connect HTTP client, per-run Memory table, batched UNION ALL of `count() > 0`). All 599 engine-test-data segments pass; the three xfails match Snowflake's set (translator-level divergences shared by every dialect). beep boop
1 parent 6adc186 commit ba16490

8 files changed

Lines changed: 762 additions & 14 deletions

File tree

README.md

Lines changed: 73 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -67,16 +67,31 @@ CREATE TABLE IF NOT EXISTS IDENTITIES (
6767
CLUSTER BY (environment_id, id);
6868
```
6969

70-
Run this DDL to create the table the translator emits against. Trait
71-
keys are *data* inside the VARIANT, so new keys appear without schema
72-
changes; at query time, trait lookups are a single columnar read with
73-
subkey extraction.
70+
For ClickHouse:
71+
72+
```sql
73+
CREATE TABLE IF NOT EXISTS IDENTITIES (
74+
environment_id String,
75+
id UInt64,
76+
identifier String,
77+
identity_key String,
78+
traits Nullable(String)
79+
)
80+
ENGINE = MergeTree()
81+
ORDER BY (environment_id, id);
82+
```
83+
84+
The Snowflake table stores traits in a single `VARIANT` column; the
85+
ClickHouse table stores the JSON-encoded trait map in a `Nullable(String)`
86+
column read via `JSONExtract*`. Either way, trait keys are *data* — new
87+
keys appear without schema changes — and the translator only sees the
88+
abstract path extraction.
7489

7590
Programmatic access:
7691

7792
```python
78-
from flagsmith_sql_flag_engine.dialects.snowflake import SCHEMA_DDL
79-
print(SCHEMA_DDL)
93+
from flagsmith_sql_flag_engine.dialects.snowflake import SCHEMA_DDL as SNOWFLAKE_DDL
94+
from flagsmith_sql_flag_engine.dialects.clickhouse import SCHEMA_DDL as CLICKHOUSE_DDL
8095
```
8196

8297
## Engine parity
@@ -91,12 +106,27 @@ To run the engine-parity suite locally:
91106

92107
```bash
93108
git submodule update --init # pull engine-test-data
109+
110+
# Snowflake
94111
export SNOWFLAKE_ACCOUNT=...
95112
export SNOWFLAKE_USER=...
96113
export SNOWFLAKE_PRIVATE_KEY_PATH=...
97-
uv run pytest -m engine_parity
114+
115+
# ClickHouse (bring up a local container first)
116+
docker run -d --rm --name clickhouse-parity \
117+
-p 18123:8123 \
118+
-e CLICKHOUSE_SKIP_USER_SETUP=1 \
119+
clickhouse/clickhouse-server:latest
120+
export CLICKHOUSE_HOST=localhost
121+
export CLICKHOUSE_PORT=18123
122+
123+
uv run pytest tests/test_engine.py
98124
```
99125

126+
Each harness's environment variables are only read at session-create
127+
time; to run a single dialect's parity, pass e.g. `-k snowflake` or
128+
`-k clickhouse` and only export that dialect's credentials.
129+
100130
Adding a new dialect's parity coverage is one harness module — see
101131
`tests/harnesses/` for the shape.
102132

@@ -105,9 +135,42 @@ Adding a new dialect's parity coverage is one harness module — see
105135
The translator is dialect-aware: a `Dialect` protocol abstracts the
106136
SQL fragments that differ across SQL engines — MD5 hex, hex-to-int
107137
parsing, prefix-anchored regex, padded-version comparison, type-aware
108-
trait predicates, regex flavour. Today only `SnowflakeDialect` is
109-
implemented; adding another engine such as DuckDB or Postgres means
110-
writing one class.
138+
trait predicates, regex flavour. Today `SnowflakeDialect` and
139+
`ClickHouseDialect` are implemented; adding another engine such as
140+
DuckDB or Postgres means writing one class.
141+
142+
### Snowflake vs ClickHouse
143+
144+
Both dialects pass the engine-parity suite with the same two xfails
145+
(prerelease semver sort and `$.`-prefixed trait names — translator-level
146+
divergences shared by every dialect). Operator coverage is identical.
147+
148+
The shape of the two implementations differs because the engines do:
149+
150+
| Concern | Snowflake | ClickHouse |
151+
| -------------------- | ---------------------------------------- | ----------------------------------------------------------- |
152+
| Trait storage | `VARIANT` (columnar JSON) | `Nullable(String)` JSON read via `JSONExtract*` |
153+
| Trait path | `i.traits:"key"` returns VARIANT | `if(JSONType=Null, NULL, ...)` returning canonical string |
154+
| Type discrimination | `TYPEOF`, `IS_BOOLEAN`, `IS_DECIMAL` | `JSONType` (Enum8) — exact type names |
155+
| Hex chunk parse | `TO_NUMBER(SUBSTR(...), 'XXXXXXXX')` | `reinterpretAsUInt32(reverse(unhex(substring(...))))` |
156+
| Anchored regex | `REGEXP_INSTR(value, pat) = 1` | `match(value, '^(pat)')``match` is unanchored |
157+
| Numeric coalesce | Lossless string fast path via VARIANT::STRING | Per-type dispatch — no analogous canonical stringify |
158+
| Nullable propagation | `(VARIANT NULL)::STRING → NULL` | `Nullable(String)` is explicit; regex funcs reject it |
159+
160+
Practical implications for callers:
161+
162+
- ClickHouse's `match`, `extractAll` reject `Nullable(String)` input
163+
because the implied result types are unrepresentable. The dialect
164+
wraps regex value expressions in `ifNull(..., '')`; trait paths are
165+
always guarded by `IS NOT NULL` upstream, so the default is
166+
unreachable at runtime.
167+
- Snowflake's VARIANT path collapses both missing keys and JSON null to
168+
SQL NULL "by accident" (cast propagation). The ClickHouse dialect
169+
collapses both explicitly via `JSONType = 'Null'`. Caller-visible
170+
behaviour is the same.
171+
- ClickHouse's batched `EXISTS`-equivalent uses windowed `count() > 0`
172+
inside a `UNION ALL``EXISTS (SELECT 1 FROM ...)` isn't a top-level
173+
expression in ClickHouse the way it is in Snowflake.
111174

112175
## Operator coverage
113176

pyproject.toml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ dev = [
2323
"mypy>=1.10",
2424
"prek>=0.3",
2525
"snowflake-snowpark-python>=1.20",
26+
"clickhouse-connect>=0.7",
2627
"json5>=0.14.0",
2728
"pytest-cov>=7.1.0",
2829
]
@@ -69,3 +70,7 @@ select = ["E", "F", "I", "B", "UP"]
6970
strict = true
7071
python_version = "3.10"
7172
files = ["src/flagsmith_sql_flag_engine", "tests"]
73+
74+
[[tool.mypy.overrides]]
75+
module = "clickhouse_connect.*"
76+
ignore_missing_imports = true
Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
1-
"""Dialect implementations. Today only Snowflake is supported."""
1+
"""Dialect implementations."""
22

3+
from flagsmith_sql_flag_engine.dialects.clickhouse import ClickHouseDialect
34
from flagsmith_sql_flag_engine.dialects.snowflake import SnowflakeDialect
45

5-
__all__ = ["SnowflakeDialect"]
6+
__all__ = ["ClickHouseDialect", "SnowflakeDialect"]

0 commit comments

Comments
 (0)