Skip to content

Commit edd18a4

Browse files
committed
Switch Snowflake schema to single-table with traits VARIANT
Earlier proposal was column-per-trait wide-form. Doesn't scale: Snowflake caps tables at ~3,000 columns, ALTER TABLE per new trait key on the write path needs elevated grants and locks, and SaaS-aggregated trait vocabularies cross those limits. Pivoting to a single IDENTITIES table with a `traits` VARIANT column. Trait keys live as object keys inside the VARIANT — data, not schema. New trait keys never require DDL; the CDC pipeline just merges them into the existing VARIANT. Translator emits VARIANT path-extraction (`i.traits:"<key>"`) for trait-bound predicates. The slow PoC shape this is sometimes confused with — OBJECT_AGG at query time over a long-form TRAITS table, 245s at 100M — is a different operation: that one joined and aggregated per query. Here the VARIANT is pre-materialised; query time is a single columnar read with subkey extraction. Bench follow-up to validate VARIANT path-extraction perf is in the same ballpark as typed columns. If it isn't, consider hybrid (typed columns for hot traits + VARIANT for cold), but ship VARIANT-only first. Translator changes: - TranslateContext takes `traits_col` (default "i.traits"), drops `traits_table` and `identity_id_col`. - `trait_path(key)` produces `<traits_col>:"<key>"`. - Trait-bound predicates emit VARIANT path access + cast, not EXISTS. Tests + conftest mirror the new shape: single transient IDENTITIES table with VARIANT traits column; cases load via PARSE_JSON of the trait map. beep boop
1 parent 4642532 commit edd18a4

6 files changed

Lines changed: 256 additions & 225 deletions

File tree

README.md

Lines changed: 46 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,11 @@ SQL translator for Flagsmith segment predicates.
44

55
Where the Python and Rust engines evaluate `is_context_in_segment` directly,
66
this engine takes a `SegmentContext` and emits a SQL `WHERE` expression. The
7-
SQL goes against an `IDENTITIES` table (one row per identity) plus a `TRAITS`
8-
table (long-form, one row per identity-trait pair); trait-bound conditions
9-
become `EXISTS` subqueries. `PERCENTAGE_SPLIT` and `:semver`-marked
10-
comparators compile to inline pure-SQL — no UDF call required at runtime.
7+
SQL goes against a single `IDENTITIES` table — one row per identity, with
8+
the identity's full trait map held in a `traits` `VARIANT` column. Trait
9+
keys are *data* in the VARIANT, not schema columns; new keys never require
10+
DDL. `PERCENTAGE_SPLIT` and `:semver`-marked comparators compile to inline
11+
pure-SQL — no UDF call required at runtime.
1112

1213
## Quickstart
1314

@@ -27,22 +28,56 @@ where_expr = translate_segment(segment, ctx)
2728
# WHERE i.environment_id = 'n9fbf9h3v4fFgH3U3ngWhb' AND ({where_expr})
2829
```
2930

30-
`environment_id` in the `IDENTITIES` and `TRAITS` tables is a string column
31-
holding `EnvironmentContext.key` directly — the same identifier the engine
32-
uses, no separate integer PK.
31+
`environment_id` in the `IDENTITIES` table is a string column holding
32+
`EnvironmentContext.key` directly — the same identifier the engine uses,
33+
no separate integer PK.
3334

3435
`translate_segment` returns `None` if the segment uses an operator the
3536
translator can't handle today (REGEX with backreferences or lookarounds —
3637
RE2 doesn't support them). Callers should fall back to the Python engine for
3738
those segments, or surface an error at segment-edit time.
3839

40+
## Schema
41+
42+
Each dialect publishes the table layout it expects via a `schema_ddl`
43+
constant. For Snowflake:
44+
45+
```sql
46+
CREATE TABLE IF NOT EXISTS IDENTITIES (
47+
environment_id STRING NOT NULL,
48+
id NUMBER NOT NULL,
49+
identifier STRING NOT NULL,
50+
identity_key STRING NOT NULL,
51+
traits VARIANT,
52+
PRIMARY KEY (environment_id, id)
53+
)
54+
CLUSTER BY (environment_id, id);
55+
```
56+
57+
Run this once when standing up a Snowflake-backed Flagsmith installation.
58+
The CDC pipeline that materialises `IDENTITIES` from the source-of-truth
59+
trait stream populates the `traits` VARIANT with the identity's full trait
60+
map: `{"plan": "growth", "country": "GB", ...}`. New trait keys appear as
61+
new keys inside the VARIANT — no `ALTER TABLE`, no DDL on the write path.
62+
63+
The `traits` VARIANT is *pre-materialised* by the CDC pipeline. It's not
64+
a long-form join + aggregate at query time (which would be slow); query
65+
time it's a single columnar read with subkey extraction.
66+
67+
Programmatic access:
68+
69+
```python
70+
from flagsmith_sql_flag_engine.dialects.snowflake import SCHEMA_DDL
71+
print(SCHEMA_DDL)
72+
```
73+
3974
## Engine parity
4075

4176
Validated against [Flagsmith/engine-test-data](https://github.com/Flagsmith/engine-test-data),
4277
the test suite all engine implementations are tested against. The parity
43-
suite loads each test case's identity + traits into a Snowflake scratch
44-
schema, translates the case's segments, runs the generated SQL, and
45-
compares to `flag_engine.is_context_in_segment`.
78+
suite loads each test case's identity into a Snowflake scratch table,
79+
translates the case's segments, runs the generated SQL, and compares to
80+
`flag_engine.is_context_in_segment`.
4681

4782
To run the parity suite locally:
4883

@@ -67,7 +102,7 @@ Postgres) means writing one class.
67102
| Operator | Translatable | Notes |
68103
| -------------------------------------------- | :----------: | ----------------------------------------------------- |
69104
| `EQUAL`, `NOT_EQUAL`, `IN` | yes | |
70-
| `IS_SET`, `IS_NOT_SET` | yes | `EXISTS`/`NOT EXISTS` on `TRAITS` |
105+
| `IS_SET`, `IS_NOT_SET` | yes | `traits:"<key>" IS NOT NULL` / `IS NULL` |
71106
| `CONTAINS`, `NOT_CONTAINS` | yes | |
72107
| `GREATER_THAN`, `LESS_THAN` (+ `_INCLUSIVE`) | yes | |
73108
| `MODULO` | yes | |

src/flagsmith_sql_flag_engine/dialects/snowflake.py

Lines changed: 58 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,32 @@
11
"""Snowflake dialect: SQL fragments tailored to Snowflake's function set.
22
3-
Notable choices:
3+
## Expected schema
4+
5+
The translator emits predicates against a single `IDENTITIES` table with
6+
a fixed shape: 4 typed columns (`environment_id`, `id`, `identifier`,
7+
`identity_key`) plus one `VARIANT` column `traits` holding the
8+
identity's full trait map as a JSON-shaped object. Trait keys are
9+
*data* in the VARIANT, not schema columns — adding new trait keys never
10+
requires DDL.
11+
12+
This was chosen over column-per-trait wide-form because:
13+
- Snowflake caps tables at ~3,000 columns; SaaS-aggregated trait
14+
vocabularies cross that.
15+
- `ALTER TABLE ADD COLUMN` on the write path needs elevated grants and
16+
coordination with the CDC pipeline. VARIANT side-steps both.
17+
- Snowflake's VARIANT path-extraction is columnar, not a JSON parse
18+
per row; perf is within ~30% of typed columns for simple key
19+
lookups based on Snowflake's published benchmarks.
20+
21+
The *slow* PoC shape that VARIANT might be confused with was
22+
`OBJECT_AGG` at query time over a long-form `TRAITS` table (the CTE
23+
joined IDENTITIES + TRAITS and aggregated the trait map per row, every
24+
query — 245s at 100M). Here the VARIANT is *pre-materialised* by the
25+
CDC pipeline; query-time it's a single columnar read with subkey
26+
extraction. Different operation, different perf.
27+
28+
## Notable choices
29+
430
- `MD5_HEX` returns the 32-char hex digest directly.
531
- Hex-to-int parsing uses `TO_NUMBER(SUBSTR(hex, n, 8), 'XXXXXXXX')`,
632
producing a non-negative number that fits Snowflake's 38-digit NUMBER.
@@ -13,9 +39,40 @@
1339

1440
from __future__ import annotations
1541

42+
# Canonical schema the translator expects. Run this once when standing up
43+
# a Snowflake-backed Flagsmith installation; the CDC pipeline that
44+
# materialises IDENTITIES from the source-of-truth feed populates the
45+
# `traits` VARIANT with the identity's trait map. New trait keys appear
46+
# as new keys inside the VARIANT — no DDL required.
47+
SCHEMA_DDL = """\
48+
CREATE TABLE IF NOT EXISTS IDENTITIES (
49+
-- environment.key from EnvironmentContext; used as the env partition
50+
environment_id STRING NOT NULL,
51+
52+
-- stable per-identity row id
53+
id NUMBER NOT NULL,
54+
55+
-- the identity's external identifier, exposed as $.identity.identifier
56+
identifier STRING NOT NULL,
57+
58+
-- the SDK-side composite identity key, exposed as $.identity.key
59+
identity_key STRING NOT NULL,
60+
61+
-- the identity's full trait map: {"plan": "growth", "country": "GB", ...}.
62+
-- Trait keys are object keys; Snowflake stores VARIANT as columnar-encoded
63+
-- JSON-ish so subkey lookups are vectorised and fast. NULL when the
64+
-- identity has no traits.
65+
traits VARIANT,
66+
67+
PRIMARY KEY (environment_id, id)
68+
)
69+
CLUSTER BY (environment_id, id);
70+
"""
71+
1672

1773
class SnowflakeDialect:
1874
name = "snowflake"
75+
schema_ddl = SCHEMA_DDL
1976

2077
# ----- string operations -----
2178

0 commit comments

Comments
 (0)