Merge pull request #267 from cipherstash/eql-v3-jsonb-domain

feat(v3): self-contained encrypted-JSONB (SteVec) document type `eql_v3.json`

Author: tobyhede

.github/workflows/test-eql.yml

@@ -267,7 +267,7 @@ jobs:
 
       - name: Clean-DB v3 install smoke (Postgres ${{ matrix.postgres-version }})
         run: |
-          mise run build
+          mise run clean && mise run build
           mise run test:clean_install_v3
 
   schema:
@@ -376,7 +376,7 @@ jobs:
           shared-key: sqlx-tests
           save-if: false
       - name: Build EQL
-        run: mise run --force build
+        run: mise run clean && mise run --force build
       - name: Assert eql_v3 is self-contained
         run: mise run test:self_contained_v3
 
@@ -405,9 +405,11 @@ jobs:
       - name: Verify the matrix test-name inventory
         run: |
           mise run test:matrix:inventory
+          mise run test:matrix:inventory:jsonb_entry
+          mise run test:v3-jsonb:inventory
           git add -N tests/sqlx/snapshots
           git diff --exit-code -- tests/sqlx/snapshots \
-            || { echo "Coverage inventory stale — run 'mise run test:matrix:inventory' and commit."; exit 1; }
+            || { echo "Coverage inventory stale — run the relevant inventory task and commit."; exit 1; }
 
   splinter:
     name: "Supabase splinter"
@@ -433,7 +435,7 @@ jobs:
           mise run postgres:up postgres-${POSTGRES_VERSION} --extra-args "--detach --wait"
       - name: Build and install EQL
         run: |
-          mise run --output prefix --force build
+          mise run clean && mise run --output prefix --force build
           cat release/cipherstash-encrypt.sql \
             | docker exec -i postgres-${POSTGRES_VERSION} \
                 psql -v ON_ERROR_STOP=1 \

.gitignore

@@ -225,6 +225,8 @@ tests/sqlx/migrations/001_install_eql.sql
 # Generated SQLx fixtures (regenerated via `mise run fixture:generate`,
 # never commit — stale fixtures hide bugs)
 tests/sqlx/fixtures/eql_v2*
+tests/sqlx/fixtures/v3_ste_vec.sql
+tests/sqlx/fixtures/v3_doc_int4.sql
 
 # Generated encrypted-domain SQL — regenerated by `tasks/build.sh` from the
 # eql-scalars::CATALOG via `cargo run -p eql-codegen` on every build. The
@@ -256,3 +258,7 @@ src/deps-ordered-protect.txt
 # Prebuilt nextest archive (transient build-once output for the sharded sqlx
 # suite; uploaded as a CI artifact, never committed — see tasks/test/sqlx-archive.sh)
 nextest.tar.zst
+
+# Working notes retained on disk but intentionally not committed
+docs/decisions/2026-06-10-eql-v3-json-type-kind.md
+docs/handoff/2026-06-10-v3-jsonb-fixture-alignment.md

CHANGELOG.md

@@ -22,6 +22,7 @@ Each entry that ships in a published release links to the PR that introduced it.
 
 ### Added
 
+- **`eql_v3` encrypted-JSONB (SteVec) document type.** A self-contained encrypted-JSONB surface in the `eql_v3` schema: the storage domain `eql_v3.json` plus `eql_v3.ste_vec_entry` (a single sv element) and `eql_v3.ste_vec_query` (a containment needle). Encrypted JSON documents are searchable without decryption via document containment (`@>`, `<@`), field/array access (`->`, `->>`, `jsonb_path_query` / `_exists` / `_query_first`, `jsonb_array_length` / `_elements` / `_elements_text`), entry equality (`=`, `<>`) on extracted leaves, and entry-level ordered range (`<`, `<=`, `>`, `>=`) on ordered leaves via CLLW ORE (`eql_v3.ore_cllw`). Comparisons are leaf-level only: root-document `=`, `<>`, `<`, `<=`, `>`, `>=` are blocked (they raise rather than falling through to native whole-`jsonb` comparison). Index via a functional index on the `eql_v3.eq_term` / `eql_v3.ore_cllw` extractors (entry equality / ordering) and `eql_v3.to_ste_vec_query(col)::jsonb jsonb_path_ops` GIN (containment); ordered leaves get a default btree opclass on `eql_v3.ore_cllw`. Every other native `jsonb` operator reachable through domain fallback (`?`, `?|`, `?&`, `@?`, `@@`, `#>`, `#>>`, `-`, `#-`, `||`, and the root comparisons above) is blocked — it raises rather than silently routing to plaintext-jsonb semantics. Note: because `eql_v3.json` is a `jsonb` domain, selector/operand literals must be typed (`col -> 'sel'::text` — the CipherStash Proxy interface passes typed parameters); a *bare untyped* literal (`col -> 'sel'`) resolves to the native `jsonb` operator instead of the encrypted one (PostgreSQL reduces the domain to its base type for unknown-typed literals), and the same caveat applies to the native-jsonb blockers — see `docs/decisions/2026-06-10-eql-v3-json-type-kind.md`. The whole surface owns its SEM index-term types (`eql_v3.hmac_256`, `eql_v3.ore_cllw`) and has no `eql_v2` dependency (CI-gated by `test:self_contained_v3` and the standalone `release/cipherstash-encrypt-v3.sql` installer). Why: a type-safe, searchable encrypted JSONB document column namespaced under `eql_v3`, complementing the scalar encrypted-domain families. The existing `eql_v2` SteVec surface on `eql_v2_encrypted` is unchanged. ([#267](https://github.com/cipherstash/encrypt-query-language/pull/267))
 - **`eql_v3` encrypted-domain schema, with the `int4` family as its first member.** Encrypted-domain type families now live in a new, additional `eql_v3` schema (the existing `eql_v2` schema is unchanged — it keeps the core types/operators and stays the documented public API). Four jsonb-backed domains for encrypted `int4` columns: `eql_v3.int4` (storage-only), `eql_v3.int4_eq` (`=` / `<>` via HMAC), and `eql_v3.int4_ord` / `eql_v3.int4_ord_ore` (also `<` `<=` `>` `>=` via ORE block terms). Supported comparisons resolve to inlinable wrappers; the native `jsonb` operator surface reachable through domain fallback is blocked (raises rather than silently mis-resolving). Each domain's `CHECK` requires the EQL envelope (`v`, `i`), the ciphertext (`c`), and the variant's index term(s), and pins the payload version (`VALUE->>'v' = '2'`, matching `eql_v2._encrypted_check_v`) — so a missing key or wrong-version payload is rejected on insert or cast rather than surfacing later at query time. Index via a functional index on the `eql_v3.eq_term` / `eql_v3.ord_term` extractors, not an operator class on the domain. The extractors return the searchable-encrypted-metadata index-term types `eql_v3.hmac_256` / `eql_v3.ore_block_u64_8_256`, which `eql_v3` owns directly (see the self-contained `eql_v3` schema entry below). Why: a type-safe, per-capability encrypted integer column instead of the untyped `eql_v2_encrypted`, namespaced under its own schema. This is the reference scalar implementation for the generated domain family. ([#239](https://github.com/cipherstash/encrypt-query-language/pull/239), supersedes [#225](https://github.com/cipherstash/encrypt-query-language/pull/225))
 - **`eql_v3.int2` encrypted-domain type family.** Four jsonb-backed domains for encrypted `int2` columns — `eql_v3.int2` (storage-only), `eql_v3.int2_eq` (`=` / `<>` via HMAC), and `eql_v3.int2_ord` / `eql_v3.int2_ord_ore` (also `<` `<=` `>` `>=` via ORE block terms, with `MIN` / `MAX` aggregates) — generated from the `int2` row in `eql-scalars::CATALOG` by the same materializer as the `eql_v3.int4` reference. Index via a functional index on the `eql_v3.eq_term` / `eql_v3.ord_term` extractors, not an operator class on the domain. Why: a type-safe, per-capability encrypted `smallint` column, proving the scalar generator generalizes beyond the `int4` reference. ([#243](https://github.com/cipherstash/encrypt-query-language/pull/243))
 - **`eql_v3.int8` encrypted-domain type family.** Four jsonb-backed domains for encrypted `int8` columns — `eql_v3.int8` (storage-only), `eql_v3.int8_eq` (`=` / `<>` via HMAC), and `eql_v3.int8_ord` / `eql_v3.int8_ord_ore` (also `<` `<=` `>` `>=` via ORE block terms, with `MIN` / `MAX` aggregates) — generated from the `int8` row in `eql-scalars::CATALOG` by the same materializer as the `eql_v3.int4` reference. Index via a functional index on the `eql_v3.eq_term` / `eql_v3.ord_term` extractors, not an operator class on the domain. Why: a type-safe, per-capability encrypted `bigint` column, extending the scalar generator across the full 64-bit integer width. ([#253](https://github.com/cipherstash/encrypt-query-language/pull/253))
@@ -30,6 +31,7 @@ Each entry that ships in a published release links to the PR that introduced it.
 - **Per-domain `MIN` / `MAX` aggregates for the encrypted-domain family.** `eql_v3.min(eql_v3.<T>_ord)` / `eql_v3.max(eql_v3.<T>_ord)` (and the `_ord_ore` twin) are generated for every ord-capable scalar variant, giving type-safe extrema on domain-typed columns — comparison routes through the variant's `<` / `>` operator (ORE block term, no decryption). The aggregates are declared `PARALLEL = SAFE` with a combine function (the state function itself — min/max are associative), so PostgreSQL can use partial/parallel aggregation on large `GROUP BY` workloads. Why: the new domain types previously had no equivalent of the composite-type aggregates. The existing `eql_v2.min(eql_v2_encrypted)` / `eql_v2.max(eql_v2_encrypted)` aggregates are **retained** and continue to work on `eql_v2_encrypted` columns; the per-domain aggregates are additive and coexist with them. ([#239](https://github.com/cipherstash/encrypt-query-language/pull/239))
 - **`eql_v3.text` encrypted-domain family (`text`, `text_eq`, `text_match`, `text_ord`, `text_ord_ore`).** Adds equality (`=` / `<>` via HMAC), match (`@>` / `<@` via a new self-contained `eql_v3.bloom_filter` SEM index term), and ORE ordering (`<` `<=` `>` `>=`, `min` / `max`) for encrypted text, at parity with EQL v2 text — generated from the `text` row in `eql-scalars::CATALOG` by the same materializer as the `eql_v3.int4` reference. `text` is the first scalar to add a new index `Term` (`Bloom`) and the first non-integer, unbounded ordered kind (lexicographic pivots, hand-written `impl ScalarType`). Index via a functional index on the `eql_v3.eq_term` / `eql_v3.ord_term` / `eql_v3.match_term` extractors, not an operator class on the domain. Why: brings searchable encrypted text to the namespaced, `eql_v2`-free `eql_v3` surface. Match is exposed as bloom-filter containment on the `text_match` domain — deliberately *not* SQL `LIKE` (no wildcard/anchoring; probabilistic ngram containment) — and never backs equality (which always routes through `Hm`). ([#260](https://github.com/cipherstash/encrypt-query-language/pull/260))
 - **Self-contained `eql_v3` schema + standalone `release/cipherstash-encrypt-v3.sql` installer.** The `eql_v3` encrypted-domain surface no longer depends on `eql_v2` at runtime: it now owns its own copies of the searchable-encrypted-metadata (SEM) index-term types — `eql_v3.hmac_256` and `eql_v3.ore_block_u64_8_256` (with its btree operator class) — so the `eql_v3.eq_term` / `eql_v3.ord_term` extractors return `eql_v3` types and no `eql_v2.<symbol>` appears anywhere in the v3 SQL. The whole v3 surface relocated under a single `src/v3/` tree (`src/v3/sem/` for the hand-written SEM types, `src/v3/scalars/` for the generated domain families). A new build variant ships the `eql_v3` schema on its own as `release/cipherstash-encrypt-v3.sql`, installable into a database with no `eql_v2` present; a CI gate greps that artifact and its dependency closure to keep it `eql_v2`-free. Why: a clean foundation for the per-scalar encrypted-domain model to stand alone, ahead of it replacing the `eql_v2_encrypted` composite column type. This is additive — a new schema and a new artifact — and leaves `eql_v2` byte-for-byte unchanged. ([#255](https://github.com/cipherstash/encrypt-query-language/pull/255))
+- **`eql_v3.min` / `eql_v3.max` aggregates over `eql_v3.ste_vec_entry`.** SteVec document entries extracted at a selector (`doc -> 'sel'`) can now be aggregated like ordered scalars: `eql_v3.min(doc -> 'sel')` / `eql_v3.max(...)` return the entry with the smallest / largest ordered leaf. Ordering routes through the entry's `oc` (CLLW ORE) term via `eql_v3.ore_cllw` — the same comparator the entry `<` / `<=` / `>` / `>=` operators use, not the scalar Block-ORE `ord_term`. Only `oc`-carrying entries are orderable: an entry without an `oc` term (`eql_v3.ore_cllw` returns NULL) is non-orderable and is ignored by the aggregate — the same way the `eql_v3.ore_cllw` btree NULL-filters such rows — so a mix of `oc`-carrying and `oc`-less entries yields the extremum of the orderable subset rather than a corrupted result. Declared `PARALLEL = SAFE` with a combine function (the state function itself), so partial / parallel aggregation is available on large `GROUP BY` workloads. Why: brings encrypted-JSONB entry ordering to parity with the scalar encrypted-domain families' `MIN` / `MAX`, and lets the shared scalar behaviour matrix cover entry aggregation. Additive — the document and entry comparison surface is otherwise unchanged. ([#267](https://github.com/cipherstash/encrypt-query-language/pull/267))
 
 ### Changed
 

CLAUDE.md

@@ -103,6 +103,22 @@ Footguns the spec exists to prevent:
 - SQL test fixtures and helpers in `tests/test_helpers.sql`
 - Database connection: `localhost:7432` (cipherstash/password)
 
+#### Tests run against real encrypted data (hard requirement)
+
+EQL is searchable encryption; tests MUST use real ciphertexts/index terms from the actual crypto,
+never hand-curated or synthetic blobs. Fixtures are **generated** by encrypting plaintext through
+cipherstash-client: `mise run test:sqlx:prep` runs `fixture:generate:all` (the
+`generate_all_fixtures` test, `--features fixture-gen`, over `eql-scalars::CATALOG`) → gitignored
+`tests/sqlx/fixtures/eql_v2_*.sql`.
+
+- The SQLx suite **requires** CipherStash creds — ZeroKMS auth (`CS_CLIENT_ACCESS_KEY` +
+  `CS_WORKSPACE_CRN`) AND a client key (`CS_CLIENT_ID` + `CS_CLIENT_KEY`); see the
+  `test:sqlx:prep` comment in `mise.toml`. CI has them. This is expected, not a reason to avoid
+  generated fixtures.
+- Do NOT add static/committed fixtures to dodge the creds dependency. The one committed
+  exception, `tests/sqlx/fixtures/v3_ste_vec.sql`, is a gap pending a SteVec-document generator
+  (`docs/handoff/2026-06-10-v3-jsonb-fixture-alignment.md`), not a pattern to copy.
+
 ## Project Learning & Retrospectives
 
 Valuable lessons and insights from completed work: