Merge pull request #260 from cipherstash/v3-domain-type-text

feat(scalars): add eql_v3.text encrypted-domain family (eq / match / ord)

Author: tobyhede

CHANGELOG.md

@@ -28,6 +28,7 @@ Each entry that ships in a published release links to the PR that introduced it.
 - **`eql_v3.date` encrypted-domain type family.** Four jsonb-backed domains for encrypted `date` columns — `eql_v3.date` (storage-only), `eql_v3.date_eq` (`=` / `<>` via HMAC), and `eql_v3.date_ord` / `eql_v3.date_ord_ore` (also `<` `<=` `>` `>=` via ORE block terms, with `MIN` / `MAX` aggregates) — generated from the `date` row in `eql-scalars::CATALOG` by the same materializer as the `eql_v3.int4` reference. Plaintexts encrypt under the `date` cast and compare via the same ORE block terms as the integer scalars (ORE is plaintext-agnostic — dates order like integers). Index via a functional index on the `eql_v3.eq_term` / `eql_v3.ord_term` extractors, not an operator class on the domain. Why: the first **non-integer ordered** scalar encrypted-domain type — a type-safe, per-capability encrypted `date` column — proving the generator and SQLx test matrix generalize beyond fixed-width integers. ([#256](https://github.com/cipherstash/encrypt-query-language/pull/256))
 - **`eql_v3.timestamptz` encrypted-domain type family (equality-only).** Two jsonb-backed domains for encrypted `timestamptz` columns — `eql_v3.timestamptz` (storage-only) and `eql_v3.timestamptz_eq` (`=` / `<>` via HMAC) — generated from the `timestamptz` row in `eql-scalars::CATALOG` by the same materializer as the `eql_v3.date` family. Values are **UTC-normalized** (cipherstash has no timezone-preserving type): plaintexts encrypt under the `timestamp` cast. Index via a functional index on the `eql_v3.eq_term` extractor, not an operator class on the domain. **Ordering (`<` `<=` `>` `>=`, `MIN` / `MAX`) is deferred:** cipherstash encrypts `Plaintext::Timestamp` at native 12-block ORE width, but EQL's only ORE comparator (`eql_v2.compare_ore_block_u64_8_256_term`) is hardcoded to 8 blocks, so ordered timestamptz domains would silently mis-order. There are no `eql_v3.timestamptz_ord` / `_ord_ore` domains and no timestamptz `MIN` / `MAX` aggregates until a wide-ORE (12-block) term lands — tracked in [#241](https://github.com/cipherstash/encrypt-query-language/issues/241). Why: a type-safe, equality-searchable encrypted UTC-timestamp column, stacking on the `date` temporal-scalar foundation; ordering follows once the comparator supports the native ciphertext width. ([#257](https://github.com/cipherstash/encrypt-query-language/pull/257))
 - **Per-domain `MIN` / `MAX` aggregates for the encrypted-domain family.** `eql_v3.min(eql_v3.<T>_ord)` / `eql_v3.max(eql_v3.<T>_ord)` (and the `_ord_ore` twin) are generated for every ord-capable scalar variant, giving type-safe extrema on domain-typed columns — comparison routes through the variant's `<` / `>` operator (ORE block term, no decryption). The aggregates are declared `PARALLEL = SAFE` with a combine function (the state function itself — min/max are associative), so PostgreSQL can use partial/parallel aggregation on large `GROUP BY` workloads. Why: the new domain types previously had no equivalent of the composite-type aggregates. The existing `eql_v2.min(eql_v2_encrypted)` / `eql_v2.max(eql_v2_encrypted)` aggregates are **retained** and continue to work on `eql_v2_encrypted` columns; the per-domain aggregates are additive and coexist with them. ([#239](https://github.com/cipherstash/encrypt-query-language/pull/239))
+- **`eql_v3.text` encrypted-domain family (`text`, `text_eq`, `text_match`, `text_ord`, `text_ord_ore`).** Adds equality (`=` / `<>` via HMAC), match (`@>` / `<@` via a new self-contained `eql_v3.bloom_filter` SEM index term), and ORE ordering (`<` `<=` `>` `>=`, `min` / `max`) for encrypted text, at parity with EQL v2 text — generated from the `text` row in `eql-scalars::CATALOG` by the same materializer as the `eql_v3.int4` reference. `text` is the first scalar to add a new index `Term` (`Bloom`) and the first non-integer, unbounded ordered kind (lexicographic pivots, hand-written `impl ScalarType`). Index via a functional index on the `eql_v3.eq_term` / `eql_v3.ord_term` / `eql_v3.match_term` extractors, not an operator class on the domain. Why: brings searchable encrypted text to the namespaced, `eql_v2`-free `eql_v3` surface. Match is exposed as bloom-filter containment on the `text_match` domain — deliberately *not* SQL `LIKE` (no wildcard/anchoring; probabilistic ngram containment) — and never backs equality (which always routes through `Hm`). ([#260](https://github.com/cipherstash/encrypt-query-language/pull/260))
 - **Self-contained `eql_v3` schema + standalone `release/cipherstash-encrypt-v3.sql` installer.** The `eql_v3` encrypted-domain surface no longer depends on `eql_v2` at runtime: it now owns its own copies of the searchable-encrypted-metadata (SEM) index-term types — `eql_v3.hmac_256` and `eql_v3.ore_block_u64_8_256` (with its btree operator class) — so the `eql_v3.eq_term` / `eql_v3.ord_term` extractors return `eql_v3` types and no `eql_v2.<symbol>` appears anywhere in the v3 SQL. The whole v3 surface relocated under a single `src/v3/` tree (`src/v3/sem/` for the hand-written SEM types, `src/v3/scalars/` for the generated domain families). A new build variant ships the `eql_v3` schema on its own as `release/cipherstash-encrypt-v3.sql`, installable into a database with no `eql_v2` present; a CI gate greps that artifact and its dependency closure to keep it `eql_v2`-free. Why: a clean foundation for the per-scalar encrypted-domain model to stand alone, ahead of it replacing the `eql_v2_encrypted` composite column type. This is additive — a new schema and a new artifact — and leaves `eql_v2` byte-for-byte unchanged. ([#255](https://github.com/cipherstash/encrypt-query-language/pull/255))
 
 ### Changed

crates/eql-codegen/src/context.rs

@@ -310,19 +310,43 @@ mod tests {
     #[test]
     fn operator_entry_emits_metadata_only_when_supported() {
         use crate::operator_surface::operator;
-        // Supported comparison operator carries its planner metadata.
-        let eq = operator_entry(&operator("="), "eql_v3.int4_eq", "eql_v3.int4_eq", true);
-        assert_eq!(eq.symbol, "=");
-        assert_eq!(eq.function_name, "eq");
-        assert_eq!(
-            eq.metadata.as_deref(),
-            Some("COMMUTATOR = =, NEGATOR = <>, RESTRICT = eqsel, JOIN = eqjoinsel")
-        );
-        // The same operator, unsupported on this domain → no metadata line.
-        let eq_unsupported = operator_entry(&operator("="), "eql_v3.int4", "eql_v3.int4", false);
-        assert_eq!(eq_unsupported.metadata, None);
-        // Supported but metadata-less operator (`@>`) → still no metadata line.
-        let contains = operator_entry(&operator("@>"), "eql_v3.int4_eq", "eql_v3.int4_eq", true);
-        assert_eq!(contains.metadata, None);
+
+        // (symbol, domain, supported) -> expected `CREATE OPERATOR` metadata
+        // clause. Adding a term that carries operator metadata is one new row
+        // here, not another hand-rolled assertion block.
+        let cases: &[(&str, &str, bool, Option<&str>)] = &[
+            // Supported comparison operator carries its planner metadata.
+            (
+                "=",
+                "eql_v3.int4_eq",
+                true,
+                Some("COMMUTATOR = =, NEGATOR = <>, RESTRICT = eqsel, JOIN = eqjoinsel"),
+            ),
+            // The same operator, unsupported on this domain → no metadata line.
+            ("=", "eql_v3.int4", false, None),
+            // Supported but metadata-less operator (`->`) → still no metadata.
+            ("->", "eql_v3.int4_eq", true, None),
+            // `@>` carries containment metadata when supported (the Bloom
+            // `text_match` path).
+            (
+                "@>",
+                "eql_v3.text_match",
+                true,
+                Some("COMMUTATOR = <@, RESTRICT = contsel, JOIN = contjoinsel"),
+            ),
+            // ... but suppressed when `@>` is a blocker (non-Bloom domains),
+            // which is why the int4 golden is unchanged.
+            ("@>", "eql_v3.int4_eq", false, None),
+        ];
+
+        for (symbol, dom, supported, expected) in cases {
+            let entry = operator_entry(&operator(symbol), dom, dom, *supported);
+            assert_eq!(entry.symbol, *symbol);
+            assert_eq!(
+                entry.metadata.as_deref(),
+                *expected,
+                "operator {symbol} on {dom} (supported={supported})",
+            );
+        }
     }
 }

crates/eql-codegen/src/operator_surface.rs

@@ -33,7 +33,7 @@ impl OperatorMetadata {
     }
 
     /// Render the `CREATE OPERATOR` metadata clause, or `None` when no hint is
-    /// present (the `@>`/`<@` symmetric-but-empty case collapses to `None`).
+    /// present (e.g. the path-selector operators, which carry no metadata).
     pub fn render(self) -> Option<String> {
         let mut extras = Vec::new();
         if let Some(c) = self.commutator {
@@ -208,6 +208,18 @@ const fn cmp_metadata(
     }
 }
 
+/// Containment-operator metadata (`@>` / `<@`): commutator is the mirror
+/// operator, no negator (a non-containment is not another listed operator),
+/// containment selectivity estimators.
+const fn containment_metadata(commutator: &'static str) -> OperatorMetadata {
+    OperatorMetadata {
+        restrict: Some("contsel"),
+        join: Some("contjoinsel"),
+        commutator: Some(commutator),
+        negator: None,
+    }
+}
+
 /// The 20-operator catalog. Order is: comparison operators, then path-selector
 /// operators, then the remaining native jsonb operators.
 pub const OPERATORS: &[Operator] = &[
@@ -251,13 +263,13 @@ pub const OPERATORS: &[Operator] = &[
         symbol: "@>",
         function_name: "contains",
         signatures: BOOL_SYMMETRIC_SIGNATURES,
-        metadata: OperatorMetadata::none(),
+        metadata: containment_metadata("<@"),
     },
     Operator {
         symbol: "<@",
         function_name: "contained_by",
         signatures: BOOL_SYMMETRIC_SIGNATURES,
-        metadata: OperatorMetadata::none(),
+        metadata: containment_metadata("@>"),
     },
     Operator {
         symbol: "->",
@@ -519,7 +531,25 @@ mod tests {
             "COMMUTATOR = =, NEGATOR = <>, RESTRICT = eqsel, JOIN = eqjoinsel"
         );
         assert_eq!(operator("->").metadata.render(), None);
-        assert_eq!(operator("@>").metadata.render(), None);
+        // `@>`/`<@` now carry containment metadata (no negator).
+        assert_eq!(
+            operator("@>").metadata.render().unwrap(),
+            "COMMUTATOR = <@, RESTRICT = contsel, JOIN = contjoinsel"
+        );
+    }
+
+    #[test]
+    fn containment_operators_have_containment_metadata() {
+        let c = operator("@>");
+        assert_eq!(c.metadata.commutator, Some("<@"));
+        assert_eq!(c.metadata.restrict, Some("contsel"));
+        assert_eq!(c.metadata.join, Some("contjoinsel"));
+        assert_eq!(c.metadata.negator, None);
+        let cb = operator("<@");
+        assert_eq!(cb.metadata.commutator, Some("@>"));
+        assert_eq!(cb.metadata.restrict, Some("contsel"));
+        assert_eq!(cb.metadata.join, Some("contjoinsel"));
+        assert_eq!(cb.metadata.negator, None);
     }
 
     #[test]

crates/eql-scalars/src/fixture.rs

@@ -25,7 +25,10 @@ impl Fixture {
                 Some(k) => Some(k.max_value()),
                 None => None,
             },
-            Fixture::Zero => Some(0),
+            Fixture::Zero => match kind.as_bounded_int() {
+                Some(_) => Some(0),
+                None => None,
+            },
             Fixture::Int(n) => Some(n),
             Fixture::Numeric(_)
             | Fixture::Text(_)

crates/eql-scalars/src/lib.rs

@@ -83,6 +83,7 @@ pub enum ScalarKind {
 pub enum Term {
     Hm,
     Ore,
+    Bloom,
 }
 
 /// A single fixture plaintext value, value-kind tagged: `Min`/`Max`/`Zero` are
@@ -298,9 +299,61 @@ pub const TIMESTAMPTZ: ScalarSpec = ScalarSpec {
     fixtures: TIMESTAMPTZ_FIXTURES,
 };
 
+/// Domains for `text`: the ordered shape plus a `_match` domain backed by the
+/// `Bloom` term (`@>`/`<@` containment). The ordered subset (`""`, `_eq`,
+/// `_ord_ore`, `_ord`) is identical to `ORDERED_INT_DOMAINS`; `_match` is the
+/// only addition, so text still runs the standard ordered matrix.
+const TEXT_DOMAINS: &[DomainSpec] = &[
+    DomainSpec {
+        suffix: "",
+        terms: &[],
+    },
+    DomainSpec {
+        suffix: "_eq",
+        terms: &[Term::Hm],
+    },
+    DomainSpec {
+        suffix: "_match",
+        terms: &[Term::Bloom],
+    },
+    DomainSpec {
+        suffix: "_ord_ore",
+        terms: &[Term::Ore],
+    },
+    DomainSpec {
+        suffix: "_ord",
+        terms: &[Term::Ore],
+    },
+];
+
+/// `text` fixture plaintexts — curated so eq/ord give a lexicographic spread
+/// and the match suite has a known substring pair (`"aardvark"`/`"aard"`,
+/// sharing 3-grams) and a disjoint value (`"zzzz"`, no shared 3-grams).
+/// `"aard"` is the lexicographic `min_pivot`, `"zzzz"` the `max_pivot`, and
+/// `"frank"` the interior `mid_pivot`; all three must be present verbatim so the
+/// matrix can fetch their ciphertext. All distinct.
+///
+/// The empty string is deliberately **not** a fixture: text is an ordered, not
+/// signed, scalar (no numeric origin), and `""` encrypts to an empty ORE term
+/// whose comparison is undefined (see issue #262). The interior pivot is a real
+/// median value, not `String::default()`.
+const TEXT_FIXTURES: &[Fixture] = fixtures!(text;
+    "aard", "aardvark", "alice", "bob", "carol",
+    "dave", "erin", "frank", "mallory", "trent", "zzzz");
+
+/// `text` — an ordered, non-integer, unbounded scalar. Adds a `_match` domain
+/// (the `Bloom` term) on top of the ordered shape. Public because the SQLx
+/// harness reads `TEXT_VALUES` (materialised below).
+pub const TEXT: ScalarSpec = ScalarSpec {
+    token: "text",
+    kind: ScalarKind::Text,
+    domains: TEXT_DOMAINS,
+    fixtures: TEXT_FIXTURES,
+};
+
 /// The scalar catalog — the single source of truth. Order is significant (it
 /// drives generation order). New types are appended as their SQL surface lands.
-pub const CATALOG: &[ScalarSpec] = &[INT4, INT2, INT8, DATE, TIMESTAMPTZ];
+pub const CATALOG: &[ScalarSpec] = &[INT4, INT2, INT8, DATE, TIMESTAMPTZ, TEXT];
 
 /// Materialise an integer scalar's fixtures into a typed `&'static` slice at
 /// compile time. This is the **single-sourced** plaintext list the SQLx test
@@ -353,5 +406,37 @@ int_values!(INT4_VALUES, i32, INT4);
 int_values!(INT2_VALUES, i16, INT2);
 int_values!(INT8_VALUES, i64, INT8);
 
+/// Materialise a `text` scalar's fixtures into a `&'static [&'static str]` at
+/// compile time — the single-sourced plaintext list the SQLx matrix reads via
+/// `ScalarType::fixture_values()` and the fixture generator encrypts. Unlike
+/// `date` (chrono is not `const`-friendly), a `Fixture::Text(&'static str)` is
+/// already const, so text materialises a typed slice like the integer kinds.
+/// A non-text fixture is a const-eval panic (compile-time guard).
+macro_rules! text_values {
+    ($name:ident, $spec:expr) => {
+        #[doc = concat!("Distinct plaintext fixture values for `", stringify!($spec), "`, ")]
+        #[doc = "materialised from its `CATALOG` row (see `text_values!`)."]
+        pub const $name: &[&'static str] = {
+            const SPEC: ScalarSpec = $spec;
+            const N: usize = SPEC.fixtures.len();
+            const ARR: [&'static str; N] = {
+                let mut out = [""; N];
+                let mut i = 0;
+                while i < N {
+                    out[i] = match SPEC.fixtures[i] {
+                        Fixture::Text(s) => s,
+                        _ => panic!("text scalar fixture must be Fixture::Text"),
+                    };
+                    i += 1;
+                }
+                out
+            };
+            &ARR
+        };
+    };
+}
+
+text_values!(TEXT_VALUES, TEXT);
+
 #[cfg(test)]
 mod tests;

crates/eql-scalars/src/term.rs

@@ -11,6 +11,7 @@ impl Term {
         match self {
             Term::Hm => "hm",
             Term::Ore => "ob",
+            Term::Bloom => "bf",
         }
     }
 
@@ -19,6 +20,7 @@ impl Term {
         match self {
             Term::Hm => "eq_term",
             Term::Ore => "ord_term",
+            Term::Bloom => "match_term",
         }
     }
 
@@ -27,6 +29,7 @@ impl Term {
         match self {
             Term::Hm => "hmac_256",
             Term::Ore => "ore_block_u64_8_256",
+            Term::Bloom => "bloom_filter",
         }
     }
 
@@ -35,6 +38,7 @@ impl Term {
         match self {
             Term::Hm => "eq",
             Term::Ore => "ord",
+            Term::Bloom => "match",
         }
     }
 
@@ -43,6 +47,7 @@ impl Term {
         match self {
             Term::Hm => &["=", "<>"],
             Term::Ore => &["=", "<>", "<", "<=", ">", ">="],
+            Term::Bloom => &["@>", "<@"],
         }
     }
 
@@ -54,6 +59,7 @@ impl Term {
                 "src/v3/sem/ore_block_u64_8_256/functions.sql",
                 "src/v3/sem/ore_block_u64_8_256/operators.sql",
             ],
+            Term::Bloom => &["src/v3/sem/bloom_filter/functions.sql"],
         }
     }
 }