|
| 1 | +--- |
| 2 | +title: Core concepts |
| 3 | +description: "The model behind every EQL page: domain variants that declare capability, the encrypted payload envelope, the typed-operand rule, and fail-loud blockers." |
| 4 | +type: reference |
| 5 | +components: [eql] |
| 6 | +verifiedAgainst: |
| 7 | + eql: "3.0.0" |
| 8 | +--- |
| 9 | + |
| 10 | +Everything in the EQL reference builds on four ideas: columns are typed as **domain variants** that declare what they can do, every value is a **`jsonb` payload** carrying encrypted index terms, **operands must be typed** for the encrypted operators to resolve, and anything a column can't do **fails loudly** instead of returning wrong rows. This page is the canonical home for all four — the per-type and per-query pages link back here rather than restating them. |
| 11 | + |
| 12 | +## Variants declare capability |
| 13 | + |
| 14 | +EQL ships its searchable-encryption surface as PostgreSQL **domains in the `eql_v3` schema**, all backed by `jsonb`. Each scalar type generates a *family* of domain variants, and the variant you type a column as fixes its query capability. Each domain carries a `CHECK` constraint that validates the encrypted payload on insert, so a malformed or wrong-version value is rejected at write time rather than surfacing at query time. |
| 15 | + |
| 16 | +There is no database-side configuration table. Earlier EQL versions tracked encryption config in the database (`config_add_table`, `config_add_column`, and friends) — those are gone in v3. The searchable surface of a column is fixed by the domain variant you type it as, and which index terms travel in a value's payload is decided by the encryption client (the [Stack SDK](/reference/stack) or [CipherStash Proxy](/reference/proxy)). The domain makes the matching operators resolve; the term in the payload is what makes them answer. |
| 17 | + |
| 18 | +For any scalar type `<T>`, the family looks like this: |
| 19 | + |
| 20 | +| Domain variant | Capability | |
| 21 | +| --- | --- | |
| 22 | +| `eql_v3.<T>` | Storage and decryption only. | |
| 23 | +| `eql_v3.<T>_eq` | Equality: `=`, `<>`, `IN`, `GROUP BY`, `DISTINCT`, equijoins. | |
| 24 | +| `eql_v3.<T>_ord` | Comparisons (`<` … `>=`), `BETWEEN`, `ORDER BY`, `MIN` / `MAX` — plus equality. | |
| 25 | +| `eql_v3.<T>_ord_ore` | As `<T>_ord`, with the ORE mechanism pinned — see [SEM specifiers](#sem-specifiers). | |
| 26 | +| `eql_v3.text_match` (text only) | Free-text token containment: `@>` / `<@`. | |
| 27 | +| `eql_v3.text_search` (text only) | Equality + ordering + token containment. | |
| 28 | + |
| 29 | +Two things worth calling out: |
| 30 | + |
| 31 | +- **The bare variant blocks everything.** `eql_v3.<T>` carries no index term. Querying it with any comparison operator raises an "operator not supported" exception. Use it for columns you only ever store and decrypt — [Booleans](/reference/eql/booleans) covers this pattern in full. |
| 32 | +- **Which index term backs each capability** is an implementation detail of the payload — covered in [Anatomy of an encrypted value](#anatomy-of-an-encrypted-value) below. |
| 33 | + |
| 34 | +### SEM specifiers |
| 35 | + |
| 36 | +A trailing mechanism suffix — the `_ore` in `_ord_ore` — is a **SEM specifier**: it pins *which* searchable-encryption mechanism implements the capability, rather than just declaring the capability itself. |
| 37 | + |
| 38 | +- `eql_v3.<T>_ord` declares *orderable* and leaves the mechanism to EQL's default — currently ORE (order-revealing encryption). |
| 39 | +- `eql_v3.<T>_ord_ore` declares *orderable via ORE, explicitly*. Today the two are byte-identical surfaces backed by the same term. |
| 40 | + |
| 41 | +The distinction earns its keep as mechanisms multiply: the EQL v3 release adds an **OPE** (order-preserving encryption) specifier for every orderable type — including `text` — at which point pinning a specifier documents and freezes a column's mechanism choice, while unspecified variants track the default. Each type page lists its available specifiers under an "SEM specifiers" heading. |
| 42 | + |
| 43 | +Declaring a table is just typing each column as the variant it needs: |
| 44 | + |
| 45 | +```sql |
| 46 | +CREATE TABLE users ( |
| 47 | + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, |
| 48 | + email eql_v3.text_eq, -- equality only |
| 49 | + salary eql_v3.int4_ord, -- equality + range + ORDER BY |
| 50 | + created_at eql_v3.timestamp_ord |
| 51 | +); |
| 52 | +``` |
| 53 | + |
| 54 | +Every scalar type — `int2`, `int4`, `int8`, `numeric`, `float4`, `float8`, `date`, `timestamp`, `text`, and `bool` in EQL 3.0.0 — ships some subset of this family. The per-category pages list exactly which variants each type has and how to choose between them: [Numbers](/reference/eql/numbers), [Dates & times](/reference/eql/dates-and-times), [Text](/reference/eql/text), and [Booleans](/reference/eql/booleans). Encrypted JSON documents use a separate domain, `eql_v3.json`, with its own operator surface — see [JSON](/reference/eql/json). |
| 55 | + |
| 56 | +## Anatomy of an encrypted value |
| 57 | + |
| 58 | +Every EQL encrypted value is a `jsonb` payload with a shared envelope plus the index terms that make it queryable. Earlier CipherStash docs called this format the **CipherCell** — this section is the current definition of the same structure. |
| 59 | + |
| 60 | +Payloads are **produced** by the encryption clients — the [Stack SDK](/reference/stack) and [CipherStash Proxy](/reference/proxy) — and **consumed** by EQL's operators and functions inside Postgres. EQL never sees plaintext: it validates, stores, and compares these payloads; it cannot produce or decrypt them. The division is strict: the clients never rely on the database for key material. |
| 61 | + |
| 62 | +### The envelope |
| 63 | + |
| 64 | +Every payload carries three envelope keys. Each `eql_v3` domain's `CHECK` constraint requires them, so a value missing any of these is rejected at write time: |
| 65 | + |
| 66 | +| Key | Contents | Notes | |
| 67 | +| --- | --- | --- | |
| 68 | +| `v` | The EQL version | `3` — the payload version matches the EQL major version. The domain `CHECK`s assert it and raise on any other value. | |
| 69 | +| `i` | Ident: `{"t": "<table>", "c": "<column>"}` | Binds the ciphertext to the table and column it was encrypted for. Both keys required. | |
| 70 | +| `c` | Ciphertext | The opaque, non-deterministic encrypted blob (mp_base85-encoded). Never used in comparisons. | |
| 71 | + |
| 72 | +<Callout> |
| 73 | +Payloads produced by EQL v2 clients carried `v: 2`; from 3.0.0 the payload version and the EQL version move together. |
| 74 | +</Callout> |
| 75 | + |
| 76 | +A `k` discriminator (`"ct"` for a scalar ciphertext, `"sv"` for a JSON document) also appears on payloads emitted by the clients, distinguishing the two top-level shapes. |
| 77 | + |
| 78 | +### Index-term keys |
| 79 | + |
| 80 | +Alongside the envelope, a payload carries the index terms for its column's capability. Each key is backed by a SEM (searchable encrypted metadata) type in the `eql_v3` schema: |
| 81 | + |
| 82 | +| Key | SEM type | Wire shape | Enables | Reveals | |
| 83 | +| --- | --- | --- | --- | --- | |
| 84 | +| `hm` | `eql_v3.hmac_256` (domain over `text`) | Hex string (HMAC-SHA-256) | `=`, `<>` on `_eq` and `text_search` domains | Whether two values are equal — nothing else | |
| 85 | +| `ob` | `eql_v3.ore_block_256` (composite: array of `bytea` block terms) | Array of hex-encoded ORE blocks (block count varies by scalar width) | `<`, `<=`, `>`, `>=`, `ORDER BY` on `_ord` / `_ord_ore` domains — and `=` / `<>`, since ORE comparison collapses to equality | The relative order of two values | |
| 86 | +| `bf` | `eql_v3.bloom_filter` (domain over `smallint[]`) | Array of set bit positions (**signed** 16-bit — large filters emit negative positions) | `@>` / `<@` token containment on `_match` domains | Probabilistic token overlap between values | |
| 87 | + |
| 88 | +The capability is encoded as **required keys**: the payload for an `eql_v3.text_eq` column must carry `hm`; an `eql_v3.int4_ord` payload must carry `ob` (and only `ob`); a `text_match` payload must carry `bf`; a `text_search` payload carries all three. A payload missing its term key fails the domain `CHECK` — and fails to deserialize in the client bindings. |
| 89 | + |
| 90 | +A scalar payload for an `eql_v3.text_search` column (lookup + ordering + free-text match, so all three terms are required): |
| 91 | + |
| 92 | +```json |
| 93 | +{ |
| 94 | + "v": 3, |
| 95 | + "i": { "t": "users", "c": "email" }, |
| 96 | + "c": "mBbKmsMM%bK#QQOx1yLDBHyD...", |
| 97 | + "hm": "9c8ec1d2f9932b979b1bf3f09f8a4e2f6a41f8de2f0c8b7a52e1f5c3d4b6a790", |
| 98 | + "ob": ["7a1fd0c2...", "d24c9be1...", "03fa66b8..."], |
| 99 | + "bf": [42, 1290, -8113, 30201] |
| 100 | +} |
| 101 | +``` |
| 102 | + |
| 103 | +- `v`, `i`, `c` — the envelope |
| 104 | +- `hm` — equality term: `WHERE email = $1` compares this |
| 105 | +- `ob` — ordering term: `ORDER BY` and range comparisons walk these blocks |
| 106 | +- `bf` — bloom-filter term: `@>` token containment tests these bit positions |
| 107 | + |
| 108 | +Encrypted JSON documents use a different payload shape — an `sv` array with one encrypted entry per path in the document instead of a root ciphertext — defined in [JSON](/reference/eql/json). |
| 109 | + |
| 110 | +### Machine-readable schemas |
| 111 | + |
| 112 | +The [EQL repository](https://github.com/cipherstash/encrypt-query-language) publishes the format as JSON Schema in two places: |
| 113 | + |
| 114 | +- **`crates/eql-bindings/schema/`** — one schema per scalar domain (`$id`s under `https://schemas.cipherstash.com/eql/v3/`), generated from the canonical Rust wire types in the `eql-bindings` crate. TypeScript bindings are generated from the same definitions, so every producer and consumer shares one source of truth. |
| 115 | +- **`docs/reference/schema/`** — full-payload schemas covering both the scalar and `sv` document shapes. These files are still named for the v2.x payload releases (`eql-payload-v2.2.schema.json`, `eql-payload-v2.3.schema.json`); the v2.3 schema describes the document shape, with the payload version field moving to `3` alongside the EQL 3.0.0 release. |
| 116 | + |
| 117 | +## The typed-operand rule |
| 118 | + |
| 119 | +The `eql_v3` domains are backed by `jsonb`. When an operand has no known type — a bare string literal, an untyped parameter — PostgreSQL reduces the domain to its `jsonb` base type and resolves the **native `jsonb` operator** instead of the encrypted one. The query doesn't fail; it silently returns native `jsonb` semantics, which are meaningless for encrypted payloads. |
| 120 | + |
| 121 | +```sql |
| 122 | +-- ❌ Wrong: untyped parameter. PostgreSQL falls back to the native jsonb `=`, |
| 123 | +-- which compares raw payloads — syntactically valid, semantically meaningless. |
| 124 | +SELECT * FROM users WHERE email = $1; |
| 125 | + |
| 126 | +-- ✅ Right: typed operand — the encrypted `=` resolves. |
| 127 | +SELECT * FROM users WHERE email = $1::eql_v3.text_eq; |
| 128 | +``` |
| 129 | + |
| 130 | +Always type the operand: a typed parameter (`$1::eql_v3.text_eq`) or an explicit cast (`'…'::eql_v3.int4_ord`). The [Stack SDK](/reference/stack) and [CipherStash Proxy](/reference/proxy) type bound parameters automatically — raw SQL must do it by hand. |
| 131 | + |
| 132 | +This is the one place where a mistake is *silent*. Everything else fails loudly: |
| 133 | + |
| 134 | +## Unsupported operations fail loudly |
| 135 | + |
| 136 | +Unsupported operators are not silent no-ops. Every operator that a variant doesn't support is still *defined* — it routes to a blocker function that raises an `operator … is not supported` exception. A mis-typed query fails loudly instead of silently returning wrong results: |
| 137 | + |
| 138 | +```sql |
| 139 | +-- salary is eql_v3.int8_eq (equality only) |
| 140 | +SELECT * FROM users WHERE salary > $1::eql_v3.int8_eq; |
| 141 | +-- ERROR: operator > is not supported for eql_v3.int8_eq |
| 142 | +``` |
| 143 | + |
| 144 | +A `NULL` operand still raises — the blockers are deliberately not `STRICT`, so PostgreSQL can't skip the check. (A SQL `NULL` column value is not encrypted, so `IS NULL` / `IS NOT NULL` themselves always work, on every variant.) |
| 145 | + |
| 146 | +`LIKE` and `ILIKE` are blocked on **every** encrypted variant — pattern matching is meaningless on ciphertext. Encrypted text matching is bloom-filter token containment instead; [Text](/reference/eql/text) covers it. |
| 147 | + |
| 148 | +One equality subtlety follows from the term table above: on `_ord` / `_ord_ore` columns, `=` and `<>` compare the **ORE (`ob`) term** — ORE comparison collapses to equality — so `_ord` payloads carry no `hm` term at all. On `_eq` and `text_search` columns, equality compares the HMAC (`hm`) term. |
| 149 | + |
| 150 | +## What the terms reveal |
| 151 | + |
| 152 | +Every index term a value carries is extra material stored in the database, and each term class reveals defined structure to an observer who can read the stored payloads: equality terms reveal *value repetition* (which rows share a value), ORE terms reveal *ordering* (which of two values is larger), and bloom terms reveal *probabilistic token overlap*. None of them reveal the plaintext — but you should only carry the terms you actually query on. The full analysis of what each term does and doesn't leak is in [Searchable encryption](/concepts/searchable-encryption). |
0 commit comments