Merge pull request #565 from cipherstash/eql-v3-drizzle-concrete-types

Add EQL v3 Drizzle support

Author: tobyhede

.changeset/eql-v3-drizzle.md

@@ -0,0 +1,17 @@
+---
+"@cipherstash/stack": minor
+---
+
+Add EQL v3 Drizzle support at `@cipherstash/stack/eql/v3/drizzle`. A Drizzle-native
+`types` namespace (same PascalCase names as `@cipherstash/stack/eql/v3`) declares
+encrypted columns whose Postgres type is the semantic `public.<domain>`; the concrete
+type drives the legal query operators. `createEncryptionOperatorsV3` provides
+capability-checked `eq`/`ne`/`gt`/`gte`/`lt`/`lte`/`between`/`contains`/`inArray`/
+`asc`/`desc`/`and`/`or` that emit the latest two-argument `eql_v3` SQL functions with
+full-envelope operands, and
+`extractEncryptionSchemaV3` rebuilds the schema for `EncryptionV3`. The existing v2
+`@cipherstash/stack/drizzle` integration is unchanged.
+
+The v3 text-search helper is `contains`; obsolete `like`/`ilike` helpers are not
+exposed because v3 free-text search is token containment rather than SQL wildcard
+matching.

packages/stack/README.md

@@ -366,6 +366,87 @@ For columns with `searchableJson: true`, three JSONB operators are available:
 
 These operators encrypt the JSON path selector using the `steVecSelector` query type and cast it to `eql_v2_encrypted` for use with the EQL PostgreSQL functions.
 
+### EQL v3 Concrete-Type Columns (Drizzle)
+
+The `@cipherstash/stack/eql/v3/drizzle` module targets EQL v3, where each encrypted column is a **concrete Postgres domain type** (`eql_v3.text_eq`, `eql_v3.integer_ord`, …). Instead of toggling capability flags, you pick a concrete type from the `types` namespace and its query capabilities are fixed by that choice. The v2 `@cipherstash/stack/drizzle` module above is unchanged — this is an additive export.
+
+Declare a Drizzle table using the `types` factories. The suffix encodes the capability: `*Eq` (equality), `*Ord` (order + range, which also covers equality), `*Match` / `TextSearch` (free-text search), and the bare name (e.g. `Text`, `Bigint`) is storage-only.
+
+```ts
+import { pgTable, integer } from "drizzle-orm/pg-core"
+import { drizzle } from "drizzle-orm/postgres-js"
+import {
+  types,
+  createEncryptionOperatorsV3,
+  extractEncryptionSchemaV3,
+} from "@cipherstash/stack/eql/v3/drizzle"
+import { EncryptionV3 } from "@cipherstash/stack/v3"
+
+// Capabilities come from the concrete type — no flags to configure.
+const users = pgTable("users", {
+  id: integer("id").primaryKey().generatedAlwaysAsIdentity(),
+  email: types.TextEq("email"),      // equality: eq / ne / inArray
+  age: types.IntegerOrd("age"),      // order + range: gt/gte/lt/lte, between, asc/desc
+  bio: types.TextMatch("bio"),       // free-text search: contains
+  balance: types.Bigint("balance"),  // storage only (no query capability)
+})
+```
+
+Derive the v3 schema from the table, build the typed client, and create the capability-checked operators:
+
+```ts
+const usersSchema = extractEncryptionSchemaV3(users)
+const client = await EncryptionV3({ schemas: [usersSchema] })
+const ops = createEncryptionOperatorsV3(client)
+
+const db = drizzle({ client: sqlClient })
+```
+
+The operators auto-encrypt their operands and validate them against the column's concrete type. Applying an operator the type doesn't support throws `EncryptionOperatorError`:
+
+```ts
+// Equality — email is TextEq
+const exact = await db.select().from(users)
+  .where(await ops.eq(users.email, "alice@example.com"))
+
+// Range + ordering — age is IntegerOrd
+const adults = await db.select().from(users)
+  .where(await ops.gte(users.age, 18))
+  .orderBy(ops.asc(users.age))
+
+const midBand = await db.select().from(users)
+  .where(await ops.between(users.age, 25, 40))
+
+// Set membership — built on equality
+const listed = await db.select().from(users)
+  .where(await ops.inArray(users.email, ["alice@example.com", "bob@example.com"]))
+
+// Free-text token containment — bio is TextMatch
+const coffee = await db.select().from(users)
+  .where(await ops.contains(users.bio, "coffee"))
+```
+
+Rows are **pre-encrypted** with `client.bulkEncryptModels(...)` before they reach `db.insert(...).values(...)` — Drizzle never sees plaintext. `Bigint` columns take a native JS `bigint`:
+
+```ts
+const rows = await client.bulkEncryptModels(
+  [
+    { email: "alice@example.com", age: 30, bio: "climbing and coffee", balance: 100_000n },
+    { email: "bob@example.com", age: 41, bio: "cycling and coffee", balance: 250_000n },
+  ],
+  usersSchema,
+)
+if (rows.failure) throw new Error(rows.failure.message)
+
+await db.insert(users).values(rows.data)
+```
+
+Notes:
+
+- **Free-text search uses `ops.contains`** (token containment on a `match` index), not SQL `like` / `ilike`. It matches whole indexed tokens, so `contains(users.bio, "coffee")` finds rows whose `bio` contains the token `coffee`.
+- **The concrete type defines the legal operators.** `TextEq` supports `eq` / `ne` / `inArray` / `notInArray`; `*Ord` types add `gt` / `gte` / `lt` / `lte` / `between` / `notBetween` and `asc` / `desc`; `*Match` and `TextSearch` add `contains`; a bare `Text` / `Integer` / `Bigint` column is storage-only. Using an unsupported operator throws `EncryptionOperatorError`.
+- Combine conditions with `ops.and` / `ops.or`, and do NULL checks with `ops.isNull` / `ops.isNotNull` (the where-clause operators are `async` and must be `await`ed; `ops.asc` / `ops.desc` are synchronous).
+
 ## Identity-Aware Encryption
 
 Lock encryption to a specific user by requiring a valid JWT for decryption.
@@ -664,6 +745,8 @@ csValue(valueName)                 // returns ProtectValue (for nested values)
 | `@cipherstash/stack/secrets` | `Secrets` class and secrets types |
 | `@cipherstash/stack/client` | Client-safe exports (schema builders and types only - no native FFI) |
 | `@cipherstash/stack/types` | All TypeScript types (`Encrypted`, `Decrypted`, `ClientConfig`, `EncryptionClientConfig`, query types, etc.) |
+| `@cipherstash/stack/v3` | `EncryptionV3` typed client plus the EQL v3 authoring DSL (`encryptedTable`, `types`, v3 type helpers) |
+| `@cipherstash/stack/eql/v3/drizzle` | EQL v3 Drizzle integration: `types` column factories, `createEncryptionOperatorsV3`, `extractEncryptionSchemaV3`, `makeEqlV3Column`, `EncryptionOperatorError` |
 
 ## Migration from @cipherstash/protect
 

packages/stack/__tests__/bulk-encrypt-validation.test.ts

@@ -0,0 +1,107 @@
+import { describe, expect, it } from 'vitest'
+import {
+  BulkEncryptOperation,
+  BulkEncryptOperationWithLockContext,
+} from '@/encryption/operations/bulk-encrypt'
+import type {
+  BuildableColumn,
+  BuildableTable,
+  BulkEncryptPayload,
+  Client,
+} from '@/types'
+
+/**
+ * `EncryptOperation` rejects NaN / ±Infinity / out-of-int64 `bigint` values
+ * client-side, before they reach protect-ffi (whose behaviour on such a value
+ * is unobservable — see `helpers/validation.ts`). `BulkEncryptOperation` must
+ * enforce the same contract: any caller that batches instead of looping — the
+ * v3 Drizzle `inArray`, for one — otherwise silently loses the guard.
+ *
+ * The stub client is never reached: validation must throw first. If a case
+ * here ever calls into protect-ffi, the fake client surfaces it as a different
+ * error than the one asserted.
+ */
+const client = {} as Client
+
+const column: BuildableColumn = {
+  getName: () => 'age',
+  build: () => ({}) as never,
+}
+
+const table: BuildableTable = {
+  tableName: 'users',
+  build: () => ({ tableName: 'users', columns: {} }),
+}
+
+const payload = (...values: unknown[]): BulkEncryptPayload =>
+  values.map((plaintext) => ({ plaintext })) as BulkEncryptPayload
+
+const lockContext = {
+  ctsToken: { accessToken: 'token' },
+} as never
+
+describe('BulkEncryptOperation numeric validation', () => {
+  it.each([
+    [Number.NaN, 'Cannot encrypt NaN value'],
+    [Number.POSITIVE_INFINITY, 'Cannot encrypt Infinity value'],
+    [Number.NEGATIVE_INFINITY, 'Cannot encrypt Infinity value'],
+    [2n ** 70n, 'Cannot encrypt bigint value out of int64 range'],
+    [-(2n ** 70n), 'Cannot encrypt bigint value out of int64 range'],
+  ])('rejects %s before reaching the FFI', async (value, message) => {
+    const op = new BulkEncryptOperation(client, payload(value), {
+      column,
+      table,
+    })
+
+    const result = await op.execute()
+
+    expect(result.failure?.message).toContain(message)
+  })
+
+  it('rejects an invalid value anywhere in the list, not just the first', async () => {
+    const op = new BulkEncryptOperation(client, payload(30, 42, Number.NaN), {
+      column,
+      table,
+    })
+
+    const result = await op.execute()
+
+    expect(result.failure?.message).toContain('Cannot encrypt NaN value')
+  })
+
+  it('still passes null entries through without tripping validation', async () => {
+    const op = new BulkEncryptOperation(client, [{ plaintext: null }], {
+      column,
+      table,
+    })
+
+    const result = await op.execute()
+
+    expect(result.failure).toBeUndefined()
+    expect(result.data).toEqual([{ id: undefined, data: null }])
+  })
+
+  it('accepts in-range bigints at the int64 boundary', async () => {
+    const op = new BulkEncryptOperation(client, payload(9223372036854775807n), {
+      column,
+      table,
+    })
+
+    const result = await op.execute()
+
+    // Validation passes, so the stub client is reached — proving the boundary
+    // value was NOT rejected by the numeric guard.
+    expect(result.failure?.message).not.toContain('out of int64 range')
+  })
+
+  it('enforces the same guard on the lock-context variant', async () => {
+    const op = new BulkEncryptOperationWithLockContext(
+      new BulkEncryptOperation(client, payload(Number.NaN), { column, table }),
+      lockContext,
+    )
+
+    const result = await op.execute()
+
+    expect(result.failure?.message).toContain('Cannot encrypt NaN value')
+  })
+})

packages/stack/__tests__/drizzle-v3/bigint.test.ts

@@ -0,0 +1,99 @@
+import { type SQL, sql } from 'drizzle-orm'
+import { integer, PgDialect, pgTable } from 'drizzle-orm/pg-core'
+import { describe, expect, it, vi } from 'vitest'
+import { getEqlV3Column, isEqlV3Column } from '@/eql/v3/drizzle/column'
+import {
+  createEncryptionOperatorsV3,
+  EncryptionOperatorError,
+} from '@/eql/v3/drizzle/operators'
+import { types } from '@/eql/v3/drizzle/types'
+
+// A representative encrypted envelope — what `client.encrypt` actually returns
+// and what a bigint column stores. Deliberately NOT a plaintext bigint.
+const TERM = { c: 'ct', v: 1 }
+const TERM_JSON = JSON.stringify(TERM)
+
+function chainable(result: unknown) {
+  const op = result as {
+    withLockContext: ReturnType<typeof vi.fn>
+    audit: ReturnType<typeof vi.fn>
+  }
+  op.withLockContext = vi.fn(() => op)
+  op.audit = vi.fn(() => op)
+  return op
+}
+
+function setup() {
+  const encrypt = vi.fn(() => chainable(Promise.resolve({ data: TERM })))
+  const ops = createEncryptionOperatorsV3({ encrypt })
+  const dialect = new PgDialect()
+  const render = (s: SQL) => dialect.sqlToQuery(s)
+  return { ops, encrypt, render }
+}
+
+// A statically-typed table via the drizzle `types` namespace (no dynamic
+// matrix, no casts) — the column set is known at compile time.
+const accounts = pgTable('accounts', {
+  id: integer().primaryKey(),
+  balance: types.BigintOrd('balance'),
+  ledgerId: types.BigintEq('ledger_id'),
+  archived: types.Bigint('archived'),
+})
+
+describe('v3 drizzle bigint columns', () => {
+  it('detects a bigint column and reports its concrete public.bigint domain', () => {
+    const storage = types.Bigint('n')
+    const ord = types.BigintOrd('n_ord')
+
+    expect(isEqlV3Column(storage)).toBe(true)
+    expect(getEqlV3Column('n', storage)?.getEqlType()).toBe('public.bigint')
+    expect(getEqlV3Column('n_ord', ord)?.getEqlType()).toBe('public.bigint_ord')
+  })
+
+  it('emits the concrete public.bigint* SQL type through pgTable', () => {
+    expect(accounts.balance.getSQLType()).toBe('public.bigint_ord')
+    expect(accounts.ledgerId.getSQLType()).toBe('public.bigint_eq')
+    expect(accounts.archived.getSQLType()).toBe('public.bigint')
+  })
+
+  it('encrypts a native bigint operand for eq without JSON-stringifying it', async () => {
+    const { ops, encrypt, render } = setup()
+    // A value beyond Number.MAX_SAFE_INTEGER to prove the bigint is passed
+    // through untouched (a JSON.stringify of a bigint would have thrown).
+    const value = 9223372036854775807n
+    const q = render(await ops.eq(accounts.ledgerId, value))
+
+    expect(q.sql).toContain('eql_v3.eq("accounts"."ledger_id", $1::jsonb)')
+    expect(q.params).toEqual([TERM_JSON])
+    expect(encrypt.mock.calls[0]?.[0]).toBe(value)
+  })
+
+  it('emits ordering and range operators for a bigint_ord column', async () => {
+    const { ops, render } = setup()
+
+    const gt = render(await ops.gt(accounts.balance, 10n))
+    expect(gt.sql).toContain('eql_v3.gt("accounts"."balance", $1::jsonb)')
+
+    const between = render(await ops.between(accounts.balance, -5n, 5n))
+    expect(between.sql).toContain('eql_v3.gte("accounts"."balance", $1::jsonb)')
+    expect(between.sql).toContain('eql_v3.lte("accounts"."balance", $2::jsonb)')
+
+    const asc = render(ops.asc(accounts.balance))
+    expect(asc.sql).toContain('eql_v3.ord_term("accounts"."balance")')
+  })
+
+  it('rejects equality/ordering on a storage-only bigint column', async () => {
+    const { ops } = setup()
+    await expect(ops.eq(accounts.archived, 1n)).rejects.toBeInstanceOf(
+      EncryptionOperatorError,
+    )
+    expect(() => ops.asc(accounts.archived)).toThrow(EncryptionOperatorError)
+  })
+
+  it('rejects a bare SQL expression that is not an encrypted column', async () => {
+    const { ops } = setup()
+    await expect(ops.eq(sql`1`, 1n)).rejects.toBeInstanceOf(
+      EncryptionOperatorError,
+    )
+  })
+})

packages/stack/__tests__/drizzle-v3/codec.test.ts

@@ -0,0 +1,31 @@
+import { describe, expect, it } from 'vitest'
+import { v3FromDriver, v3ToDriver } from '@/eql/v3/drizzle/codec'
+
+describe('v3 codec', () => {
+  it('serialises an object to a jsonb string', () => {
+    expect(v3ToDriver({ v: 1, c: 'ct' })).toBe('{"v":1,"c":"ct"}')
+  })
+
+  it('maps null/undefined to SQL NULL (JS null), never the JSON null literal', () => {
+    expect(v3ToDriver(null)).toBeNull()
+    expect(v3ToDriver(undefined)).toBeNull()
+  })
+
+  it('parses a jsonb string back to an object', () => {
+    expect(v3FromDriver('{"v":1,"c":"ct"}')).toEqual({ v: 1, c: 'ct' })
+  })
+
+  it('passes an already-parsed object through unchanged', () => {
+    const obj = { v: 1 }
+    expect(v3FromDriver(obj)).toBe(obj)
+  })
+
+  it('normalises null/undefined to SQL NULL (JS null) on read', () => {
+    expect(v3FromDriver(null)).toBeNull()
+    expect(v3FromDriver(undefined)).toBeNull()
+  })
+
+  it('does not throw on a stray bigint in the envelope (defensive)', () => {
+    expect(v3ToDriver({ v: 1n } as never)).toBe('{"v":"1"}')
+  })
+})