fix(stack): count codepoints in the free-text needle floor

Review of #607 found the needle guard it added was itself incomplete, and two
claims it made were false. Three agents verified each finding against live
Postgres before anything changed here.

The guard counted UTF-16 code units; the ngram tokenizer counts CODEPOINTS. A
needle of 1-2 astral-plane characters cleared the floor, produced no trigram,
and so built an EMPTY bloom filter -- `stored_bf @> '{}'` matches every row.
Measured live against three seeded rows:

    "👍👍"    utf16=4 cp=2 -> 3/3 rows   <- passed the guard
    "👍👍👍"  utf16=6 cp=3 -> 0/3 rows
    "joh"     utf16=3 cp=3 -> 1/3 rows

Codepoints, not graphemes: NFD "e+combining-acute" twice is 4 codepoints but 2
grapheme clusters, and does build a non-empty filter (0/3 rows live). A grapheme
floor would wrongly reject it; a byte floor would wrongly accept the emoji. The
three measurements are consistent only with codepoint counting.

Also rejects the empty needle under ANY tokenizer. `matchNeedleMinLength`
returns undefined for the `standard` tokenizer, so the guard short-circuited and
`contains(col, '')` tokenized to nothing and matched every row. Unreachable via
the v3 `types` namespace today, which hardcodes ngram, but the schema permits it.

Tests, written first and watched fail:

- `__tests__/match-needle-guard.test.ts` pins the unit. The astral and
  empty-needle cases failed before the fix; the NFD case passed before and
  after, proving the fix does not over-correct into rejecting usable needles.
  Unicode is built from explicit escapes -- the precomposed NFC form is a
  different, 2-codepoint string, so a bare literal would silently test the
  opposite case depending on the file's on-disk normalisation.
- The live `contains` rejection now asserts the guard's own message, not just
  `EncryptionOperatorError`. `operandFailure` throws the same class, and that is
  not hypothetical: `text_search` carries an `ore` index and cannot encrypt any
  non-ASCII operand ("Can only order strings that are pure ASCII"), so the
  astral case was passing there for the wrong reason. That constraint now has
  its own test, and the codepoint complement runs only on match-only columns.

Two false claims corrected rather than papered over:

- The changeset and the doc comment both said the floor was "shared with the v2
  adapter". It is not -- `matchNeedleError` has exactly one caller, in v3. v2's
  `like`/`ilike` reduces to `eql_v2.bloom_filter(a) @> eql_v2.bloom_filter(b)`
  (cipherstash-encrypt.sql), native array containment, so v2 has the same
  fail-open and is NOT fixed here. Wiring it in needs its own measurement first:
  v2 needles carry SQL wildcards, so the floor may belong on the unwrapped term.
- `decryptDenied` returned a bare boolean and treated any throw as denial, so a
  ZeroKMS outage read as "the identity boundary held". It now returns the
  message and the call sites assert /^Failed to retrieve key/, matching the
  assertion this PR already made in `lock-context.test.ts`.

The `as SQL` cast in `inArrayOp` was reviewed and kept: drizzle returns
undefined only for an empty filtered list, and the empty-list guard throws
first. Restoring the old `?? sql`false`` fallback would be worse -- it would
mask a `notInArray` that silently matches everything.

Author: tobyhede

.changeset/eql-v3-drizzle-fail-open-guards.md

@@ -4,14 +4,20 @@
 
 Close two fail-open paths in the EQL v3 Drizzle adapter.
 
-`ops.contains()` now throws `EncryptionOperatorError` when the search term is
-shorter than the match index tokenizer's `token_length` (3 by default). Such a
-term produces no ngrams, so its bloom filter is empty — and `stored_bf @> '{}'`
-is true for every row. Previously a user searching `"ad"` silently received the
-entire table; measured live, the needles `"ad"`, `"a"` and `"x"` each returned
-every seeded row, including one in which `"x"` did not appear. The length floor
-is shared with the v2 adapter via `matchNeedleError` in `schema/match-defaults`,
-since both build byte-identical bloom filters.
+`ops.contains()` now throws `EncryptionOperatorError` for a search term that
+tokenizes to nothing: the empty string, or a term shorter than the match index
+tokenizer's `token_length` (3 by default). Such a term produces an empty bloom
+filter, and `stored_bf @> '{}'` is true for every row — so a user searching
+`"ad"` silently received the entire table. Measured live, the terms `"ad"`,
+`"a"` and `"x"` each returned every seeded row, including one in which `"x"`
+did not appear.
+
+The floor counts Unicode codepoints, matching the tokenizer. A UTF-16 length
+check would wave through an astral-plane term — `"👍👍"` is 4 code units but
+only 2 codepoints, yields no trigram, and matched every row.
+
+**Breaking for callers passing short terms:** `contains()` calls that previously
+returned every row now throw. Terms of 3+ codepoints are unaffected.
 
 `v3FromDriver()` now throws the new `EqlV3CodecError` on a payload that is not
 an EQL envelope, instead of surfacing a raw `SyntaxError` for malformed JSON and
@@ -23,3 +29,8 @@ accepts both scalar envelopes (ciphertext at `c`) and SteVec documents
 
 Also removes an unreachable branch in `inArray`/`notInArray`, whose empty-list
 guard already throws before it.
+
+Note: the v2 Drizzle adapter's `like`/`ilike` path builds the same bloom filters
+and has the same short-term fail-open. It is **not** fixed here — v2 terms carry
+SQL wildcards, so the floor must be measured against what its tokenizer actually
+receives before the shared guard can be reused. Tracked separately.

packages/stack/__tests__/drizzle-v3/operators-live-pg.test.ts

@@ -549,12 +549,74 @@ describeLivePg('v3 drizzle operators (live pg matrix)', () => {
   // A needle below the tokenizer's `token_length` builds an EMPTY bloom filter,
   // and `stored_bf @> '{}'` holds for every row — so before the SDK guard this
   // silently returned the whole table.
+  //
+  // `'👍👍'` is the regression case: 4 UTF-16 code units but only 2 codepoints,
+  // so a `needle.length` floor waved it through and it matched every row. The
+  // tokenizer counts codepoints. `''` tokenizes to nothing under any tokenizer.
   it.each(
-    matchDomains,
-  )('%s contains rejects a needle shorter than token_length', async (eqlType) => {
-    await expect(
-      ops.contains(matrixColumn(eqlType), 'ad'),
-    ).rejects.toBeInstanceOf(EncryptionOperatorError)
+    matchDomains.flatMap(([eqlType]) =>
+      ['ad', '👍👍', ''].map((needle) => [eqlType, needle] as const),
+    ),
+  )(
+    '%s contains rejects the unanswerable needle %j before encrypting',
+    async (eqlType, needle) => {
+      const attempt = ops.contains(matrixColumn(eqlType), needle)
+      await expect(attempt).rejects.toBeInstanceOf(EncryptionOperatorError)
+      // Assert the GUARD rejected it, not the encryption layer.
+      // `operandFailure` also throws `EncryptionOperatorError`, so matching only
+      // the class would let an encryption error stand in for the guard and the
+      // test would pass for the wrong reason.
+      await expect(attempt).rejects.toThrow(/free-text search needs/)
+    },
+    30000,
+  )
+
+  // The complement: a needle that DOES reach the floor in codepoints must be
+  // ANSWERED, not over-rejected — otherwise the fix for the astral fail-open
+  // would just over-correct into rejecting usable needles.
+  //
+  // Restricted to match-ONLY columns: a column that also carries an `ore` index
+  // (`text_search`) cannot encrypt a non-ASCII operand at all — the ORE term
+  // raises "Can only order strings that are pure ASCII" — so a non-ASCII needle
+  // there fails inside encryption, long after the guard has passed it. That
+  // constraint is pinned by its own test below rather than silently conflated
+  // with the codepoint floor.
+  //
+  // Escapes, not literals: the precomposed NFC form of `ee-with-accents` is only
+  // 2 codepoints and IS rejected, so a bare literal would test the opposite case
+  // depending on the file's on-disk normalisation.
+  const NFD_EE = 'e\u0301e\u0301' // 4 codepoints, 2 grapheme clusters
+  const matchOnlyDomains = matchDomains.filter(([, spec]) => !spec.indexes.ore)
+
+  it.each(
+    matchOnlyDomains.flatMap(([eqlType]) =>
+      ['\u{1F44D}\u{1F44D}\u{1F44D}', NFD_EE].map(
+        (needle) => [eqlType, needle] as const,
+      ),
+    ),
+  )(
+    '%s contains answers the codepoint-sufficient needle %j with no match',
+    async (eqlType, needle) => {
+      const rows = await selectRowKeys(
+        await ops.contains(matrixColumn(eqlType), needle),
+      )
+      expect(rows).toEqual([])
+    },
+    30000,
+  )
+
+  // An `ore`-bearing match column rejects a non-ASCII needle inside ENCRYPTION,
+  // not in the needle guard. Pinned so the distinction stays visible: the guard
+  // is about tokenization, this is about the ORE term's ASCII-only ordering.
+  it.each(
+    matchDomains.filter(([, spec]) => spec.indexes.ore),
+  )('%s contains rejects a non-ASCII needle in the ORE term, not the guard', async (eqlType) => {
+    const attempt = ops.contains(
+      matrixColumn(eqlType),
+      '\u{1F44D}\u{1F44D}\u{1F44D}',
+    )
+    await expect(attempt).rejects.toThrow(/pure ASCII/)
+    await expect(attempt).rejects.not.toThrow(/free-text search needs/)
   }, 30000)
 
   it.each(

packages/stack/__tests__/drizzle-v3/operators-lock-context-live-pg.test.ts

@@ -84,22 +84,38 @@ function unwrap<T>(result: { data?: T; failure?: { message: string } }): T {
 }
 
 /**
- * True when a decrypt attempt was DENIED. `decrypt` reports denial as a
- * `Result.failure` rather than throwing, so a bare `await` would resolve and a
- * naive `expect(...).rejects` would never fire — denial has to be read off the
- * result. A throw also counts as denial.
+ * The outcome of a decrypt attempt that is EXPECTED to be denied. `decrypt`
+ * reports denial as a `Result.failure` rather than throwing, so a bare `await`
+ * would resolve and a naive `expect(...).rejects` would never fire — denial has
+ * to be read off the result. A throw also counts as denial.
+ *
+ * Returns the message, not just a boolean, so callers can assert WHY it was
+ * denied. A bare boolean would let any infrastructure fault — a DNS failure, an
+ * expired CTS token, a ZeroKMS outage ("SendRequest: Failed to send request") —
+ * read as "the identity boundary held", and the security test would pass for
+ * the wrong reason.
  */
-async function decryptDenied(
+async function decryptOutcome(
   attempt: () => PromiseLike<{ failure?: { message: string } }>,
-): Promise<boolean> {
+): Promise<{ denied: boolean; message: string }> {
   try {
     const result = await attempt()
-    return Boolean(result.failure)
-  } catch {
-    return true
+    return {
+      denied: Boolean(result.failure),
+      message: result.failure?.message ?? '',
+    }
+  } catch (error) {
+    return { denied: true, message: (error as Error).message }
   }
 }
 
+/**
+ * A genuine key-derivation denial. ZeroKMS cannot reproduce the key tag without
+ * the identity claim the row was sealed under, and says so. Asserted verbatim
+ * elsewhere in the suite (`lock-context.test.ts`, `protect-ops.test.ts`).
+ */
+const KEY_DENIAL = /^Failed to retrieve key/
+
 /** Run-scoped SELECT of row keys under an already-encrypted SQL condition. */
 async function selectRowKeys(condition: SQL): Promise<string[]> {
   const rows = (await db
@@ -230,10 +246,14 @@ describeLivePg('v3 drizzle operators with lock context (live pg)', () => {
     )
 
     // `decrypt` reports denial as a `Result.failure` and does not throw, so a
-    // bare `await` here would silently pass. Require denial via either channel.
-    expect(await decryptDenied(() => client.decrypt(row.value as never))).toBe(
-      true,
+    // bare `await` here would silently pass. Require denial via either channel,
+    // AND require it be a key-derivation denial rather than an outage.
+    const { denied, message } = await decryptOutcome(() =>
+      client.decrypt(row.value as never),
     )
+
+    expect(denied).toBe(true)
+    expect(message).toMatch(KEY_DENIAL)
   }, 30000)
 
   it('an identity-bound row does not decrypt under a different identity claim', async () => {
@@ -244,12 +264,13 @@ describeLivePg('v3 drizzle operators with lock context (live pg)', () => {
       [RUN, ROW_A],
     )
 
-    expect(
-      await decryptDenied(() =>
-        client
-          .decrypt(row.value as never)
-          .withLockContext({ identityClaim: ['email'] } as never),
-      ),
-    ).toBe(true)
+    const { denied, message } = await decryptOutcome(() =>
+      client
+        .decrypt(row.value as never)
+        .withLockContext({ identityClaim: ['email'] } as never),
+    )
+
+    expect(denied).toBe(true)
+    expect(message).toMatch(KEY_DENIAL)
   }, 30000)
 })

packages/stack/__tests__/match-needle-guard.test.ts

@@ -0,0 +1,93 @@
+import { describe, expect, it } from 'vitest'
+import type { MatchIndexOpts } from '@/schema'
+import { matchNeedleError, matchNeedleMinLength } from '@/schema/match-defaults'
+
+// The floor a free-text needle must clear before it yields any ngram. A needle
+// below it tokenizes to nothing, so its bloom filter is EMPTY — and
+// `stored_bf @> '{}'` is true for every row. Such a query is unanswerable, not
+// merely unmatched, so the guard must reject it rather than silently return the
+// whole table.
+const ngram: MatchIndexOpts = { tokenizer: { kind: 'ngram', token_length: 3 } }
+
+describe('matchNeedleMinLength', () => {
+  it('is the ngram tokenizer token_length', () => {
+    expect(matchNeedleMinLength(ngram)).toBe(3)
+    expect(
+      matchNeedleMinLength({ tokenizer: { kind: 'ngram', token_length: 4 } }),
+    ).toBe(4)
+  })
+
+  it('defaults an absent tokenizer to the schema default rather than skipping the floor', () => {
+    expect(matchNeedleMinLength({})).toBe(3)
+  })
+})
+
+describe('matchNeedleError', () => {
+  it('accepts a needle at or above the floor', () => {
+    expect(matchNeedleError('joh', ngram)).toBeUndefined()
+    expect(matchNeedleError('lovelace', ngram)).toBeUndefined()
+  })
+
+  it('rejects a needle below the floor, naming the floor and the term', () => {
+    expect(matchNeedleError('ad', ngram)).toMatch(/at least 3 characters/)
+    expect(matchNeedleError('ad', ngram)).toMatch(/"ad"/)
+  })
+
+  // The tokenizer counts Unicode CODEPOINTS. `'👍👍'` is 2 codepoints but 4
+  // UTF-16 code units, so a `needle.length` check waves it through — and it
+  // then matches every row. Measured live: 3/3 rows returned.
+  it('rejects an astral-plane needle below the floor in CODEPOINTS, not UTF-16 units', () => {
+    expect('👍👍'.length).toBe(4) // pins why a naive length check passes it
+    expect([...'👍👍'].length).toBe(2)
+
+    expect(matchNeedleError('👍👍', ngram)).toMatch(/at least 3 characters/)
+  })
+
+  it('reports the codepoint count, not the UTF-16 length, in the message', () => {
+    expect(matchNeedleError('👍👍', ngram)).toMatch(/has 2\b/)
+  })
+
+  it('accepts an astral-plane needle that reaches the floor in codepoints', () => {
+    expect(matchNeedleError('👍👍👍', ngram)).toBeUndefined()
+  })
+
+  // Combining acute accents. NFD 'e\u0301e\u0301' is 4 codepoints but only 2
+  // grapheme clusters, and (measured live) builds a NON-empty filter — so the
+  // unit is codepoints, not graphemes. A grapheme floor would wrongly reject it.
+  //
+  // Built from explicit escapes: the PRECOMPOSED NFC form is a different string
+  // of only 2 codepoints, which the guard correctly rejects. A bare literal here
+  // would test whichever form the file happened to be normalised to on disk.
+  const NFD_EE = 'e\u0301e\u0301'
+  const NFC_EE = '\u00e9\u00e9'
+
+  it('accepts a combining-accent needle that is 4 codepoints but only 2 graphemes', () => {
+    expect([...NFD_EE].length).toBe(4)
+    expect(matchNeedleError(NFD_EE, ngram)).toBeUndefined()
+  })
+
+  it('rejects the precomposed NFC form, which really is 2 codepoints', () => {
+    expect([...NFC_EE].length).toBe(2)
+    expect(matchNeedleError(NFC_EE, ngram)).toMatch(/at least 3 characters/)
+  })
+
+  it('rejects the empty needle whatever the tokenizer', () => {
+    expect(matchNeedleError('', ngram)).toBeTypeOf('string')
+    // A `standard` tokenizer imposes no ngram floor, but the empty string still
+    // yields zero tokens and so still matches every row.
+    expect(
+      matchNeedleError('', { tokenizer: { kind: 'standard' } }),
+    ).toBeTypeOf('string')
+  })
+
+  it('imposes no floor on a non-empty needle under a standard tokenizer', () => {
+    expect(
+      matchNeedleError('a', { tokenizer: { kind: 'standard' } }),
+    ).toBeUndefined()
+  })
+
+  it('ignores non-string operands, leaving them to the encryption layer', () => {
+    expect(matchNeedleError(42, ngram)).toBeUndefined()
+    expect(matchNeedleError(null, ngram)).toBeUndefined()
+  })
+})

packages/stack/src/schema/match-defaults.ts

@@ -95,21 +95,43 @@ export function matchNeedleMinLength(opts: MatchIndexOpts): number | undefined {
  * Why a needle cannot be answered by this match index, or `undefined` when it
  * can. Callers throw their own error type with this as the reason.
  *
- * A needle shorter than the ngram tokenizer's `token_length` produces NO
- * ngrams, so its bloom filter is empty — and `stored_bf @> '{}'` is true for
- * every row ("contains nothing, contained by everything"). Such a query is
- * unanswerable rather than merely unmatched, so it must fail loudly instead of
- * silently returning the whole table.
+ * A needle that tokenizes to NOTHING has an empty bloom filter, and
+ * `stored_bf @> '{}'` is true for every row ("contains nothing, contained by
+ * everything"). Such a query is unanswerable rather than merely unmatched, so
+ * it must fail loudly instead of silently returning the whole table. Two ways
+ * to tokenize to nothing:
  *
- * Shared by the v2 and v3 adapters: both build byte-identical bloom filters, so
- * the floor is a property of the index config, not of the adapter version.
+ *  - the empty needle, under ANY tokenizer;
+ *  - a needle shorter than the ngram tokenizer's `token_length`.
+ *
+ * The ngram floor counts Unicode CODEPOINTS, because that is what the tokenizer
+ * counts. `needle.length` would count UTF-16 code units and wave through an
+ * astral-plane needle: `'👍👍'` is 4 code units but only 2 codepoints, yields no
+ * trigram, and (measured live) matched every row. Graphemes are the wrong unit
+ * in the other direction — NFD `'éé'` is 4 codepoints but 2 grapheme clusters,
+ * and does yield trigrams.
+ *
+ * Currently wired ONLY into the v3 Drizzle adapter (`eql/v3/drizzle/operators`).
+ * It lives here, beside the shared match defaults, because v2 builds the same
+ * bloom filters and needs the same floor — but v2's `like`/`ilike` path remains
+ * unguarded and still fails open. Do not reuse this for v2 without first
+ * measuring what its tokenizer actually receives: v2 needles carry SQL wildcards
+ * (`'%ada%'`), so the floor may have to apply to the unwrapped term rather than
+ * to the string the caller passed.
  */
 export function matchNeedleError(
   needle: unknown,
   opts: MatchIndexOpts,
 ): string | undefined {
   if (typeof needle !== 'string') return undefined
+
+  // Codepoints, not UTF-16 code units — see above.
+  const length = [...needle].length
   const min = matchNeedleMinLength(opts)
-  if (min === undefined || needle.length >= min) return undefined
-  return `free-text search needs at least ${min} characters (the index tokenizer's token_length), but the search term ${JSON.stringify(needle)} has ${needle.length}. A shorter term produces no tokens and would match every row.`
+
+  if (length === 0) {
+    return `free-text search needs a non-empty search term; an empty term produces no tokens and would match every row.`
+  }
+  if (min === undefined || length >= min) return undefined
+  return `free-text search needs at least ${min} characters (the index tokenizer's token_length), but the search term ${JSON.stringify(needle)} has ${length}. A shorter term produces no tokens and would match every row.`
 }