test(integration): no skipped tests, and fail the run if any appear

A skipped test reads exactly like a passing one. Every silent hole this suite
has found took that shape, and the skips it was itself carrying hid one more.

**The bug the skips were hiding.** `dbVariant()` inferred the database from the
presence of `PGRST_URL`. The new Drizzle `db=supabase` cell needs no PostgREST
and left it unset, so the variant reported `postgres`: EQL installed WITHOUT
`--supabase`, the role grants were never applied, and the grants test quietly
did not run. The suite passed because Drizzle connects as `postgres`, never as
`anon`. The variant is now explicit (`CS_IT_DB_VARIANT`) in both workflows, and
the inference survives only as a documented fallback for a hand-run suite.

**The skips are gone**, all ten:

- The eight per-family "defers <domain>" notices were `it.skip`. They asserted
  nothing. `domainsForFamily` already excludes deferred domains, and
  `test-kit-families.test.ts` asserts — as a PASSING test — that the excluded
  set is exactly the nine block-ORE domains, each with a reason.

- The two harness gates were `it.runIf`. They are now single tests that assert
  both branches: on Supabase the three roles exist and hold the `eql_v3` and
  `eql_v3_internal` grants; on plain Postgres those roles do not exist at all,
  which is a fact worth stating and makes the variant claim falsifiable.
  PostgREST is asserted on its own axis (`PGRST_URL` set) rather than on the
  variant — conflating "is this Supabase" with "is PostgREST up" is what made
  `dbVariant()` lie.

**And a guard so they cannot come back.** `no-skips-reporter.ts` fails the
integration run if any test is skipped, naming each one. Verified: adding a
single `it.skip` exits 1.

Scoped to the integration config. The unit suites still carry the `LIVE_*` gates
and their 16 skips; working those out is a separate change.

Verified on all three CI cells, each 0 skipped, 0 failed:
  drizzle / postgres   619 passed
  drizzle / supabase   619 passed
  supabase / supabase  665 passed

Author: coderdan

.github/workflows/integration-drizzle.yml

@@ -59,8 +59,13 @@ jobs:
         include:
           - db: postgres
             database-url: postgres://cipherstash:password@localhost:55432/cipherstash
+            pgrest-url: ''
           - db: supabase
             database-url: postgres://postgres:password@localhost:55433/postgres
+            # The supabase compose file starts PostgREST anyway. Drizzle does not
+            # use it, but supplying the URL lets the harness assert the `anon`
+            # path on the database it is actually running against.
+            pgrest-url: http://localhost:55430
 
     env:
       CS_WORKSPACE_CRN: ${{ secrets.CS_WORKSPACE_CRN }}
@@ -70,8 +75,14 @@ jobs:
       # Job-level env, not a `.env` file: `dotenv/config` does not override an
       # already-set `process.env`, so these win and no secret is written to disk.
       DATABASE_URL: ${{ matrix.database-url }}
-      # Scoped to the Drizzle suites plus the adapter-agnostic ones. `PGRST_URL`
-      # is deliberately unset: a Supabase suite scoped here would throw, not skip.
+      PGRST_URL: ${{ matrix.pgrest-url }}
+      # EXPLICIT, never inferred. The variant decides whether EQL is installed
+      # with `--supabase` (and therefore whether the role grants are applied).
+      # Inferring it from `PGRST_URL` reported `postgres` for this job's Supabase
+      # cell and silently skipped the grants.
+      CS_IT_DB_VARIANT: ${{ matrix.db }}
+      # Scoped to the Drizzle suites plus the adapter-agnostic ones. The Supabase
+      # adapter suites are not run here; they have their own job.
       CS_IT_SUITE: >-
         integration/drizzle-v3/**/*.integration.test.ts,
         integration/harness.integration.test.ts,

.github/workflows/integration-supabase.yml

@@ -62,6 +62,8 @@ jobs:
       # as they do on a real Supabase project.
       DATABASE_URL: postgres://postgres:password@localhost:55433/postgres
       PGRST_URL: http://localhost:55430
+      # EXPLICIT, never inferred from `PGRST_URL` — see `dbVariant()`.
+      CS_IT_DB_VARIANT: ${{ matrix.db }}
       # Scoped to the suites this database serves. The Drizzle suites talk to
       # plain Postgres and get their own job; the harness and bloom suites are
       # adapter-agnostic and run here because a database is already up.

packages/stack/integration/harness.integration.test.ts

@@ -66,26 +66,60 @@ it('installs the ORE opclass only when the connecting role is a superuser', asyn
   }
 })
 
-it.runIf(dbVariant() === 'supabase')(
-  'grants anon USAGE on both eql_v3 and eql_v3_internal',
-  async () => {
+/**
+ * Both database variants are asserted, neither is skipped.
+ *
+ * A skipped test reads exactly like a passing one, and the two skips that used to
+ * live here hid a real bug: `dbVariant()` inferred `postgres` for the Drizzle job
+ * running against `supabase/postgres`, so EQL installed without `--supabase`, the
+ * grants were never applied, and the grants test quietly did not run.
+ *
+ * So assert the whole truth: on a Supabase database the roles exist and hold the
+ * grants; on a plain one they do not exist at all.
+ */
+it('applies the anon grants on Supabase, and has no such roles on plain Postgres', async () => {
+  const [roles] = await sql<{ count: string }[]>`
+    SELECT count(*)::text AS count FROM pg_roles
+    WHERE rolname IN ('anon', 'authenticated', 'service_role')
+  `
+
+  if (dbVariant() === 'supabase') {
+    expect(Number(roles?.count)).toBe(3)
+
     // `eql_v3_internal` is load-bearing: the SECURITY INVOKER extractors resolve
     // it with the CALLER's privileges, so without this grant every encrypted
     // filter fails for `anon` with "permission denied for schema".
-    const [row] = await sql<{ eql_v3: boolean; internal: boolean }[]>`
+    const [privs] = await sql<{ eql_v3: boolean; internal: boolean }[]>`
       SELECT has_schema_privilege('anon', 'eql_v3', 'USAGE') AS eql_v3,
              has_schema_privilege('anon', 'eql_v3_internal', 'USAGE') AS internal
     `
+    expect(privs).toEqual({ eql_v3: true, internal: true })
+    return
+  }
 
-    expect(row).toEqual({ eql_v3: true, internal: true })
-  },
-)
+  // Plain Postgres: the Supabase roles do not exist, so there is nothing to
+  // grant. Asserting their ABSENCE is what makes the variant claim falsifiable —
+  // if this database ever grew them, the `--supabase` install path would have to
+  // run here too.
+  expect(Number(roles?.count)).toBe(0)
+})
 
-it.runIf(dbVariant() === 'supabase')(
-  'serves PostgREST as anon through the authenticator role',
-  async () => {
-    const response = await fetch(`${pgrestUrl()}/`)
+/**
+ * PostgREST is asserted on its own axis, not on the variant. The Drizzle job runs
+ * against the Supabase database and does not need PostgREST; conflating "is this
+ * Supabase" with "is PostgREST up" is precisely what made `dbVariant()` lie.
+ */
+it('serves PostgREST when configured, and is not configured otherwise', async () => {
+  const url = process.env['PGRST_URL']
 
-    expect(response.status).toBe(200)
-  },
-)
+  if (!url) {
+    // Only the plain-Postgres compose file omits PostgREST. A Supabase database
+    // with no `PGRST_URL` means the job forgot to pass it.
+    expect(dbVariant()).toBe('postgres')
+    return
+  }
+
+  const response = await fetch(`${pgrestUrl()}/`)
+
+  expect(response.status).toBe(200)
+})

packages/stack/integration/no-skips-reporter.ts

@@ -0,0 +1,47 @@
+import type { Reporter, TestModule } from 'vitest/node'
+
+/**
+ * Fail the integration run if any test was skipped.
+ *
+ * A skipped test reads exactly like a passing one. Every silent hole this suite
+ * has found took that shape: `contains` never ran on `text_match` because its
+ * needle was `''`; the non-ASCII ORE needle test had zero cases after the OPE
+ * re-pin; the Supabase grants check quietly did not run for the Drizzle job
+ * because `dbVariant()` mis-inferred the database. In each case a green run
+ * reported coverage it did not have.
+ *
+ * So the integration suites carry no `skip`, no `todo`, no `runIf` and no
+ * `skipIf`. Environmental differences are asserted rather than skipped: a plain
+ * Postgres has no `anon` role, and that is a fact worth stating. Where a
+ * behaviour genuinely cannot be exercised — the block-ORE domains cannot hold
+ * data on managed Postgres — the domain is excluded from the matrix by the
+ * catalog's `deferred` field, and a separate, PASSING test asserts that the
+ * excluded set is exactly what it should be.
+ *
+ * The unit suites still skip (the `LIVE_*` gates), so this is scoped to the
+ * integration config rather than imposed repo-wide.
+ */
+export default class NoSkipsReporter implements Reporter {
+  private readonly skipped: string[] = []
+
+  onTestModuleEnd(testModule: TestModule): void {
+    for (const test of Array.from(testModule.children.allTests())) {
+      const result = test.result()
+      if (result.state === 'skipped') this.skipped.push(test.fullName)
+    }
+  }
+
+  onTestRunEnd(): void {
+    if (this.skipped.length === 0) return
+
+    const list = this.skipped.map((name) => `  - ${name}`).join('\n')
+    console.error(
+      `\n${this.skipped.length} test(s) were SKIPPED. The integration suites must not skip:\n` +
+        `${list}\n\n` +
+        'A skipped test reads exactly like a passing one. Assert the environmental\n' +
+        "difference instead, or exclude the domain via the catalog's `deferred`\n" +
+        'field and assert the excluded set separately.\n',
+    )
+    process.exitCode = 1
+  }
+}

packages/stack/integration/vitest.config.ts

@@ -80,5 +80,9 @@ export default defineConfig({
      * only place to do it without changing product logging behaviour.
      */
     silent: 'passed-only',
+
+    // Fail the run if anything is skipped. A skipped test reads exactly like a
+    // passing one, and every silent hole this suite has found took that shape.
+    reporters: ['default', resolve(__dirname, 'no-skips-reporter.ts')],
   },
 })

packages/test-kit/src/install.ts

@@ -15,10 +15,15 @@ export type DbVariant = 'postgres' | 'supabase'
 /**
  * Which database the integration suite is pointed at.
  *
- * `PGRST_URL` is only ever set for the Supabase variant — the Drizzle suite
- * talks straight to Postgres — so its presence identifies the variant without a
- * second environment variable to keep in sync. `CS_IT_DB_VARIANT` overrides,
- * for the case where that stops being true.
+ * Set `CS_IT_DB_VARIANT` explicitly. The `PGRST_URL` fallback exists only for a
+ * developer running one suite by hand, and it is a guess: it once inferred
+ * `postgres` for the Drizzle job running against `supabase/postgres`, because
+ * that job needs no PostgREST and leaves the variable unset. The consequence was
+ * silent — EQL installed without `--supabase`, so the role grants were never
+ * applied and the grants test skipped, while the suite passed because Drizzle
+ * connects as `postgres` rather than as `anon`.
+ *
+ * The variant decides how EQL is installed. It must not be a guess in CI.
  */
 export function dbVariant(): DbVariant {
   const explicit = process.env['CS_IT_DB_VARIANT']

packages/test-kit/src/run-family-suite.ts

@@ -1,12 +1,7 @@
 import { afterAll, beforeAll, describe, expect, it } from 'vitest'
 import type { IntegrationAdapter } from './adapter.ts'
 import type { DomainSpec } from './catalog.ts'
-import {
-  deferredForFamily,
-  domainsForFamily,
-  type FamilyDomain,
-  type FamilyName,
-} from './families.ts'
+import { domainsForFamily, type FamilyName } from './families.ts'
 import { negativeOps, type Plain, positiveOps, type QueryOp } from './ops.ts'
 import {
   comparePlain,
@@ -148,7 +143,6 @@ export function runFamilySuite(
 ): void {
   const adapter = makeAdapter()
   const domains = domainsForFamily(family)
-  const deferred = deferredForFamily(family)
 
   // Unique per run so a crashed run never leaves a table that shadows the next.
   const runId = Math.random().toString(36).slice(2, 8)
@@ -179,9 +173,6 @@ export function runFamilySuite(
       await adapter.teardown()
     })
 
-    if (deferred.length > 0) {
-      it.skip(`defers ${deferred.map((d) => d.bare).join(', ')}: ${deferred[0]?.reason}`, () => {})
-    }
 
     for (const domain of domains) {
       describe(domain.bare, () => {