ci(integration): isolate live-DB stacks instead of serialising them (#731)

* ci(integration): isolate live-DB stacks instead of serialising them

Integration jobs were being CANCELLED, not run — four times in ~80 minutes
on 2026-07-20, most recently the `db=supabase` leg of #712, which is the
exact variant that PR exists to fix.

Cause: the three integration workflows share one job-level concurrency
group, `integration-live-db-<db>`, deliberately not scoped by ref
(d02bf63) because two PRs contend for a fixed host port as much as two
pushes to one PR. But GitHub Actions allows only ONE in-progress plus ONE
pending run per group — a third contender is cancelled outright. With
three workflows funnelling into two keys, that fires under any load.

The failure mode is worse than a flake: a cancelled job never runs, so
its check goes ABSENT rather than red. The signal is silently lost, and
`main` is not branch-protected, so nothing blocks the merge.

Remove the contention rather than serialising it. A new `integration-db`
composite action gives every job:

  - its own compose project name (run id + attempt + job + variant), so
    container names cannot collide; and
  - EPHEMERAL host ports — `CS_PG_PORTS` / `CS_PGRST_PORTS` carry a bare
    container port, so compose publishes with no fixed host side and the
    assigned port is read back with `docker compose port`.

Nothing is shared between jobs, so nothing needs to queue, and the
concurrency groups are gone from all three workflows. The compose files
keep their fixed 55432 / 55433 / 55430 mappings when the vars are unset,
so local dev and the help text in `test-kit/src/env.ts` are unchanged.

Dropped the pre-`up` `docker compose down` too. With unique project names
it is a no-op, and blanket-pruning is now actively unsafe: without the
concurrency group, another job's stack may be live on the same runner.

Also, since this adds a composite action: `.github/actions/` had no
CODEOWNERS rule, though such actions run arbitrary steps in the same job
with the same secrets as the workflow calling them. Added the rule and
the matching assertion in `supply-chain.e2e.test.ts` (verified it fails
without it). Fixed a stale claim in the Drizzle workflow header that
`PGRST_URL` is left unset — the matrix had been setting it.

Verified: compose interpolation resolves to the fixed mapping by default
and to an ephemeral publish under the CI vars (`docker compose config`);
YAML parses; the action's shell body is syntax-clean with no GitHub
templating interpolated into it; 18/18 supply-chain e2e tests pass.

* fix(ci): validate every port readback, not just Postgres

CodeRabbit caught a real asymmetry in the new action: `pg_port` was checked
for emptiness but the supabase branch built `pgrest_url` straight from
`$(port_of postgrest 3000)`. `docker compose port` exits 0 with EMPTY
stdout when a port is not bound, so an unbound PostgREST would have
silently yielded `http://localhost:` and handed every `PGRST_URL` consumer
(the Drizzle-supabase leg, both Supabase-workflow steps) a malformed URL
instead of the immediate error the Postgres path already gave.

Move the check into `port_of` itself so it applies by construction and the
asymmetry cannot come back, rather than adding a second call-site guard.
Each port is now captured in its own bare assignment so `set -e` fails the
step unambiguously on a non-zero `port_of`.

Verified with a stubbed `docker` that returns a port for `db` and nothing
for `postgrest`: the Postgres read succeeds, the guard fires for postgrest,
and the script exits 1 before exporting anything.

Author: coderdan

.github/CODEOWNERS

@@ -4,6 +4,9 @@
 # Supply-chain critical paths — changes here require explicit review
 # (lirantal/npm-security-best-practices; see skills/stash-supply-chain-security/).
 /.github/workflows/                     @cipherstash/developers
+# Composite actions are workflow code by another name: they run arbitrary steps
+# in the same job, with the same secrets, so they need the same review gate.
+/.github/actions/                       @cipherstash/developers
 /.github/dependabot.yml                 @cipherstash/developers
 /.github/CODEOWNERS                     @cipherstash/developers
 /pnpm-workspace.yaml                    @cipherstash/developers

.github/actions/integration-db/action.yml

@@ -0,0 +1,113 @@
+name: Integration database
+description: >-
+  Bring up an integration docker-compose stack that is ISOLATED from every other
+  job on the runner, and hand back the URLs to reach it.
+
+  Isolation is the point. These stacks used to bind fixed host ports
+  (55430 / 55432 / 55433) under a fixed compose project name, so two jobs
+  running the same variant on one runner collided with "address already in
+  use". The guard was a cross-workflow job-level concurrency group
+  (`integration-live-db-<db>`), but GitHub Actions allows only ONE in-progress
+  plus ONE pending run per group — a third contender is CANCELLED outright,
+  not queued. With three integration workflows sharing each key, that fired
+  regularly and silently dropped the signal for whichever variant lost (a
+  cancelled job never runs, so its check is absent rather than red).
+
+  This action removes the contention instead of serialising it: a unique
+  compose project name per job, and EPHEMERAL host ports assigned by Docker
+  (`CS_PG_PORTS` / `CS_PGRST_PORTS` carry a bare container port, so the compose
+  files publish with no fixed host side) read back via `docker compose port`.
+  Nothing is shared between jobs, so nothing needs to queue.
+
+inputs:
+  db:
+    description: Compose variant to bring up — `postgres` or `supabase`.
+    required: true
+
+outputs:
+  database-url:
+    description: Direct Postgres URL for the stack this job just started.
+    value: ${{ steps.up.outputs.database-url }}
+  pgrest-url:
+    description: PostgREST base URL. Empty for the `postgres` variant, which runs none.
+    value: ${{ steps.up.outputs.pgrest-url }}
+
+runs:
+  using: composite
+  steps:
+    - name: Start ${{ inputs.db }}
+      id: up
+      shell: bash
+      env:
+        # A bare container port publishes to an ephemeral host port. The compose
+        # files fall back to their fixed local-dev mappings when these are unset.
+        CS_PG_PORTS: '5432'
+        CS_PGRST_PORTS: '3000'
+        # Passed through the environment rather than templated into the script
+        # body: `${{ }}` is textual substitution, so interpolating a value
+        # straight into shell is a script-injection seam. These are static today,
+        # but the safe shape costs nothing.
+        DB_VARIANT: ${{ inputs.db }}
+        JOB_ID: ${{ github.job }}
+      run: |
+        set -euo pipefail
+
+        # Unique per job AND per attempt: a re-run must not adopt the previous
+        # attempt's containers. `github.job` alone is not enough — matrix legs of
+        # one job share it — so the variant goes in too.
+        project="cs-it-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${JOB_ID}-${DB_VARIANT}"
+        # Compose project names must be [a-z0-9][a-z0-9_-]*.
+        project="$(printf '%s' "$project" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9_-' '-')"
+        file="local/docker-compose.${DB_VARIANT}.yml"
+
+        # Exported for the teardown step, which must target THIS job's stack.
+        echo "CS_COMPOSE_PROJECT=$project" >> "$GITHUB_ENV"
+        echo "CS_COMPOSE_FILE=$file" >> "$GITHUB_ENV"
+
+        docker compose -p "$project" -f "$file" up -d --wait
+
+        # `docker compose port` prints `0.0.0.0:49153`, and may print one line
+        # per address family — take the first, then the port after the last
+        # colon (IPv6 forms contain several).
+        #
+        # The emptiness check lives HERE, not at the call sites: `docker compose
+        # port` exits 0 with empty stdout when a port is not bound, so "" is the
+        # failure signal and every caller needs the same guard. Checking once, by
+        # construction, is what stops the asymmetry this function had in review —
+        # Postgres was validated and PostgREST was not, so an unbound PostgREST
+        # would have silently produced `http://localhost:` and handed every
+        # consumer a malformed URL instead of an immediate, clear error.
+        port_of() {
+          local svc="$1" cport="$2" hport
+          hport="$(docker compose -p "$project" -f "$file" port "$svc" "$cport" \
+            | head -n 1 | sed 's/.*://' | tr -d '[:space:]')"
+          if [ -z "$hport" ]; then
+            echo "::error::could not read the published host port for service '$svc' (container port $cport) in the '$DB_VARIANT' stack" >&2
+            docker compose -p "$project" -f "$file" ps >&2
+            return 1
+          fi
+          printf '%s' "$hport"
+        }
+
+        # Each port is captured in its own bare assignment so `set -e` fails the
+        # step on a non-zero `port_of` unambiguously.
+        case "$DB_VARIANT" in
+          postgres)
+            pg_port="$(port_of postgres 5432)"
+            database_url="postgres://cipherstash:password@localhost:${pg_port}/cipherstash"
+            pgrest_url=''
+            ;;
+          supabase)
+            pg_port="$(port_of db 5432)"
+            pgrest_port="$(port_of postgrest 3000)"
+            database_url="postgres://postgres:password@localhost:${pg_port}/postgres"
+            pgrest_url="http://localhost:${pgrest_port}"
+            ;;
+          *)
+            echo "::error::unknown db variant '$DB_VARIANT'" && exit 1
+            ;;
+        esac
+
+        echo "database-url=$database_url" >> "$GITHUB_OUTPUT"
+        echo "pgrest-url=$pgrest_url" >> "$GITHUB_OUTPUT"
+        echo "Started '$project' — postgres on :$pg_port${pgrest_url:+, postgrest on ${pgrest_url##*:}}"

.github/workflows/integration-drizzle.yml

@@ -6,8 +6,7 @@ name: Integration — Drizzle (EQL v3)
 # PostgREST — but it does need to work on managed Postgres, where the `postgres`
 # role is not a superuser, the EQL install takes its self-skipping path, and the
 # ORE domains cannot hold data. The Supabase compose file brings up PostgREST
-# too; this job simply ignores it, and leaves `PGRST_URL` unset so a Supabase
-# suite scoped here would throw rather than silently skip.
+# too; the Drizzle suites simply ignore it.
 #
 # Separate from `tests.yml` on purpose: these suites need CipherStash credentials
 # and a database, and they THROW rather than skip when unconfigured. Keeping them
@@ -35,6 +34,7 @@ on:
       - 'local/supabase-init.sql'
       - '.github/workflows/integration-drizzle.yml'
       - '.github/actions/integration-setup/**'
+      - '.github/actions/integration-db/**'
   pull_request:
     branches: ['**']
     # Repeated verbatim: GitHub Actions does not support YAML anchors/aliases.
@@ -57,23 +57,18 @@ on:
       - 'local/supabase-init.sql'
       - '.github/workflows/integration-drizzle.yml'
       - '.github/actions/integration-setup/**'
+      - '.github/actions/integration-db/**'
 
 jobs:
   integration:
     name: Drizzle v3 integration (db=${{ matrix.db }})
     runs-on: blacksmith-4vcpu-ubuntu-2404
-    # Serialize every job that brings up the SAME docker-compose stack, so no two
-    # bind the same fixed host ports on a shared runner at once (the "address
-    # already in use" flake). Keyed by the compose variant, NOT the ref (two PRs
-    # contend for a host port just as much as two pushes to one PR), so it
-    # serializes across refs. The `supabase` leg uses the SAME supabase compose
-    # (55430 / 55433) as the Supabase workflow and shares its key
-    # (`integration-live-db-supabase`), so the two workflows queue rather than
-    # collide; the `postgres` leg (55432) has its own key and still runs in
-    # parallel with them.
-    concurrency:
-      group: integration-live-db-${{ matrix.db }}
-      cancel-in-progress: false
+    # No concurrency group: `integration-db` gives each job its own compose
+    # project and ephemeral host ports, so live-DB jobs no longer contend and do
+    # not need serialising. See that action for why the old
+    # `integration-live-db-<db>` group had to go (it cancelled a third
+    # contender rather than queueing it).
+    #
     # Fork PRs have no secrets. Skip cleanly rather than fail on something the
     # contributor cannot fix — `tests.yml` still gives them a green signal.
     if: ${{ github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository }}
@@ -85,17 +80,12 @@ jobs:
       # superuser, so the EQL install takes its self-skipping path and the ORE
       # domains become unusable. A suite that passes on a superuser database can
       # still fail there.
+      # (The supabase compose file starts PostgREST anyway. Drizzle does not use
+      # it, but `integration-db` supplies the URL regardless, which lets the
+      # harness assert the `anon` path on the database it is actually running
+      # against.)
       matrix:
-        include:
-          - db: postgres
-            database-url: postgres://cipherstash:password@localhost:55432/cipherstash
-            pgrest-url: ''
-          - db: supabase
-            database-url: postgres://postgres:password@localhost:55433/postgres
-            # The supabase compose file starts PostgREST anyway. Drizzle does not
-            # use it, but supplying the URL lets the harness assert the `anon`
-            # path on the database it is actually running against.
-            pgrest-url: http://localhost:55430
+        db: [postgres, supabase]
 
     env:
       CS_WORKSPACE_CRN: ${{ vars.CS_WORKSPACE_CRN }}
@@ -110,10 +100,6 @@ jobs:
       # instance, used only by the cross-identity test.
       CLERK_MACHINE_TOKEN: ${{ secrets.CLERK_MACHINE_TOKEN }}
       CLERK_MACHINE_TOKEN_B: ${{ secrets.CLERK_MACHINE_TOKEN_B }}
-      # Job-level env, not a `.env` file: `dotenv/config` does not override an
-      # already-set `process.env`, so these win and no secret is written to disk.
-      DATABASE_URL: ${{ matrix.database-url }}
-      PGRST_URL: ${{ matrix.pgrest-url }}
       # EXPLICIT, never inferred. The variant decides whether EQL is installed
       # with `--supabase` (and therefore whether the role grants are applied).
       # Inferring it from `PGRST_URL` reported `postgres` for this job's Supabase
@@ -152,30 +138,42 @@ jobs:
           client-key: ${{ secrets.CS_CLIENT_KEY }}
           client-access-key: ${{ secrets.CS_CLIENT_ACCESS_KEY }}
 
-      # Belt-and-braces for the concurrency guard: a run that was hard-killed
-      # (runner crash / forced cancel) can skip its `down` and leak a container
-      # holding the host port onto a REUSED runner. Tear any prior stack down
-      # before starting a fresh one. `|| true` so a clean runner (nothing to
-      # remove) is not an error.
-      - name: Clear any leaked containers from a prior run
-        run: docker compose -f local/docker-compose.${{ matrix.db }}.yml down -v --remove-orphans || true
-
+      # No pre-`up` cleanup step any more: the project name is unique per job, so
+      # a container leaked by a hard-killed prior run cannot hold this job's
+      # name or its (ephemeral) port. Blanket-pruning would now be actively
+      # unsafe — without the concurrency group, another job's stack may be live
+      # on this runner.
       - name: Start ${{ matrix.db }}
-        run: docker compose -f local/docker-compose.${{ matrix.db }}.yml up -d --wait
+        id: db
+        uses: ./.github/actions/integration-db
+        with:
+          db: ${{ matrix.db }}
 
       # `globalSetup` installs EQL v3 by shelling out to the real
       # `stash eql install --eql-version 3`, so an installer regression fails
       # here rather than hiding behind a test-only SQL apply.
+      #
+      # Step env, not a `.env` file: `dotenv/config` does not override an
+      # already-set `process.env`, so these win and no secret is written to disk.
       - name: Drizzle v3 integration suites
         run: pnpm --filter @cipherstash/stack-drizzle run test:integration
+        env:
+          DATABASE_URL: ${{ steps.db.outputs.database-url }}
+          PGRST_URL: ${{ steps.db.outputs.pgrest-url }}
 
       # A second vitest invocation (stack's shared/ + identity suites live in a
       # different package now). Its globalSetup calls the same EQL v3 install, but
       # `isInstalled` short-circuits against the DB the first invocation already
       # provisioned — a fast no-op check, not a second schema apply.
       - name: Shared core + identity + wasm integration suites
         run: pnpm --filter @cipherstash/stack run test:integration
+        env:
+          DATABASE_URL: ${{ steps.db.outputs.database-url }}
+          PGRST_URL: ${{ steps.db.outputs.pgrest-url }}
 
+      # Guarded on the project being set: if the stack never came up, there is
+      # nothing to tear down and an unguarded `-p ""` would fail the job with a
+      # confusing error that masks the real one.
       - name: Stop ${{ matrix.db }}
-        if: always()
-        run: docker compose -f local/docker-compose.${{ matrix.db }}.yml down -v
+        if: always() && env.CS_COMPOSE_PROJECT != ''
+        run: docker compose -p "$CS_COMPOSE_PROJECT" -f "$CS_COMPOSE_FILE" down -v

.github/workflows/integration-prisma-next.yml

@@ -33,6 +33,7 @@ on:
       - 'local/supabase-init.sql'
       - '.github/workflows/integration-prisma-next.yml'
       - '.github/actions/integration-setup/**'
+      - '.github/actions/integration-db/**'
   pull_request:
     branches: ['**']
     # Repeated verbatim: GitHub Actions does not support YAML anchors/aliases.
@@ -52,17 +53,18 @@ on:
       - 'local/supabase-init.sql'
       - '.github/workflows/integration-prisma-next.yml'
       - '.github/actions/integration-setup/**'
+      - '.github/actions/integration-db/**'
 
 jobs:
   integration:
     name: prisma-next v3 integration (db=${{ matrix.db }})
     runs-on: blacksmith-4vcpu-ubuntu-2404
-    # Serialize every job that brings up the SAME docker-compose stack — same
-    # rationale and keys as the Drizzle/Supabase workflows, so the jobs queue
-    # on a shared runner rather than collide on the fixed host ports.
-    concurrency:
-      group: integration-live-db-${{ matrix.db }}
-      cancel-in-progress: false
+    # No concurrency group: `integration-db` gives each job its own compose
+    # project and ephemeral host ports, so live-DB jobs no longer contend and do
+    # not need serialising. See that action for why the old
+    # `integration-live-db-<db>` group had to go (it cancelled a third
+    # contender rather than queueing it).
+    #
     # Fork PRs have no secrets. Skip cleanly rather than fail on something the
     # contributor cannot fix — `tests.yml` still gives them a green signal.
     if: ${{ github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository }}
@@ -74,23 +76,13 @@ jobs:
       # is not a superuser, so the EQL install takes its self-skipping path. A
       # suite that passes on a superuser database can still fail there.
       matrix:
-        include:
-          - db: postgres
-            database-url: postgres://cipherstash:password@localhost:55432/cipherstash
-            pgrest-url: ''
-          - db: supabase
-            database-url: postgres://postgres:password@localhost:55433/postgres
-            pgrest-url: http://localhost:55430
+        db: [postgres, supabase]
 
     env:
       CS_WORKSPACE_CRN: ${{ vars.CS_WORKSPACE_CRN }}
       CS_CLIENT_ID: ${{ vars.CS_CLIENT_ID }}
       CS_CLIENT_KEY: ${{ secrets.CS_CLIENT_KEY }}
       CS_CLIENT_ACCESS_KEY: ${{ secrets.CS_CLIENT_ACCESS_KEY }}
-      # Job-level env, not a `.env` file: `dotenv/config` does not override an
-      # already-set `process.env`, so these win and no secret is written to disk.
-      DATABASE_URL: ${{ matrix.database-url }}
-      PGRST_URL: ${{ matrix.pgrest-url }}
       # EXPLICIT, never inferred — same rationale as the Drizzle workflow.
       CS_IT_DB_VARIANT: ${{ matrix.db }}
 
@@ -109,21 +101,31 @@ jobs:
           client-key: ${{ secrets.CS_CLIENT_KEY }}
           client-access-key: ${{ secrets.CS_CLIENT_ACCESS_KEY }}
 
-      # Belt-and-braces for the concurrency guard: a run that was hard-killed
-      # can skip its `down` and leak a container holding the host port onto a
-      # REUSED runner. `|| true` so a clean runner is not an error.
-      - name: Clear any leaked containers from a prior run
-        run: docker compose -f local/docker-compose.${{ matrix.db }}.yml down -v --remove-orphans || true
-
+      # No pre-`up` cleanup step any more: the project name is unique per job, so
+      # a container leaked by a hard-killed prior run cannot hold this job's
+      # name or its (ephemeral) port. Blanket-pruning would now be actively
+      # unsafe — without the concurrency group, another job's stack may be live
+      # on this runner.
       - name: Start ${{ matrix.db }}
-        run: docker compose -f local/docker-compose.${{ matrix.db }}.yml up -d --wait
+        id: db
+        uses: ./.github/actions/integration-db
+        with:
+          db: ${{ matrix.db }}
 
       # `globalSetup` installs EQL v3 by shelling out to the real
       # `stash eql install --eql-version 3`, so an installer regression fails
       # here rather than hiding behind a test-only SQL apply.
       - name: prisma-next v3 family suites
         run: pnpm --filter @cipherstash/prisma-next run test:integration
+        env:
+          # Step env, not a `.env` file: `dotenv/config` does not override an
+          # already-set `process.env`, so these win and no secret hits disk.
+          DATABASE_URL: ${{ steps.db.outputs.database-url }}
+          PGRST_URL: ${{ steps.db.outputs.pgrest-url }}
 
+      # Guarded on the project being set: if the stack never came up, there is
+      # nothing to tear down and an unguarded `-p ""` would fail the job with a
+      # confusing error that masks the real one.
       - name: Stop ${{ matrix.db }}
-        if: always()
-        run: docker compose -f local/docker-compose.${{ matrix.db }}.yml down -v
+        if: always() && env.CS_COMPOSE_PROJECT != ''
+        run: docker compose -p "$CS_COMPOSE_PROJECT" -f "$CS_COMPOSE_FILE" down -v