From c28b825f2aac7189c2f00e60037d5bee932b8460 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 21 Jul 2026 13:12:25 +1000 Subject: [PATCH 1/2] ci(integration): isolate live-DB stacks instead of serialising them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Integration jobs were being CANCELLED, not run — four times in ~80 minutes on 2026-07-20, most recently the `db=supabase` leg of #712, which is the exact variant that PR exists to fix. Cause: the three integration workflows share one job-level concurrency group, `integration-live-db-`, deliberately not scoped by ref (d02bf63f) because two PRs contend for a fixed host port as much as two pushes to one PR. But GitHub Actions allows only ONE in-progress plus ONE pending run per group — a third contender is cancelled outright. With three workflows funnelling into two keys, that fires under any load. The failure mode is worse than a flake: a cancelled job never runs, so its check goes ABSENT rather than red. The signal is silently lost, and `main` is not branch-protected, so nothing blocks the merge. Remove the contention rather than serialising it. A new `integration-db` composite action gives every job: - its own compose project name (run id + attempt + job + variant), so container names cannot collide; and - EPHEMERAL host ports — `CS_PG_PORTS` / `CS_PGRST_PORTS` carry a bare container port, so compose publishes with no fixed host side and the assigned port is read back with `docker compose port`. Nothing is shared between jobs, so nothing needs to queue, and the concurrency groups are gone from all three workflows. The compose files keep their fixed 55432 / 55433 / 55430 mappings when the vars are unset, so local dev and the help text in `test-kit/src/env.ts` are unchanged. Dropped the pre-`up` `docker compose down` too. With unique project names it is a no-op, and blanket-pruning is now actively unsafe: without the concurrency group, another job's stack may be live on the same runner. Also, since this adds a composite action: `.github/actions/` had no CODEOWNERS rule, though such actions run arbitrary steps in the same job with the same secrets as the workflow calling them. Added the rule and the matching assertion in `supply-chain.e2e.test.ts` (verified it fails without it). Fixed a stale claim in the Drizzle workflow header that `PGRST_URL` is left unset — the matrix had been setting it. Verified: compose interpolation resolves to the fixed mapping by default and to an ephemeral publish under the CI vars (`docker compose config`); YAML parses; the action's shell body is syntax-clean with no GitHub templating interpolated into it; 18/18 supply-chain e2e tests pass. --- .github/CODEOWNERS | 3 + .github/actions/integration-db/action.yml | 103 ++++++++++++++++++ .github/workflows/integration-drizzle.yml | 76 +++++++------ .github/workflows/integration-prisma-next.yml | 54 ++++----- .github/workflows/integration-supabase.yml | 67 +++++++----- e2e/tests/supply-chain.e2e.test.ts | 4 + local/docker-compose.postgres.yml | 8 +- local/docker-compose.supabase.yml | 10 +- 8 files changed, 227 insertions(+), 98 deletions(-) create mode 100644 .github/actions/integration-db/action.yml diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 08f82b188..671089d21 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -4,6 +4,9 @@ # Supply-chain critical paths — changes here require explicit review # (lirantal/npm-security-best-practices; see skills/stash-supply-chain-security/). /.github/workflows/ @cipherstash/developers +# Composite actions are workflow code by another name: they run arbitrary steps +# in the same job, with the same secrets, so they need the same review gate. +/.github/actions/ @cipherstash/developers /.github/dependabot.yml @cipherstash/developers /.github/CODEOWNERS @cipherstash/developers /pnpm-workspace.yaml @cipherstash/developers diff --git a/.github/actions/integration-db/action.yml b/.github/actions/integration-db/action.yml new file mode 100644 index 000000000..7d0963933 --- /dev/null +++ b/.github/actions/integration-db/action.yml @@ -0,0 +1,103 @@ +name: Integration database +description: >- + Bring up an integration docker-compose stack that is ISOLATED from every other + job on the runner, and hand back the URLs to reach it. + + Isolation is the point. These stacks used to bind fixed host ports + (55430 / 55432 / 55433) under a fixed compose project name, so two jobs + running the same variant on one runner collided with "address already in + use". The guard was a cross-workflow job-level concurrency group + (`integration-live-db-`), but GitHub Actions allows only ONE in-progress + plus ONE pending run per group — a third contender is CANCELLED outright, + not queued. With three integration workflows sharing each key, that fired + regularly and silently dropped the signal for whichever variant lost (a + cancelled job never runs, so its check is absent rather than red). + + This action removes the contention instead of serialising it: a unique + compose project name per job, and EPHEMERAL host ports assigned by Docker + (`CS_PG_PORTS` / `CS_PGRST_PORTS` carry a bare container port, so the compose + files publish with no fixed host side) read back via `docker compose port`. + Nothing is shared between jobs, so nothing needs to queue. + +inputs: + db: + description: Compose variant to bring up — `postgres` or `supabase`. + required: true + +outputs: + database-url: + description: Direct Postgres URL for the stack this job just started. + value: ${{ steps.up.outputs.database-url }} + pgrest-url: + description: PostgREST base URL. Empty for the `postgres` variant, which runs none. + value: ${{ steps.up.outputs.pgrest-url }} + +runs: + using: composite + steps: + - name: Start ${{ inputs.db }} + id: up + shell: bash + env: + # A bare container port publishes to an ephemeral host port. The compose + # files fall back to their fixed local-dev mappings when these are unset. + CS_PG_PORTS: '5432' + CS_PGRST_PORTS: '3000' + # Passed through the environment rather than templated into the script + # body: `${{ }}` is textual substitution, so interpolating a value + # straight into shell is a script-injection seam. These are static today, + # but the safe shape costs nothing. + DB_VARIANT: ${{ inputs.db }} + JOB_ID: ${{ github.job }} + run: | + set -euo pipefail + + # Unique per job AND per attempt: a re-run must not adopt the previous + # attempt's containers. `github.job` alone is not enough — matrix legs of + # one job share it — so the variant goes in too. + project="cs-it-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${JOB_ID}-${DB_VARIANT}" + # Compose project names must be [a-z0-9][a-z0-9_-]*. + project="$(printf '%s' "$project" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9_-' '-')" + file="local/docker-compose.${DB_VARIANT}.yml" + + # Exported for the teardown step, which must target THIS job's stack. + echo "CS_COMPOSE_PROJECT=$project" >> "$GITHUB_ENV" + echo "CS_COMPOSE_FILE=$file" >> "$GITHUB_ENV" + + docker compose -p "$project" -f "$file" up -d --wait + + # `docker compose port` prints `0.0.0.0:49153`, and may print one line + # per address family — take the first, then the port after the last + # colon (IPv6 forms contain several). + port_of() { + docker compose -p "$project" -f "$file" port "$1" "$2" \ + | head -n 1 | sed 's/.*://' | tr -d '[:space:]' + } + + case "$DB_VARIANT" in + postgres) + pg_port="$(port_of postgres 5432)" + database_url="postgres://cipherstash:password@localhost:${pg_port}/cipherstash" + pgrest_url='' + ;; + supabase) + pg_port="$(port_of db 5432)" + database_url="postgres://postgres:password@localhost:${pg_port}/postgres" + pgrest_url="http://localhost:$(port_of postgrest 3000)" + ;; + *) + echo "::error::unknown db variant '$DB_VARIANT'" && exit 1 + ;; + esac + + # Fail loudly here rather than letting a suite connect to port "" and + # report an unrelated error. + if [ -z "${pg_port:-}" ]; then + echo "::error::could not read the published Postgres port for '$DB_VARIANT'" + docker compose -p "$project" -f "$file" ps + exit 1 + fi + + echo "database-url=$database_url" >> "$GITHUB_OUTPUT" + echo "pgrest-url=$pgrest_url" >> "$GITHUB_OUTPUT" + echo "Started '$project' — postgres on :$pg_port${pgrest_url:+, postgrest on ${pgrest_url##*:}}" diff --git a/.github/workflows/integration-drizzle.yml b/.github/workflows/integration-drizzle.yml index 3db687917..ef8a42adc 100644 --- a/.github/workflows/integration-drizzle.yml +++ b/.github/workflows/integration-drizzle.yml @@ -6,8 +6,7 @@ name: Integration — Drizzle (EQL v3) # PostgREST — but it does need to work on managed Postgres, where the `postgres` # role is not a superuser, the EQL install takes its self-skipping path, and the # ORE domains cannot hold data. The Supabase compose file brings up PostgREST -# too; this job simply ignores it, and leaves `PGRST_URL` unset so a Supabase -# suite scoped here would throw rather than silently skip. +# too; the Drizzle suites simply ignore it. # # Separate from `tests.yml` on purpose: these suites need CipherStash credentials # and a database, and they THROW rather than skip when unconfigured. Keeping them @@ -35,6 +34,7 @@ on: - 'local/supabase-init.sql' - '.github/workflows/integration-drizzle.yml' - '.github/actions/integration-setup/**' + - '.github/actions/integration-db/**' pull_request: branches: ['**'] # Repeated verbatim: GitHub Actions does not support YAML anchors/aliases. @@ -57,23 +57,18 @@ on: - 'local/supabase-init.sql' - '.github/workflows/integration-drizzle.yml' - '.github/actions/integration-setup/**' + - '.github/actions/integration-db/**' jobs: integration: name: Drizzle v3 integration (db=${{ matrix.db }}) runs-on: blacksmith-4vcpu-ubuntu-2404 - # Serialize every job that brings up the SAME docker-compose stack, so no two - # bind the same fixed host ports on a shared runner at once (the "address - # already in use" flake). Keyed by the compose variant, NOT the ref (two PRs - # contend for a host port just as much as two pushes to one PR), so it - # serializes across refs. The `supabase` leg uses the SAME supabase compose - # (55430 / 55433) as the Supabase workflow and shares its key - # (`integration-live-db-supabase`), so the two workflows queue rather than - # collide; the `postgres` leg (55432) has its own key and still runs in - # parallel with them. - concurrency: - group: integration-live-db-${{ matrix.db }} - cancel-in-progress: false + # No concurrency group: `integration-db` gives each job its own compose + # project and ephemeral host ports, so live-DB jobs no longer contend and do + # not need serialising. See that action for why the old + # `integration-live-db-` group had to go (it cancelled a third + # contender rather than queueing it). + # # Fork PRs have no secrets. Skip cleanly rather than fail on something the # contributor cannot fix — `tests.yml` still gives them a green signal. if: ${{ github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository }} @@ -85,17 +80,12 @@ jobs: # superuser, so the EQL install takes its self-skipping path and the ORE # domains become unusable. A suite that passes on a superuser database can # still fail there. + # (The supabase compose file starts PostgREST anyway. Drizzle does not use + # it, but `integration-db` supplies the URL regardless, which lets the + # harness assert the `anon` path on the database it is actually running + # against.) matrix: - include: - - db: postgres - database-url: postgres://cipherstash:password@localhost:55432/cipherstash - pgrest-url: '' - - db: supabase - database-url: postgres://postgres:password@localhost:55433/postgres - # The supabase compose file starts PostgREST anyway. Drizzle does not - # use it, but supplying the URL lets the harness assert the `anon` - # path on the database it is actually running against. - pgrest-url: http://localhost:55430 + db: [postgres, supabase] env: CS_WORKSPACE_CRN: ${{ vars.CS_WORKSPACE_CRN }} @@ -110,10 +100,6 @@ jobs: # instance, used only by the cross-identity test. CLERK_MACHINE_TOKEN: ${{ secrets.CLERK_MACHINE_TOKEN }} CLERK_MACHINE_TOKEN_B: ${{ secrets.CLERK_MACHINE_TOKEN_B }} - # Job-level env, not a `.env` file: `dotenv/config` does not override an - # already-set `process.env`, so these win and no secret is written to disk. - DATABASE_URL: ${{ matrix.database-url }} - PGRST_URL: ${{ matrix.pgrest-url }} # EXPLICIT, never inferred. The variant decides whether EQL is installed # with `--supabase` (and therefore whether the role grants are applied). # Inferring it from `PGRST_URL` reported `postgres` for this job's Supabase @@ -152,22 +138,28 @@ jobs: client-key: ${{ secrets.CS_CLIENT_KEY }} client-access-key: ${{ secrets.CS_CLIENT_ACCESS_KEY }} - # Belt-and-braces for the concurrency guard: a run that was hard-killed - # (runner crash / forced cancel) can skip its `down` and leak a container - # holding the host port onto a REUSED runner. Tear any prior stack down - # before starting a fresh one. `|| true` so a clean runner (nothing to - # remove) is not an error. - - name: Clear any leaked containers from a prior run - run: docker compose -f local/docker-compose.${{ matrix.db }}.yml down -v --remove-orphans || true - + # No pre-`up` cleanup step any more: the project name is unique per job, so + # a container leaked by a hard-killed prior run cannot hold this job's + # name or its (ephemeral) port. Blanket-pruning would now be actively + # unsafe — without the concurrency group, another job's stack may be live + # on this runner. - name: Start ${{ matrix.db }} - run: docker compose -f local/docker-compose.${{ matrix.db }}.yml up -d --wait + id: db + uses: ./.github/actions/integration-db + with: + db: ${{ matrix.db }} # `globalSetup` installs EQL v3 by shelling out to the real # `stash eql install --eql-version 3`, so an installer regression fails # here rather than hiding behind a test-only SQL apply. + # + # Step env, not a `.env` file: `dotenv/config` does not override an + # already-set `process.env`, so these win and no secret is written to disk. - name: Drizzle v3 integration suites run: pnpm --filter @cipherstash/stack-drizzle run test:integration + env: + DATABASE_URL: ${{ steps.db.outputs.database-url }} + PGRST_URL: ${{ steps.db.outputs.pgrest-url }} # A second vitest invocation (stack's shared/ + identity suites live in a # different package now). Its globalSetup calls the same EQL v3 install, but @@ -175,7 +167,13 @@ jobs: # provisioned — a fast no-op check, not a second schema apply. - name: Shared core + identity + wasm integration suites run: pnpm --filter @cipherstash/stack run test:integration + env: + DATABASE_URL: ${{ steps.db.outputs.database-url }} + PGRST_URL: ${{ steps.db.outputs.pgrest-url }} + # Guarded on the project being set: if the stack never came up, there is + # nothing to tear down and an unguarded `-p ""` would fail the job with a + # confusing error that masks the real one. - name: Stop ${{ matrix.db }} - if: always() - run: docker compose -f local/docker-compose.${{ matrix.db }}.yml down -v + if: always() && env.CS_COMPOSE_PROJECT != '' + run: docker compose -p "$CS_COMPOSE_PROJECT" -f "$CS_COMPOSE_FILE" down -v diff --git a/.github/workflows/integration-prisma-next.yml b/.github/workflows/integration-prisma-next.yml index 55eb9585b..f3b128e01 100644 --- a/.github/workflows/integration-prisma-next.yml +++ b/.github/workflows/integration-prisma-next.yml @@ -33,6 +33,7 @@ on: - 'local/supabase-init.sql' - '.github/workflows/integration-prisma-next.yml' - '.github/actions/integration-setup/**' + - '.github/actions/integration-db/**' pull_request: branches: ['**'] # Repeated verbatim: GitHub Actions does not support YAML anchors/aliases. @@ -52,17 +53,18 @@ on: - 'local/supabase-init.sql' - '.github/workflows/integration-prisma-next.yml' - '.github/actions/integration-setup/**' + - '.github/actions/integration-db/**' jobs: integration: name: prisma-next v3 integration (db=${{ matrix.db }}) runs-on: blacksmith-4vcpu-ubuntu-2404 - # Serialize every job that brings up the SAME docker-compose stack — same - # rationale and keys as the Drizzle/Supabase workflows, so the jobs queue - # on a shared runner rather than collide on the fixed host ports. - concurrency: - group: integration-live-db-${{ matrix.db }} - cancel-in-progress: false + # No concurrency group: `integration-db` gives each job its own compose + # project and ephemeral host ports, so live-DB jobs no longer contend and do + # not need serialising. See that action for why the old + # `integration-live-db-` group had to go (it cancelled a third + # contender rather than queueing it). + # # Fork PRs have no secrets. Skip cleanly rather than fail on something the # contributor cannot fix — `tests.yml` still gives them a green signal. if: ${{ github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository }} @@ -74,23 +76,13 @@ jobs: # is not a superuser, so the EQL install takes its self-skipping path. A # suite that passes on a superuser database can still fail there. matrix: - include: - - db: postgres - database-url: postgres://cipherstash:password@localhost:55432/cipherstash - pgrest-url: '' - - db: supabase - database-url: postgres://postgres:password@localhost:55433/postgres - pgrest-url: http://localhost:55430 + db: [postgres, supabase] env: CS_WORKSPACE_CRN: ${{ vars.CS_WORKSPACE_CRN }} CS_CLIENT_ID: ${{ vars.CS_CLIENT_ID }} CS_CLIENT_KEY: ${{ secrets.CS_CLIENT_KEY }} CS_CLIENT_ACCESS_KEY: ${{ secrets.CS_CLIENT_ACCESS_KEY }} - # Job-level env, not a `.env` file: `dotenv/config` does not override an - # already-set `process.env`, so these win and no secret is written to disk. - DATABASE_URL: ${{ matrix.database-url }} - PGRST_URL: ${{ matrix.pgrest-url }} # EXPLICIT, never inferred — same rationale as the Drizzle workflow. CS_IT_DB_VARIANT: ${{ matrix.db }} @@ -109,21 +101,31 @@ jobs: client-key: ${{ secrets.CS_CLIENT_KEY }} client-access-key: ${{ secrets.CS_CLIENT_ACCESS_KEY }} - # Belt-and-braces for the concurrency guard: a run that was hard-killed - # can skip its `down` and leak a container holding the host port onto a - # REUSED runner. `|| true` so a clean runner is not an error. - - name: Clear any leaked containers from a prior run - run: docker compose -f local/docker-compose.${{ matrix.db }}.yml down -v --remove-orphans || true - + # No pre-`up` cleanup step any more: the project name is unique per job, so + # a container leaked by a hard-killed prior run cannot hold this job's + # name or its (ephemeral) port. Blanket-pruning would now be actively + # unsafe — without the concurrency group, another job's stack may be live + # on this runner. - name: Start ${{ matrix.db }} - run: docker compose -f local/docker-compose.${{ matrix.db }}.yml up -d --wait + id: db + uses: ./.github/actions/integration-db + with: + db: ${{ matrix.db }} # `globalSetup` installs EQL v3 by shelling out to the real # `stash eql install --eql-version 3`, so an installer regression fails # here rather than hiding behind a test-only SQL apply. - name: prisma-next v3 family suites run: pnpm --filter @cipherstash/prisma-next run test:integration + env: + # Step env, not a `.env` file: `dotenv/config` does not override an + # already-set `process.env`, so these win and no secret hits disk. + DATABASE_URL: ${{ steps.db.outputs.database-url }} + PGRST_URL: ${{ steps.db.outputs.pgrest-url }} + # Guarded on the project being set: if the stack never came up, there is + # nothing to tear down and an unguarded `-p ""` would fail the job with a + # confusing error that masks the real one. - name: Stop ${{ matrix.db }} - if: always() - run: docker compose -f local/docker-compose.${{ matrix.db }}.yml down -v + if: always() && env.CS_COMPOSE_PROJECT != '' + run: docker compose -p "$CS_COMPOSE_PROJECT" -f "$CS_COMPOSE_FILE" down -v diff --git a/.github/workflows/integration-supabase.yml b/.github/workflows/integration-supabase.yml index 64252ff59..7ab83ca46 100644 --- a/.github/workflows/integration-supabase.yml +++ b/.github/workflows/integration-supabase.yml @@ -27,6 +27,7 @@ on: - 'local/supabase-init.sql' - '.github/workflows/integration-supabase.yml' - '.github/actions/integration-setup/**' + - '.github/actions/integration-db/**' pull_request: branches: ['**'] paths: @@ -45,23 +46,18 @@ on: - 'local/supabase-init.sql' - '.github/workflows/integration-supabase.yml' - '.github/actions/integration-setup/**' + - '.github/actions/integration-db/**' jobs: integration: name: Supabase v3 integration (db=${{ matrix.db }}) runs-on: blacksmith-4vcpu-ubuntu-2404 - # Serialize every job that brings up the SAME docker-compose stack, so no two - # bind the same fixed host ports (55430 / 55433) on a shared runner at once — - # the "address already in use" flake. The group is keyed by the compose - # variant, NOT the ref: two different PRs contend for the same host port just - # as much as two pushes to one PR, so it must serialize across refs. It is the - # SAME key the Drizzle workflow's matching leg uses (`integration-live-db-`), - # so those two workflows also queue against each other rather than colliding. - # `cancel-in-progress: false` queues (a live-DB job that shares one ZeroKMS - # workspace should finish, not be interrupted mid-run). - concurrency: - group: integration-live-db-${{ matrix.db }} - cancel-in-progress: false + # No concurrency group: `integration-db` gives each job its own compose + # project and ephemeral host ports, so live-DB jobs no longer contend and do + # not need serialising. See that action for why the old + # `integration-live-db-` group had to go (it cancelled a third + # contender rather than queueing it). + # # Fork PRs have no secrets. Skip cleanly rather than fail loudly on something # the contributor cannot fix — `tests.yml` still gives them a green signal. if: ${{ github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository }} @@ -78,14 +74,13 @@ jobs: CS_CLIENT_ID: ${{ vars.CS_CLIENT_ID }} CS_CLIENT_KEY: ${{ secrets.CS_CLIENT_KEY }} CS_CLIENT_ACCESS_KEY: ${{ secrets.CS_CLIENT_ACCESS_KEY }} - # Job-level env, not a `.env` file: `dotenv/config` does not override an - # already-set `process.env`, so these win and no secret is written to disk. + # `DATABASE_URL` / `PGRST_URL` are set per step from the `integration-db` + # outputs — the host ports are assigned by Docker at start-up, so they + # cannot be written down here. The role that URL connects as (`postgres`) + # is deliberately NOT a superuser on this image, which is what makes the + # EQL install, the grants, and the ORE opclass skip behave as they do on a + # real Supabase project. # - # `postgres` here is deliberately NOT a superuser on this image — that is - # what makes the EQL install, the grants, and the ORE opclass skip behave - # as they do on a real Supabase project. - DATABASE_URL: postgres://postgres:password@localhost:55433/postgres - PGRST_URL: http://localhost:55430 # EXPLICIT, never inferred from `PGRST_URL` — see `dbVariant()`. CS_IT_DB_VARIANT: ${{ matrix.db }} # Scoped by directory, never by named file, so a renamed suite cannot @@ -118,22 +113,28 @@ jobs: client-key: ${{ secrets.CS_CLIENT_KEY }} client-access-key: ${{ secrets.CS_CLIENT_ACCESS_KEY }} - # Belt-and-braces for the concurrency guard: a run that was hard-killed - # (runner crash / forced cancel) can skip its `down` and leak a container - # holding the host port onto a REUSED runner. Tear any prior stack down - # before starting a fresh one. `|| true` so a clean runner (nothing to - # remove) is not an error. - - name: Clear any leaked containers from a prior run - run: docker compose -f local/docker-compose.${{ matrix.db }}.yml down -v --remove-orphans || true - + # No pre-`up` cleanup step any more: the project name is unique per job, so + # a container leaked by a hard-killed prior run cannot hold this job's + # name or its (ephemeral) port. Blanket-pruning would now be actively + # unsafe — without the concurrency group, another job's stack may be live + # on this runner. - name: Start ${{ matrix.db }} - run: docker compose -f local/docker-compose.${{ matrix.db }}.yml up -d --wait + id: db + uses: ./.github/actions/integration-db + with: + db: ${{ matrix.db }} # `globalSetup` installs EQL v3 by shelling out to the real # `stash eql install --eql-version 3 --supabase --direct`, so an installer # regression fails here rather than hiding behind a test-only SQL apply. + # + # Step env, not a `.env` file: `dotenv/config` does not override an + # already-set `process.env`, so these win and no secret is written to disk. - name: Supabase v3 integration suites run: pnpm --filter @cipherstash/stack-supabase run test:integration + env: + DATABASE_URL: ${{ steps.db.outputs.database-url }} + PGRST_URL: ${{ steps.db.outputs.pgrest-url }} # A second vitest invocation (stack's shared/ suites live in a different # package now). Its globalSetup calls the same EQL v3 install, but @@ -141,7 +142,13 @@ jobs: # provisioned — so this is a fast no-op check, not a second schema apply. - name: Shared core integration suites (against Supabase Postgres) run: pnpm --filter @cipherstash/stack run test:integration + env: + DATABASE_URL: ${{ steps.db.outputs.database-url }} + PGRST_URL: ${{ steps.db.outputs.pgrest-url }} + # Guarded on the project being set: if the stack never came up, there is + # nothing to tear down and an unguarded `-p ""` would fail the job with a + # confusing error that masks the real one. - name: Stop ${{ matrix.db }} - if: always() - run: docker compose -f local/docker-compose.${{ matrix.db }}.yml down -v + if: always() && env.CS_COMPOSE_PROJECT != '' + run: docker compose -p "$CS_COMPOSE_PROJECT" -f "$CS_COMPOSE_FILE" down -v diff --git a/e2e/tests/supply-chain.e2e.test.ts b/e2e/tests/supply-chain.e2e.test.ts index c387eefd5..5dc9b87a7 100644 --- a/e2e/tests/supply-chain.e2e.test.ts +++ b/e2e/tests/supply-chain.e2e.test.ts @@ -360,6 +360,10 @@ describe('supply chain — governance (CODEOWNERS)', () => { 'dependabot.yml', '.npmrc', '.github/workflows/', + // Composite actions run arbitrary steps inside the same job, with the + // same secrets, as the workflow that calls them — same blast radius, + // same review gate. + '.github/actions/', '.github/CODEOWNERS', 'skills/stash-supply-chain-security/', ]) { diff --git a/local/docker-compose.postgres.yml b/local/docker-compose.postgres.yml index 81ce55bef..54a20e407 100644 --- a/local/docker-compose.postgres.yml +++ b/local/docker-compose.postgres.yml @@ -8,6 +8,12 @@ # their own Postgres on 5432. Binding 5432 here would either fail to start or, # worse, shadow the database they think they are talking to. # +# `CS_PG_PORTS` overrides the mapping so CI can publish to an EPHEMERAL host +# port (`CS_PG_PORTS=5432`, container port only) and read the assigned port back +# with `docker compose port`. That is what lets concurrent integration jobs share +# a runner without contending for a fixed port. Unset — the local default — keeps +# the stable `55432` a developer can put in a connection string. +# # Connect DIRECT to the database, never through a pooler: `SET ROLE` and session # advisory locks (both used by the EQL installer) are unstable over PgBouncer in # transaction mode. @@ -26,7 +32,7 @@ services: PGUSER: cipherstash POSTGRES_PASSWORD: password ports: - - "55432:5432" + - "${CS_PG_PORTS:-55432:5432}" # No volume: every run starts from a clean database, so a stale EQL # generation can never make a suite pass for the wrong reason. # diff --git a/local/docker-compose.supabase.yml b/local/docker-compose.supabase.yml index 4931db194..9274106a1 100644 --- a/local/docker-compose.supabase.yml +++ b/local/docker-compose.supabase.yml @@ -17,6 +17,12 @@ # Postgres bound to 5432/3000. Binding those here would either fail to start or, # worse, silently shadow the database they think they are talking to. # +# `CS_PG_PORTS` / `CS_PGRST_PORTS` override the mappings so CI can publish to +# EPHEMERAL host ports (container port only, e.g. `CS_PG_PORTS=5432`) and read the +# assigned ports back with `docker compose port`. That is what lets concurrent +# integration jobs share a runner without contending for fixed ports. Unset — the +# local default — keeps the stable `55433`/`55430` a developer can point at. +# # Connect DIRECT to the database, never through a pooler: `SET ROLE` and session # advisory locks (both used by the EQL installer) are unstable over PgBouncer in # transaction mode. @@ -28,7 +34,7 @@ services: environment: POSTGRES_PASSWORD: password ports: - - "55433:5432" + - "${CS_PG_PORTS:-55433:5432}" volumes: # Supplies the one thing the image omits: a password for `authenticator`. # @@ -74,7 +80,7 @@ services: # after boot becomes visible without a restart. PGRST_DB_CHANNEL_ENABLED: "true" ports: - - "55430:3000" + - "${CS_PGRST_PORTS:-55430:3000}" # No healthcheck: the postgrest image is distroless — it has no shell, no # wget, no curl, so every in-container probe fails with "executable file not # found" and `docker compose up --wait` reports the service unhealthy while From dccc32217eb30001970bd42e56bd28879fc56d72 Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Tue, 21 Jul 2026 13:21:00 +1000 Subject: [PATCH 2/2] fix(ci): validate every port readback, not just Postgres CodeRabbit caught a real asymmetry in the new action: `pg_port` was checked for emptiness but the supabase branch built `pgrest_url` straight from `$(port_of postgrest 3000)`. `docker compose port` exits 0 with EMPTY stdout when a port is not bound, so an unbound PostgREST would have silently yielded `http://localhost:` and handed every `PGRST_URL` consumer (the Drizzle-supabase leg, both Supabase-workflow steps) a malformed URL instead of the immediate error the Postgres path already gave. Move the check into `port_of` itself so it applies by construction and the asymmetry cannot come back, rather than adding a second call-site guard. Each port is now captured in its own bare assignment so `set -e` fails the step unambiguously on a non-zero `port_of`. Verified with a stubbed `docker` that returns a port for `db` and nothing for `postgrest`: the Postgres read succeeds, the guard fires for postgrest, and the script exits 1 before exporting anything. --- .github/actions/integration-db/action.yml | 32 +++++++++++++++-------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/.github/actions/integration-db/action.yml b/.github/actions/integration-db/action.yml index 7d0963933..726c6a95b 100644 --- a/.github/actions/integration-db/action.yml +++ b/.github/actions/integration-db/action.yml @@ -69,11 +69,28 @@ runs: # `docker compose port` prints `0.0.0.0:49153`, and may print one line # per address family — take the first, then the port after the last # colon (IPv6 forms contain several). + # + # The emptiness check lives HERE, not at the call sites: `docker compose + # port` exits 0 with empty stdout when a port is not bound, so "" is the + # failure signal and every caller needs the same guard. Checking once, by + # construction, is what stops the asymmetry this function had in review — + # Postgres was validated and PostgREST was not, so an unbound PostgREST + # would have silently produced `http://localhost:` and handed every + # consumer a malformed URL instead of an immediate, clear error. port_of() { - docker compose -p "$project" -f "$file" port "$1" "$2" \ - | head -n 1 | sed 's/.*://' | tr -d '[:space:]' + local svc="$1" cport="$2" hport + hport="$(docker compose -p "$project" -f "$file" port "$svc" "$cport" \ + | head -n 1 | sed 's/.*://' | tr -d '[:space:]')" + if [ -z "$hport" ]; then + echo "::error::could not read the published host port for service '$svc' (container port $cport) in the '$DB_VARIANT' stack" >&2 + docker compose -p "$project" -f "$file" ps >&2 + return 1 + fi + printf '%s' "$hport" } + # Each port is captured in its own bare assignment so `set -e` fails the + # step on a non-zero `port_of` unambiguously. case "$DB_VARIANT" in postgres) pg_port="$(port_of postgres 5432)" @@ -82,22 +99,15 @@ runs: ;; supabase) pg_port="$(port_of db 5432)" + pgrest_port="$(port_of postgrest 3000)" database_url="postgres://postgres:password@localhost:${pg_port}/postgres" - pgrest_url="http://localhost:$(port_of postgrest 3000)" + pgrest_url="http://localhost:${pgrest_port}" ;; *) echo "::error::unknown db variant '$DB_VARIANT'" && exit 1 ;; esac - # Fail loudly here rather than letting a suite connect to port "" and - # report an unrelated error. - if [ -z "${pg_port:-}" ]; then - echo "::error::could not read the published Postgres port for '$DB_VARIANT'" - docker compose -p "$project" -f "$file" ps - exit 1 - fi - echo "database-url=$database_url" >> "$GITHUB_OUTPUT" echo "pgrest-url=$pgrest_url" >> "$GITHUB_OUTPUT" echo "Started '$project' — postgres on :$pg_port${pgrest_url:+, postgrest on ${pgrest_url##*:}}"