Skip to content

test(sqlx): eql_v3 SQL-function property tests (CIP-3141) #293

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tobyhede wants to merge 5 commits into
base: eql_v3
Choose a base branch
from
Open

test(sqlx): eql_v3 SQL-function property tests (CIP-3141) #293

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c55fa2b
test(sqlx): function-double oracles over committed fixtures (CIP-3141)
tobyhede Jun 19, 2026
6eebf1d
ci: gate e2e property suite + unconditional source doc validation (CI…
tobyhede Jun 19, 2026
34d3e9e
docs: property-test suite docs + changelog (CIP-3141)
tobyhede Jun 19, 2026
5bbd959
fix(test): extract ob ORE term as array in float cross-width e2e orac…
tobyhede Jun 19, 2026
9768032
fix(test): compare cross-width float ORE terms via the SQL ORE operat…
tobyhede Jun 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
165 changes: 134 additions & 31 deletions .github/workflows/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,11 +1,41 @@
# CI: `test-eql.yml`
# CI workflows

This directory holds every GitHub Actions workflow for EQL. This README is the
authoritative **inventory** (what workflows exist and when they fire) and
**coverage map** (which job runs which checks, and where each test suite
actually runs).

- [Workflow inventory](#workflow-inventory)
- [`test-eql.yml` — the merge gate](#test-eqlyml--the-merge-gate)
- [Coverage map: job → task → what it checks](#coverage-map-job--task--what-it-checks)
- [Where each test suite runs](#where-each-test-suite-runs)
- [Known gaps](#known-gaps)

---

## Workflow inventory

| Workflow | Triggers | What it does | Gates merge? |
|---|---|---|---|
| **test-eql.yml** | `pull_request`, `merge_group`, `workflow_dispatch` | Full test/lint/validate matrix; the one required check | **Yes** — `ci-required` |
| **release-eql.yml** | `release: published`, `pull_request` (paths), `workflow_dispatch` | Build release SQL + docs; PR runs everything **but** the publish step | No |
| **release-postgres-eql-image.yml** | `release: published`, `workflow_dispatch` | Build & push the Postgres+EQL Docker image to GHCR | No |
| **bench-eql.yml** | `push: main` (paths), `schedule` 02:00 UTC daily, `workflow_dispatch` | `test:bench` (bench cargo feature). **Never runs on PRs** | No |
| **macro-expand-eql.yml** | `schedule` 03:00 UTC daily, `workflow_dispatch` | Regenerate the int4 `cargo expand` matrix snapshot; needs pinned nightly | **No — explicitly non-blocking** |
| **rebuild-docs.yml** | `push: tags` | Fire a docs-site rebuild webhook | N/A |

Only **test-eql.yml** gates merges. Bench regressions and stale `cargo expand`
snapshots surface on the nightly schedule, not on the PR that caused them.

---

## `test-eql.yml` — the merge gate

Fast PR feedback + a thorough pre-merge gate, using a merge queue and a single
aggregated required check.

## Two run shapes
### Two run shapes

`test-eql.yml` triggers on `pull_request`, `merge_group`, and `workflow_dispatch`.
`setup` derives the matrix from the event:

| Event | Trigger | Matrix | Purpose |
Expand All @@ -16,47 +46,120 @@ aggregated required check.

The PR run is feedback only. The merge-queue run is the gate.

## Relevance skip applies to PRs only
> **No `push` trigger.** Under a required merge queue, push-to-main validation is
> redundant — the queue already validated the exact merge commit, and branch
> protection blocks direct pushes.

### Relevance skip applies to PRs only

Each job runs when:
Each heavy job runs when:
`merge_group || workflow_dispatch || (pull_request && relevant == 'true')`.

So the `changes` relevance filter (`relevant:` paths) **only gates the
`pull_request` event** — a docs-only PR skips the heavy jobs on its PR run. On
`merge_group` (and `workflow_dispatch`) every job runs **unconditionally**: a
queued PR always pays the full gate regardless of which files it touched.
So the `changes` relevance filter only gates the **`pull_request`** event — a
docs-only PR skips the heavy jobs on its PR run. On `merge_group` (and
`workflow_dispatch`) every job runs **unconditionally**: a queued PR always pays
the full gate regardless of which files it touched.

## How the queue works
The relevance filter (`changes` job) marks a PR relevant when any of these
changed: `.github/workflows/test-eql.yml`, `src/**`, `sql/**`, `tests/**`,
`tasks/**`, `crates/**`, `Cargo.toml`, `Cargo.lock`, `mise.toml`. Note `docs/**`
is **not** in the filter — see [Known gaps](#known-gaps).

### How the queue works

1. Click **Merge when ready** — the PR is queued, not merged.
2. GitHub builds a temporary branch = `main` + this PR (+ any PRs ahead in the
queue) and fires `merge_group`, so CI tests the **post-merge state**, not the
stale PR branch.
queue) and fires `merge_group`, so CI tests the **post-merge state**.
3. The full PG14–17 × 2 matrix (plus the single-run jobs) runs and feeds
`ci-required`.
4. `ci-required` green → the PR is **merged into `main` using the queue's
configured merge method**. Red → the PR is **removed from the queue**; `main`
is untouched.
4. `ci-required` green → PR is **merged**. Red → PR is **removed from the
queue**; `main` is untouched.

This catches semantic conflicts — two PRs that each pass alone but break
together — which PR-only checks never test.

## The `ci-required` aggregator
### The `ci-required` aggregator

Per-event matrices make leaf job names unstable (a `test` job is displayed as
`Shard PG17 1/4`, but the queue produces `Shard PG14 1/2` … `Shard PG17 2/2`),
so leaf names can't be named as required checks. Instead, one aggregator job
(id and display name `ci-required`) `needs:` every job, runs with
`if: always()`, and passes only if each needed result is `success` **or**
`skipped`. Mark **only `ci-required`** as the required status check.
Per-event matrices make leaf job names unstable (`Shard PG17 1/4` on a PR vs.
`Shard PG14 1/2` in the queue), so leaf names can't be named as required checks.
Instead, one aggregator job (id and display name `ci-required`) `needs:` every
job, runs with `if: always()`, and passes only if each needed result is
`success` **or** `skipped`. Mark **only `ci-required`** as the required status
check.

- `if: always()` — runs even when dependencies fail/skip, so the check always
reports (a never-reported required check leaves the queue stuck *Pending*).
- `skipped` counts as pass — a docs-only PR skips the heavy jobs on its PR run
but must still report Success so the PR stays eligible to queue.

This is the well-known "aggregate / final gate job" pattern for matrix +
merge-queue workflows.
---

## Coverage map: job → task → what it checks

All jobs run on `blacksmith-16vcpu-ubuntu-2204`. "PG set" follows the event
(PG17 on PR / dispatch, PG14–17 in the queue).

| Job | mise task(s) | Checks | DB | `CS_*` creds |
|---|---|---|---|---|
| **changes** | — | compute relevance | no | no |
| **setup** | — | compute PG × shard matrix | no | no |
| **build-archive** | `test:sqlx:archive` | Build EQL, run prep, **generate fixtures**, compile every `tests/sqlx` binary (**default features**) into a nextest archive; upload archive + `release/*.sql` | yes (PG17) | **yes (sole holder)** |
| **test** (sharded) | `test:sqlx:partition` | Run the archived sqlx binaries (default features), hash-partitioned across shards | yes (per PG) | no (replays archive) |
| **e2e** | `test:sqlx:e2e` | The `proptest-e2e` fresh-encryption property suite (`e2e_oracle`) — PG17 only, version-independent | yes (PG17) | **yes** |
| **validate** (per PG) | `docs:validate:documented-sql` + `test:clean_install_v3` | DB-backed SQL doc-syntax check; clean-DB `eql_v3` install smoke | yes | no |
| **docs-static** | `docs:validate:source` | SQL doxygen coverage + required-tags (DB-free); **unconditional — runs on every PR incl. docs-only** | no | no |
| **schema** | `test:schema` | v2.2 / v2.3 payload JSON-schema validation | no | no |
| **rust-crates** | `test:crates` + `types:check` | `cargo fmt --check`, clippy + `cargo test` for `eql-scalars` / `eql-codegen` / `eql-tests-macros` / `eql-types`; verify TS bindings + JSON schemas are fresh | no | no |
| **codegen** | `codegen:parity` | Generated encrypted-domain SQL matches the golden output | no | no |
| **self-contained-v3** | `test:self_contained_v3` | `eql_v3` surface has no `eql_v2` dependency | no | no |
| **matrix-coverage** | `test:matrix:inventory` (+`:jsonb_entry`, `:v3-jsonb`) + `test:matrix:catalog-coverage` | Scalar-matrix test-name snapshots are not silently dropped; catalog surface is covered | no | no |
| **splinter** | `test:splinter` | Supabase/Splinter lints over the installed EQL | yes (PG17) | no |
| **ci-required** | — | aggregator: every needed job is `success`/`skipped` | no | no |

---

## Where each test suite runs

The `eql_v3` property-test suites (see
`tests/sqlx/tests/encrypted_domain/property/README.md`) land in three different
CI jobs:

| Suite | Job | Trigger coverage | DB | `CS_*` | Notes |
|---|---|---|---|---|---|
| **catalog** (`eql-scalars` `proptest_invariants`) | **rust-crates** (`cargo test -p eql-scalars`; proptest is a dev-dep) | relevant PR + queue | no | no | pure-Rust catalog invariants; shrinking enabled |
| **fixture** (function-double oracles, extractor identity, `match_smoke`, `edge_cases`) | **test** shards (default features) | relevant PR (PG17×4) + queue (PG14–17×2) | yes | no | oracle over the **committed** real-ciphertext fixtures |
| **e2e** (`e2e_oracle`, `#[cfg(feature = "proptest-e2e")]`) | **e2e** job (`test:sqlx:e2e`) | relevant PR (PG17) + queue (PG17) | yes | yes | oracle over **fresh** ZeroKMS encryption; PG-version-independent, so one PG17 run |

The wider sqlx suite (everything under `tests/sqlx/tests/`) runs in the **test**
shards, which replay the default-feature archive — so any
`#[cfg(feature = …)]`-gated test that isn't in the default feature set does not
run there. The `proptest-e2e` suite is the one such gate, and it has its own
**e2e** job (it can't reuse the credential-free archive: it both compiles with a
non-default feature and needs `CS_*` at run time).

---

## Known gaps

1. **bench + macro-expand are nightly / non-blocking** — a bench regression or a
stale `cargo expand` snapshot surfaces on the daily schedule, not on the PR
that introduced it. Accepted trade-off.

2. **`docs/**` markdown is not content-validated.** The `docs-static` job
guarantees the SQL `--!` doxygen comments under `src/**` are always checked,
but nothing lints the prose/links in `docs/**` itself. A docs-only PR now runs
`docs-static` (so it is no longer un-gated), but that job validates *source*
documentation, not the markdown the PR changed. Adding a markdown
linter/link-checker is a separate, unfilled capability.

### Recently closed

- *The e2e (fresh-encryption) suite never ran in CI.* Now covered by the **e2e**
job (`test:sqlx:e2e`), PG17, on relevant PRs + the queue.
- *Docs-only PRs ran no doc validation.* The **docs-static** job now runs the
source-only doc checks unconditionally on every PR.

---

## Operator setup (one-time, GitHub UI)

Expand All @@ -68,13 +171,13 @@ Settings → Branches → rule for `main`:

Then verify (see `docs/plans/2026-06-09-ci-pr-feedback-sharding-rollout.md`):

- **Queue a relevant PR** → `merge_group` runs the full gate — 8 `Shard …` jobs
+ 4 `Validate …` jobs + `build-archive`, `schema`, `rust-crates`, `codegen`,
`self-contained-v3`, `matrix-coverage`, `splinter` — all green → `ci-required`
green → PR merges.
- **Open a docs-only PR** → on its `pull_request` run the heavy jobs skip and
`ci-required` reports **Success** (not stuck *Pending*), so the PR can be
queued.
- **Queue a relevant PR** → `merge_group` runs the full gate (8 `Shard …` jobs +
4 `Validate …` jobs + `build-archive`, `e2e`, `docs-static`, `schema`,
`rust-crates`, `codegen`, `self-contained-v3`, `matrix-coverage`, `splinter`) →
`ci-required` green → PR merges.
- **Open a docs-only PR** → on its `pull_request` run the relevance-gated heavy
jobs skip, but `docs-static` still runs; `ci-required` reports **Success** (not
stuck *Pending*), so the PR can be queued.

## References

Expand Down
79 changes: 76 additions & 3 deletions .github/workflows/test-eql.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,9 +268,13 @@ jobs:
run: |
mise run postgres:up postgres-${POSTGRES_VERSION} --extra-args "--detach --wait"

- name: Validate SQL documentation (Postgres ${{ matrix.postgres-version }})
# Source-only doc checks (coverage + required-tags) moved to the
# unconditional `docs-static` job so they run on every PR (incl. docs-only)
# and exactly once, not per-Postgres. This step keeps only the DB-backed
# SQL-syntax validation, which genuinely needs the per-version Postgres.
- name: Validate documented SQL syntax (Postgres ${{ matrix.postgres-version }})
run: |
mise run docs:validate
mise run docs:validate:documented-sql

- name: Clean-DB v3 install smoke (Postgres ${{ matrix.postgres-version }})
run: |
Expand Down Expand Up @@ -460,6 +464,75 @@ jobs:
run: |
mise run --output prefix test:splinter --postgres ${POSTGRES_VERSION}

# Source-only SQL documentation validation (coverage + required Doxygen tags).
# Deliberately NOT relevance-gated: it runs on EVERY pull_request — including
# docs-only PRs that skip the heavy jobs — so documentation is always
# validated. DB-free and creds-free (the psql-backed syntax check stays in the
# per-version `validate` job).
docs-static:
name: "SQL doc validation"
runs-on: blacksmith-16vcpu-ubuntu-2204
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
- uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4
with:
version: 2026.4.0
install: true
cache: true
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
workspaces: .
shared-key: sqlx-tests
save-if: false
- name: Validate SQL doc coverage + required tags
run: |
mise run docs:validate:source

# The e2e (fresh-encryption) property suite. Encrypts random values through
# ZeroKMS at run time, so it needs CS_* creds and is PG-version-independent —
# one PG17 run, never the matrix. Compiles the `proptest-e2e`-gated binaries
# (which the default-feature sharded archive excludes) and runs only the
# e2e oracle. Like build-archive, it holds CS_* and so carries the same
# fork-PR guard to keep the secrets off fork runs.
e2e:
name: "e2e property suite (fresh encryption)"
needs: [changes, setup]
if: >-
(github.event_name == 'merge_group'
|| github.event_name == 'workflow_dispatch'
|| (github.event_name == 'pull_request' && needs.changes.outputs.relevant == 'true'))
&& (github.event_name != 'pull_request'
|| github.event.pull_request.head.repo.full_name == github.repository)
runs-on: blacksmith-16vcpu-ubuntu-2204
env:
POSTGRES_VERSION: "17"
CS_CLIENT_ACCESS_KEY: ${{ secrets.CS_CLIENT_ACCESS_KEY }}
CS_WORKSPACE_CRN: ${{ secrets.CS_WORKSPACE_CRN }}
CS_CLIENT_ID: ${{ secrets.CS_CLIENT_ID }}
CS_CLIENT_KEY: ${{ secrets.CS_CLIENT_KEY }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
- uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4
with:
version: 2026.4.0
install: true
cache: true
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
with:
workspaces: .
shared-key: sqlx-tests
save-if: false
- name: Setup database (Postgres 17)
run: |
mise run postgres:up postgres-${POSTGRES_VERSION} --extra-args "--detach --wait"
- name: Run e2e property suite
run: |
mise run test:sqlx:e2e

# The ONE required status check. Stable name on every event, so branch
# protection never references an event-dependent leaf name (which would
# deadlock). Passes iff every needed job is success or skipped. Treating
Expand All @@ -469,7 +542,7 @@ jobs:
ci-required:
name: "ci-required"
needs: [changes, setup, build-archive, test, validate, schema, rust-crates,
codegen, self-contained-v3, matrix-coverage, splinter]
codegen, self-contained-v3, matrix-coverage, splinter, docs-static, e2e]
if: always()
runs-on: blacksmith-16vcpu-ubuntu-2204
steps:
Expand Down
Loading
Loading