Skip to content

Commit 37335ed

Browse files
tobyhedetobyhede
authored
Merge pull request #293 from cipherstash/v3-property-tests-fn-doubles
test(sqlx): eql_v3 SQL-function property tests (CIP-3141)
2 parents c26d700 + 9768032 commit 37335ed

16 files changed

Lines changed: 1030 additions & 152 deletions

File tree

‎.github/workflows/README.md‎

Lines changed: 134 additions & 31 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,41 @@
1-
# CI: `test-eql.yml`
1+
# CI workflows
2+
3+
This directory holds every GitHub Actions workflow for EQL. This README is the
4+
authoritative **inventory** (what workflows exist and when they fire) and
5+
**coverage map** (which job runs which checks, and where each test suite
6+
actually runs).
7+
8+
- [Workflow inventory](#workflow-inventory)
9+
- [`test-eql.yml` — the merge gate](#test-eqlyml--the-merge-gate)
10+
- [Coverage map: job → task → what it checks](#coverage-map-job--task--what-it-checks)
11+
- [Where each test suite runs](#where-each-test-suite-runs)
12+
- [Known gaps](#known-gaps)
13+
14+
---
15+
16+
## Workflow inventory
17+
18+
| Workflow | Triggers | What it does | Gates merge? |
19+
|---|---|---|---|
20+
| **test-eql.yml** | `pull_request`, `merge_group`, `workflow_dispatch` | Full test/lint/validate matrix; the one required check | **Yes** — `ci-required` |
21+
| **release-eql.yml** | `release: published`, `pull_request` (paths), `workflow_dispatch` | Build release SQL + docs; PR runs everything **but** the publish step | No |
22+
| **release-postgres-eql-image.yml** | `release: published`, `workflow_dispatch` | Build & push the Postgres+EQL Docker image to GHCR | No |
23+
| **bench-eql.yml** | `push: main` (paths), `schedule` 02:00 UTC daily, `workflow_dispatch` | `test:bench` (bench cargo feature). **Never runs on PRs** | No |
24+
| **macro-expand-eql.yml** | `schedule` 03:00 UTC daily, `workflow_dispatch` | Regenerate the int4 `cargo expand` matrix snapshot; needs pinned nightly | **No — explicitly non-blocking** |
25+
| **rebuild-docs.yml** | `push: tags` | Fire a docs-site rebuild webhook | N/A |
26+
27+
Only **test-eql.yml** gates merges. Bench regressions and stale `cargo expand`
28+
snapshots surface on the nightly schedule, not on the PR that caused them.
29+
30+
---
31+
32+
## `test-eql.yml` — the merge gate
233

334
Fast PR feedback + a thorough pre-merge gate, using a merge queue and a single
435
aggregated required check.
536

6-
## Two run shapes
37+
### Two run shapes
738

8-
`test-eql.yml` triggers on `pull_request`, `merge_group`, and `workflow_dispatch`.
939
`setup` derives the matrix from the event:
1040

1141
| Event | Trigger | Matrix | Purpose |
@@ -16,47 +46,120 @@ aggregated required check.
1646

1747
The PR run is feedback only. The merge-queue run is the gate.
1848

19-
## Relevance skip applies to PRs only
49+
> **No `push` trigger.** Under a required merge queue, push-to-main validation is
50+
> redundant — the queue already validated the exact merge commit, and branch
51+
> protection blocks direct pushes.
52+
53+
### Relevance skip applies to PRs only
2054

21-
Each job runs when:
55+
Each heavy job runs when:
2256
`merge_group || workflow_dispatch || (pull_request && relevant == 'true')`.
2357

24-
So the `changes` relevance filter (`relevant:` paths) **only gates the
25-
`pull_request` event** — a docs-only PR skips the heavy jobs on its PR run. On
26-
`merge_group` (and `workflow_dispatch`) every job runs **unconditionally**: a
27-
queued PR always pays the full gate regardless of which files it touched.
58+
So the `changes` relevance filter only gates the **`pull_request`** event — a
59+
docs-only PR skips the heavy jobs on its PR run. On `merge_group` (and
60+
`workflow_dispatch`) every job runs **unconditionally**: a queued PR always pays
61+
the full gate regardless of which files it touched.
2862

29-
## How the queue works
63+
The relevance filter (`changes` job) marks a PR relevant when any of these
64+
changed: `.github/workflows/test-eql.yml`, `src/**`, `sql/**`, `tests/**`,
65+
`tasks/**`, `crates/**`, `Cargo.toml`, `Cargo.lock`, `mise.toml`. Note `docs/**`
66+
is **not** in the filter — see [Known gaps](#known-gaps).
67+
68+
### How the queue works
3069

3170
1. Click **Merge when ready** — the PR is queued, not merged.
3271
2. GitHub builds a temporary branch = `main` + this PR (+ any PRs ahead in the
33-
queue) and fires `merge_group`, so CI tests the **post-merge state**, not the
34-
stale PR branch.
72+
queue) and fires `merge_group`, so CI tests the **post-merge state**.
3573
3. The full PG14–17 × 2 matrix (plus the single-run jobs) runs and feeds
3674
`ci-required`.
37-
4. `ci-required` green → the PR is **merged into `main` using the queue's
38-
configured merge method**. Red → the PR is **removed from the queue**; `main`
39-
is untouched.
75+
4. `ci-required` green → PR is **merged**. Red → PR is **removed from the
76+
queue**; `main` is untouched.
4077

4178
This catches semantic conflicts — two PRs that each pass alone but break
4279
together — which PR-only checks never test.
4380

44-
## The `ci-required` aggregator
81+
### The `ci-required` aggregator
4582

46-
Per-event matrices make leaf job names unstable (a `test` job is displayed as
47-
`Shard PG17 1/4`, but the queue produces `Shard PG14 1/2` … `Shard PG17 2/2`),
48-
so leaf names can't be named as required checks. Instead, one aggregator job
49-
(id and display name `ci-required`) `needs:` every job, runs with
50-
`if: always()`, and passes only if each needed result is `success` **or**
51-
`skipped`. Mark **only `ci-required`** as the required status check.
83+
Per-event matrices make leaf job names unstable (`Shard PG17 1/4` on a PR vs.
84+
`Shard PG14 1/2` in the queue), so leaf names can't be named as required checks.
85+
Instead, one aggregator job (id and display name `ci-required`) `needs:` every
86+
job, runs with `if: always()`, and passes only if each needed result is
87+
`success` **or** `skipped`. Mark **only `ci-required`** as the required status
88+
check.
5289

5390
- `if: always()` — runs even when dependencies fail/skip, so the check always
5491
reports (a never-reported required check leaves the queue stuck *Pending*).
5592
- `skipped` counts as pass — a docs-only PR skips the heavy jobs on its PR run
5693
but must still report Success so the PR stays eligible to queue.
5794

58-
This is the well-known "aggregate / final gate job" pattern for matrix +
59-
merge-queue workflows.
95+
---
96+
97+
## Coverage map: job → task → what it checks
98+
99+
All jobs run on `blacksmith-16vcpu-ubuntu-2204`. "PG set" follows the event
100+
(PG17 on PR / dispatch, PG14–17 in the queue).
101+
102+
| Job | mise task(s) | Checks | DB | `CS_*` creds |
103+
|---|---|---|---|---|
104+
| **changes** | — | compute relevance | no | no |
105+
| **setup** | — | compute PG × shard matrix | no | no |
106+
| **build-archive** | `test:sqlx:archive` | Build EQL, run prep, **generate fixtures**, compile every `tests/sqlx` binary (**default features**) into a nextest archive; upload archive + `release/*.sql` | yes (PG17) | **yes (sole holder)** |
107+
| **test** (sharded) | `test:sqlx:partition` | Run the archived sqlx binaries (default features), hash-partitioned across shards | yes (per PG) | no (replays archive) |
108+
| **e2e** | `test:sqlx:e2e` | The `proptest-e2e` fresh-encryption property suite (`e2e_oracle`) — PG17 only, version-independent | yes (PG17) | **yes** |
109+
| **validate** (per PG) | `docs:validate:documented-sql` + `test:clean_install_v3` | DB-backed SQL doc-syntax check; clean-DB `eql_v3` install smoke | yes | no |
110+
| **docs-static** | `docs:validate:source` | SQL doxygen coverage + required-tags (DB-free); **unconditional — runs on every PR incl. docs-only** | no | no |
111+
| **schema** | `test:schema` | v2.2 / v2.3 payload JSON-schema validation | no | no |
112+
| **rust-crates** | `test:crates` + `types:check` | `cargo fmt --check`, clippy + `cargo test` for `eql-scalars` / `eql-codegen` / `eql-tests-macros` / `eql-types`; verify TS bindings + JSON schemas are fresh | no | no |
113+
| **codegen** | `codegen:parity` | Generated encrypted-domain SQL matches the golden output | no | no |
114+
| **self-contained-v3** | `test:self_contained_v3` | `eql_v3` surface has no `eql_v2` dependency | no | no |
115+
| **matrix-coverage** | `test:matrix:inventory` (+`:jsonb_entry`, `:v3-jsonb`) + `test:matrix:catalog-coverage` | Scalar-matrix test-name snapshots are not silently dropped; catalog surface is covered | no | no |
116+
| **splinter** | `test:splinter` | Supabase/Splinter lints over the installed EQL | yes (PG17) | no |
117+
| **ci-required** | — | aggregator: every needed job is `success`/`skipped` | no | no |
118+
119+
---
120+
121+
## Where each test suite runs
122+
123+
The `eql_v3` property-test suites (see
124+
`tests/sqlx/tests/encrypted_domain/property/README.md`) land in three different
125+
CI jobs:
126+
127+
| Suite | Job | Trigger coverage | DB | `CS_*` | Notes |
128+
|---|---|---|---|---|---|
129+
| **catalog** (`eql-scalars` `proptest_invariants`) | **rust-crates** (`cargo test -p eql-scalars`; proptest is a dev-dep) | relevant PR + queue | no | no | pure-Rust catalog invariants; shrinking enabled |
130+
| **fixture** (function-double oracles, extractor identity, `match_smoke`, `edge_cases`) | **test** shards (default features) | relevant PR (PG17×4) + queue (PG14–17×2) | yes | no | oracle over the **committed** real-ciphertext fixtures |
131+
| **e2e** (`e2e_oracle`, `#[cfg(feature = "proptest-e2e")]`) | **e2e** job (`test:sqlx:e2e`) | relevant PR (PG17) + queue (PG17) | yes | yes | oracle over **fresh** ZeroKMS encryption; PG-version-independent, so one PG17 run |
132+
133+
The wider sqlx suite (everything under `tests/sqlx/tests/`) runs in the **test**
134+
shards, which replay the default-feature archive — so any
135+
`#[cfg(feature = …)]`-gated test that isn't in the default feature set does not
136+
run there. The `proptest-e2e` suite is the one such gate, and it has its own
137+
**e2e** job (it can't reuse the credential-free archive: it both compiles with a
138+
non-default feature and needs `CS_*` at run time).
139+
140+
---
141+
142+
## Known gaps
143+
144+
1. **bench + macro-expand are nightly / non-blocking** — a bench regression or a
145+
stale `cargo expand` snapshot surfaces on the daily schedule, not on the PR
146+
that introduced it. Accepted trade-off.
147+
148+
2. **`docs/**` markdown is not content-validated.** The `docs-static` job
149+
guarantees the SQL `--!` doxygen comments under `src/**` are always checked,
150+
but nothing lints the prose/links in `docs/**` itself. A docs-only PR now runs
151+
`docs-static` (so it is no longer un-gated), but that job validates *source*
152+
documentation, not the markdown the PR changed. Adding a markdown
153+
linter/link-checker is a separate, unfilled capability.
154+
155+
### Recently closed
156+
157+
- *The e2e (fresh-encryption) suite never ran in CI.* Now covered by the **e2e**
158+
job (`test:sqlx:e2e`), PG17, on relevant PRs + the queue.
159+
- *Docs-only PRs ran no doc validation.* The **docs-static** job now runs the
160+
source-only doc checks unconditionally on every PR.
161+
162+
---
60163

61164
## Operator setup (one-time, GitHub UI)
62165

@@ -68,13 +171,13 @@ Settings → Branches → rule for `main`:
68171

69172
Then verify (see `docs/plans/2026-06-09-ci-pr-feedback-sharding-rollout.md`):
70173

71-
- **Queue a relevant PR** → `merge_group` runs the full gate — 8 `Shard …` jobs
72-
+ 4 `Validate …` jobs + `build-archive`, `schema`, `rust-crates`, `codegen`,
73-
`self-contained-v3`, `matrix-coverage`, `splinter` — all green → `ci-required`
74-
green → PR merges.
75-
- **Open a docs-only PR** → on its `pull_request` run the heavy jobs skip and
76-
`ci-required` reports **Success** (not stuck *Pending*), so the PR can be
77-
queued.
174+
- **Queue a relevant PR** → `merge_group` runs the full gate (8 `Shard …` jobs +
175+
4 `Validate …` jobs + `build-archive`, `e2e`, `docs-static`, `schema`,
176+
`rust-crates`, `codegen`, `self-contained-v3`, `matrix-coverage`, `splinter`) →
177+
`ci-required` green → PR merges.
178+
- **Open a docs-only PR** → on its `pull_request` run the relevance-gated heavy
179+
jobs skip, but `docs-static` still runs; `ci-required` reports **Success** (not
180+
stuck *Pending*), so the PR can be queued.
78181

79182
## References
80183

‎.github/workflows/test-eql.yml‎

Lines changed: 76 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -268,9 +268,13 @@ jobs:
268268
run: |
269269
mise run postgres:up postgres-${POSTGRES_VERSION} --extra-args "--detach --wait"
270270
271-
- name: Validate SQL documentation (Postgres ${{ matrix.postgres-version }})
271+
# Source-only doc checks (coverage + required-tags) moved to the
272+
# unconditional `docs-static` job so they run on every PR (incl. docs-only)
273+
# and exactly once, not per-Postgres. This step keeps only the DB-backed
274+
# SQL-syntax validation, which genuinely needs the per-version Postgres.
275+
- name: Validate documented SQL syntax (Postgres ${{ matrix.postgres-version }})
272276
run: |
273-
mise run docs:validate
277+
mise run docs:validate:documented-sql
274278
275279
- name: Clean-DB v3 install smoke (Postgres ${{ matrix.postgres-version }})
276280
run: |
@@ -460,6 +464,75 @@ jobs:
460464
run: |
461465
mise run --output prefix test:splinter --postgres ${POSTGRES_VERSION}
462466
467+
# Source-only SQL documentation validation (coverage + required Doxygen tags).
468+
# Deliberately NOT relevance-gated: it runs on EVERY pull_request — including
469+
# docs-only PRs that skip the heavy jobs — so documentation is always
470+
# validated. DB-free and creds-free (the psql-backed syntax check stays in the
471+
# per-version `validate` job).
472+
docs-static:
473+
name: "SQL doc validation"
474+
runs-on: blacksmith-16vcpu-ubuntu-2204
475+
steps:
476+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
477+
with:
478+
persist-credentials: false
479+
- uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4
480+
with:
481+
version: 2026.4.0
482+
install: true
483+
cache: true
484+
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
485+
with:
486+
workspaces: .
487+
shared-key: sqlx-tests
488+
save-if: false
489+
- name: Validate SQL doc coverage + required tags
490+
run: |
491+
mise run docs:validate:source
492+
493+
# The e2e (fresh-encryption) property suite. Encrypts random values through
494+
# ZeroKMS at run time, so it needs CS_* creds and is PG-version-independent —
495+
# one PG17 run, never the matrix. Compiles the `proptest-e2e`-gated binaries
496+
# (which the default-feature sharded archive excludes) and runs only the
497+
# e2e oracle. Like build-archive, it holds CS_* and so carries the same
498+
# fork-PR guard to keep the secrets off fork runs.
499+
e2e:
500+
name: "e2e property suite (fresh encryption)"
501+
needs: [changes, setup]
502+
if: >-
503+
(github.event_name == 'merge_group'
504+
|| github.event_name == 'workflow_dispatch'
505+
|| (github.event_name == 'pull_request' && needs.changes.outputs.relevant == 'true'))
506+
&& (github.event_name != 'pull_request'
507+
|| github.event.pull_request.head.repo.full_name == github.repository)
508+
runs-on: blacksmith-16vcpu-ubuntu-2204
509+
env:
510+
POSTGRES_VERSION: "17"
511+
CS_CLIENT_ACCESS_KEY: ${{ secrets.CS_CLIENT_ACCESS_KEY }}
512+
CS_WORKSPACE_CRN: ${{ secrets.CS_WORKSPACE_CRN }}
513+
CS_CLIENT_ID: ${{ secrets.CS_CLIENT_ID }}
514+
CS_CLIENT_KEY: ${{ secrets.CS_CLIENT_KEY }}
515+
steps:
516+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
517+
with:
518+
persist-credentials: false
519+
- uses: jdx/mise-action@1648a7812b9aeae629881980618f079932869151 # v4
520+
with:
521+
version: 2026.4.0
522+
install: true
523+
cache: true
524+
- uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
525+
with:
526+
workspaces: .
527+
shared-key: sqlx-tests
528+
save-if: false
529+
- name: Setup database (Postgres 17)
530+
run: |
531+
mise run postgres:up postgres-${POSTGRES_VERSION} --extra-args "--detach --wait"
532+
- name: Run e2e property suite
533+
run: |
534+
mise run test:sqlx:e2e
535+
463536
# The ONE required status check. Stable name on every event, so branch
464537
# protection never references an event-dependent leaf name (which would
465538
# deadlock). Passes iff every needed job is success or skipped. Treating
@@ -469,7 +542,7 @@ jobs:
469542
ci-required:
470543
name: "ci-required"
471544
needs: [changes, setup, build-archive, test, validate, schema, rust-crates,
472-
codegen, self-contained-v3, matrix-coverage, splinter]
545+
codegen, self-contained-v3, matrix-coverage, splinter, docs-static, e2e]
473546
if: always()
474547
runs-on: blacksmith-16vcpu-ubuntu-2204
475548
steps:

0 commit comments

Comments
 (0)