Skip to content

Commit 07dd0f6

Browse files
committed
docs(v3): document property-test suite structure (catalog/fixture/e2e)
Add a README for the eql_v3 property tests describing the three suites, the shared all-pairs oracle engine, why ciphertext can't be Arbitrary-derived, and the conventions/footguns (not under scalars::, e2e gated behind proptest-e2e, shrinking disabled for e2e). Reference it from CLAUDE.md's Testing section.
1 parent 095ab03 commit 07dd0f6

2 files changed

Lines changed: 110 additions & 0 deletions

File tree

CLAUDE.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,7 @@ This project uses `mise` for task management. Common commands:
2929
- Run SQLx tests directly: `mise run test:sqlx`
3030
- Run SQLx tests in watch mode: `mise run test:sqlx:watch`
3131
- Tests are located in `tests/sqlx/` using Rust and SQLx framework
32+
- Property-based tests for the `eql_v3` encrypted scalar domains live in three suites — **catalog** (pure-Rust catalog invariants, no DB), **fixture** (oracle over committed ciphertext), and **e2e** (oracle over fresh end-to-end encryption, gated behind the `proptest-e2e` cargo feature). The structure, the shared all-pairs oracle engine, and the conventions/footguns (e.g. why they must not live under `scalars::`) are documented in `tests/sqlx/tests/encrypted_domain/property/README.md`.
3233
- Verify the scalar matrix coverage snapshot: `mise run test:matrix:inventory` (no database required). ONE committed `tests/sqlx/snapshots/matrix_tests.txt` baseline pins the token-normalized set of `scalars::<T>::*` test names so a silently dropped/renamed/`#[cfg]`-gated test fails CI's `matrix-coverage` job. The task discovers the present scalar types from the test binary's `--list` and cross-checks them against `cargo run -p eql-codegen -- list-types`, so a catalog type missing its matrix wiring also fails. When you change which matrix tests the macro emits, regenerate and commit the single snapshot in the same change. See `tests/sqlx/snapshots/README.md`.
3334

3435
### Build System
Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# Property-based tests for the `eql_v3` encrypted scalar domains
2+
3+
These tests assert that SQL operator results on the `eql_v3` encrypted-domain
4+
types agree with a **plaintext oracle** across a *generated* input space — the
5+
fixed-pivot scalar matrix only checks hand-picked values, so property tests are
6+
what catch operator/oracle disagreements across the whole value space. Origin:
7+
CIP-3141.
8+
9+
## The three suites
10+
11+
The harness is **one unit-level suite plus two integration suites**. They are
12+
named for what they operate on, not by an abstract tier letter:
13+
14+
| Suite | Location | Kind | Inputs | DB / creds |
15+
|-------|----------|------|--------|------------|
16+
| **catalog** | [`crates/eql-scalars/src/proptest_invariants.rs`](../../../../../crates/eql-scalars/src/proptest_invariants.rs) | unit (pure Rust) | generated terms / kinds | none — runs in fork CI |
17+
| **fixture** | [`fixture_oracle.rs`](./fixture_oracle.rs) | integration | committed fixture corpus (real ciphertext) | shared test DB |
18+
| **e2e** | [`e2e_oracle.rs`](./e2e_oracle.rs) | integration | freshly generated plaintexts, encrypted each run | shared test DB **+ ZeroKMS creds** |
19+
20+
Plus [`edge_cases.rs`](./edge_cases.rs): example-based unit tests for
21+
NULL propagation, blockers raising on unsupported operators (including the
22+
native-`jsonb` `->`/`@>` domain-fallback paths), `timestamptz` ordering
23+
deferral, and CHECK rejection of malformed payloads.
24+
25+
### catalog — catalog invariants, no database
26+
27+
Pure-Rust `proptest` over the `eql-scalars` catalog: term/operator/extractor
28+
consistency, "every blocker is non-`STRICT` + `plpgsql`", payload-key set ==
29+
declared terms, integer-range ordering. No DB, no encryption, no creds, so it
30+
runs in the lean `cargo test -p eql-scalars` path (and on fork PRs). This is the
31+
only suite where `proptest` shrinking is meaningful and enabled.
32+
33+
### fixture — oracle over committed ciphertext
34+
35+
Runs the shared all-pairs oracle engine over the real, committed fixture corpus
36+
(`fixtures.eql_v2_<T>.sql`, generated by `cipherstash-client` during
37+
`mise run test:sqlx:prep`). `proptest` selects a sub-multiset of fixture rows
38+
(with repeats), and the engine checks **every ordered pair**. No new encryption,
39+
so it runs whenever the fixtures are present, and it generalises across the whole
40+
catalog for free (every fixtured type gets an `eq_oracle`, ordered types also get
41+
an `ord_oracle`).
42+
43+
### e2e — oracle over fresh end-to-end encryption
44+
45+
Same oracle engine, but each case **generates fresh random plaintexts and
46+
encrypts them end-to-end through ZeroKMS** (one batched call per case) before
47+
querying. Gated behind the `proptest-e2e` cargo feature — `mise run test:sqlx`
48+
enables it (CI has the secrets); a bare `cargo test` compiles it out. It is the
49+
**only** suite that can exercise "same plaintext, *different* ciphertext"
50+
(equality across independently-encrypted values), because the committed fixture
51+
corpus has no duplicate plaintexts. Integer scalars only for now (random `T`
52+
generation is trivial for integers).
53+
54+
## The shared oracle engine
55+
56+
`assert_eq_oracle` / `assert_ord_oracle` in
57+
[`../../../src/property.rs`](../../../src/property.rs) take a corpus of
58+
`(plaintext, payload_json)` rows and check, over every ordered pair, that:
59+
60+
- `=` / `<>` on the `_eq` domain agree with plaintext `==` / `!=`, and
61+
- (ordered domains) `<` `<=` `>` `>=` and `ord_term` sort order agree with the
62+
plaintext comparison.
63+
64+
The fixture and e2e suites differ only in **where the rows come from**; the
65+
engine is identical.
66+
67+
## Why ciphertext can't be `Arbitrary`-derived
68+
69+
A valid payload's `hm`/`ob` terms are real ciphertext from `cipherstash-client`
70+
(`encrypt_eql`), which needs a live ZeroKMS handshake — there is no offline/mock
71+
cipher. `proptest` can only generate **plaintexts**; turning them into payloads
72+
means encrypting them. That is exactly why there are two integration suites: the
73+
fixture suite reuses already-encrypted values, and the e2e suite pays for fresh
74+
encryption to reach inputs the fixtures can't.
75+
76+
## Conventions and footguns
77+
78+
- **Never put these tests under a `scalars::` module.** `mise run test:matrix:inventory`
79+
discovers scalar types from every `scalars::<X>::` test-name prefix, so a
80+
`scalars::property::…` test would be mis-read as a scalar type and break the
81+
catalog cross-check. They live under `property::` for this reason.
82+
- **The e2e suite is gated behind `proptest-e2e`.** Keep it that way so a
83+
credential-less `cargo test` (and the lean crate path) still compiles.
84+
- **The e2e suite disables shrinking and failure persistence.** Every shrink
85+
attempt would burn another ZeroKMS batch, ciphertext can't be meaningfully
86+
shrunk, and fresh-each-run ciphertext can't be replayed.
87+
- **proptest + async bridge.** The `proptest!` body is sync; the DB-backed suites
88+
run their async oracle on a per-case current-thread `tokio` runtime and connect
89+
via `connect_pool()` (they cannot use `#[sqlx::test]`'s injected pool from a
90+
sync body). Operator evaluation is read-only `SELECT`, so no per-test schema
91+
isolation is needed.
92+
- **Equality-true must actually fire.** Random integer pairs almost never
93+
collide, so the e2e corpus injects deliberate duplicate plaintexts (plus signed
94+
extremes and zero) to exercise the `a == b ⇒ eq` branch across distinct
95+
ciphertexts.
96+
97+
## Running
98+
99+
```bash
100+
# catalog suite only (no DB, no creds)
101+
cargo test -p eql-scalars proptest_invariants
102+
103+
# fixture + edge-case suites (needs a prepared DB)
104+
mise run test:sqlx:prep
105+
cd tests/sqlx && cargo test --test encrypted_domain property::fixture_oracle property::edge_cases
106+
107+
# all suites incl. e2e (needs DB + CS_* creds)
108+
mise run test:sqlx # enables --features proptest-e2e
109+
```

0 commit comments

Comments
 (0)