docs(v3): code generator + fixture generator walkthroughs (+ eql-types numeric/timestamptz) by tobyhede · Pull Request #283 · cipherstash/encrypt-query-language

tobyhede · 2026-06-16T04:50:46Z

Summary

Adds developer-facing walkthroughs for the v3 code-generation toolchain, plus the supporting eql-types changes they document.

Stacked on #276 (eql-v3-ore-block-256). Review that stack first; this PR's own diff is the 5 commits below and will narrow as the stack lands.

Docs

docs/walkthroughs/sql-code-generator.md — end-to-end walkthrough of the eql-codegen Rust SQL generator (catalog → rendered eql_v3 scalar surface).
docs/walkthroughs/fixture-generator.md — walkthrough of the catalog-driven SQLx fixture generation pipeline (real ciphertexts via cipherstash-client).

eql-types

feat(eql-types): add numeric payload type and promote timestamptz to the ordered (ORE) domains.
refactor(eql-types): rename the OreBlock newtype U64_8_256 → 256, tracking the SEM-type rename in the underlying stack.
Doc-accuracy fixes: corrected stale v3_ste_vec "committed exception" framing in CLAUDE.md and the fixture walkthrough; fixed counts and line references flagged in review.

Review status

CodeRabbit reviewed clean — 0 findings across all distinct commits.

Test plan

mise run build (regenerates v3 scalar surface from the catalog)
mise run test
Docs render-check the two new walkthroughs

…repo-wide) Behaviour-preserving rename of the self-contained eql_v3 SEM index-term type to its width-agnostic name, ahead of the N-block ORE comparator work. The eql_v2 public API (eql_v2.ore_block_u64_8_256, src/ore_block_u64_8_256/, v2 operators/docs/tests) is deliberately UNCHANGED. Scope (eql_v3 only): - src/v3/sem/ore_block_u64_8_256/ -> src/v3/sem/ore_block_256/ (dir + symbols) - eql-scalars Term::Ore ctor + REQUIRE paths; eql-codegen doc/assertion - committed codegen goldens (int2/int4/int8/date/text incl text_search) - eql-types: OreBlockU64_8_256 newtype -> OreBlock256, SQL constructor doc - tasks/pin_search_path.sql (v3 block only), splinter.sh (v3 rows only), clean_install_v3.sh, drop_operator_classes.sql (v3 opclass only) - v3 SQLx family tests (sem/mutations/inlinability), preserving the eql_v2.compare_ore_block_u64_8_256_terms parity reference in sem.rs - v3 docs (CLAUDE, adding-a-scalar, eql-functions, sql-support, analysis), CHANGELOG eql_v3-qualified mentions Covers the new old-name references introduced by #280 (text-search). Verified: codegen:parity (byte-for-byte), test:crates (fmt+clippy+tests), clean build + v3 release artifact carries no eql_v2 symbol. The DB-backed clean_install_v3 gate runs in CI (Docker unavailable locally). Deliberately NOT renamed: tests/sqlx/src/index_types.rs ORE_BLOCK_U64_8_256 (crypto wire index-term identifier, not the SQL type name).

Derive the ORE block count N from term length instead of hardcoding 8, and wire the ordered numeric scalar while promoting timestamptz to ordered.

Direct comparator tests over generated fixtures: numeric terms are 702 bytes (N=14) and order across a full 14-value ascending chain spanning sign, magnitude, and fractional scale (so the left blocks decide ordering — the regression the missed 9 -> 1+n offset would fail); timestamptz terms are 604 bytes (N=12) and order 1900 < 2099. Verified end-to-end: numeric + timestamptz ordered matrix suites (211 each, incl. < <= > >= / ORDER BY / MIN / MAX) green; full SQLx suite 2026 passed; test:matrix:inventory reconciles both new ordered types; self-contained v3 install green.

…stamptz CHANGELOG: rewrite the timestamptz entry (ordering now ships), add the numeric ordered-domain entry, and add a Fixed entry for the N-block ORE comparator generalization + the eql_v3 ore_block_u64_8_256 -> ore_block_256 rename. Reference guide: document the fourth (numeric/Decimal) fixture discriminator — proc-macro routing, scalar_fixture arm, numeric_values accessor + distinctness guard, rust_decimal dep.

Complete the eql_v3.ore_block_u64_8_256 -> ore_block_256 rename across the docs (CLAUDE.md, the scalar guide, eql-functions, sql-support; eql_v2 names left unchanged) and the v3 SEM file header — the @file/subset comment had over-applied the rename to the v2-origin path (src/ore_block_u64_8_256). Regenerate the codegen reference goldens for every catalog type after rebasing onto eql_v3: add the numeric reference dir and expand timestamptz to its now-ordered shape so the parity gate passes. Also address review feedback: - ScalarKind::rust_type returns the now-real numeric type (rust_decimal::Decimal); only jsonb remains surfaceless. De-stale its doc and the 'timestamptz is equality-only' test comment; add numeric_maps_to_decimal. - Correct the guide's stale 'timestamptz is equality-only' prose and a dangling link to the removed design plan doc. - Add comparator_rejects_mismatched_block_widths (8-block vs 14-block terms must raise via the different-lengths guard). - Add the PR link (#276) to the numeric and N-block changelog entries.

has_ore_block_256 used "val ->> 'ob' IS NOT NULL", which stringifies a scalar/object 'ob' and reports it present. ore_block_256 then fed the malformed payload into jsonb_array_to_ore_block_256, which returns NULL instead of raising, silently degrading a structurally invalid ORE term into a NULL comparison/index term. Tighten the guard to require a JSON array (jsonb_typeof(val->'ob') = 'array'); a present-but-non-array 'ob' now RAISEs at the extractor boundary. '{}' (absent ob) and '{"ob": null}' (JSON null) remain absent (false). Adds T5 characterization cases for the scalar/object 'ob' presence checks and the extractor RAISE. Addresses CodeRabbit review thread on PR #276.

… fixture Strengthen the N-block ORE comparator tests so they run creds-free on no-creds CI shards, sourcing real ORE terms from committed fixtures: - assert_orders_like_oracle: all-pairs oracle agreement + antisymmetry (replaces the adjacent-pair-only ascending chain), with a row-count drift guard against the catalog fixture order - comparator_length_guard_sweep: boundary/off-by lengths for the 49*N+16 guard across N=1..14 - wide_block_term_compares_equal_to_itself: reflexive path at N=14/12 - numeric_scale_equivalents_collide: 1 == 1.0 ORE collision via the new hand-written v3_numeric_collision fixture (the value-equal pair the catalog distinctness guard forbids in eql_v2_numeric) Also fix the stale timestamptz comment in scalar_domains.rs (now ordered, not eq-only).

The eql-types crate landed on eql_v3 (PR #236) after this branch forked, so merging the base in surfaced a catalog_parity failure: CATALOG now has the numeric family and ordered timestamptz, but v3::all() didn't. - numeric.rs: four ordered domains (storage/_eq/_ord/_ord_ore), mirroring date.rs; numeric is the first scalar with a >8-block ORE term (14) - timestamptz.rs: add the two ordered domains; the eq-only/8-block-limit rationale is gone now that eql_v3.ore_block_256 derives N from term length - mod.rs: register the six new domains in all(), in CATALOG order - terms.rs: the ob term's SQL constructor is eql_v3.ore_block_256 (renamed this branch) and is width-agnostic (8/12/14 blocks) - v3_conformance.rs: cover numeric + timestamptz ord wire shapes; drop the stale equality-only claim - README: timestamptz no longer eq-only; add numeric

coderabbitai · 2026-06-16T04:50:55Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b2d15ec0-5890-45af-a8d4-ac8e1771acfd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch docs/generator-walkthroughs

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Add two self-contained walkthroughs under docs/walkthroughs/ covering the catalog-driven Rust generation systems: - sql-code-generator.md: how eql-codegen renders the eql_v3 encrypted-domain SQL surface (types/functions/operators/aggregates) from eql-scalars::CATALOG, the ScalarSpec/DomainSpec/Term data model, a worked int4 trace, the enforced invariants/footguns (blocker plpgsql/non-STRICT, inlinable SQL fns, no domain-over-domain, no opclass on a domain), and determinism/parity gating. - fixture-generator.md: how the generate_all_fixtures test materializes catalog fixture values (int_values!/temporal_values!) and encrypts them through cipherstash-client/ZeroKMS into the gitignored eql_v2_*.sql fixtures, the real-crypto hard requirement and credential gating, and a worked int4 trace. Both include Mermaid pipeline and data-model diagrams with path:line references.

…e refs)

…md + walkthroughs

coderdan · 2026-06-16T23:28:03Z

+- The generator needs live CipherStash credentials (two pairs, not alternatives):
+  - `CS_CLIENT_ACCESS_KEY` + `CS_WORKSPACE_CRN` — ZeroKMS auth, via `AutoStrategy`.
+  - `CS_CLIENT_ID` + `CS_CLIENT_KEY` — client key material, via `EnvKeyProvider`.
+  (`tasks/fixtures.toml:12-17`; built in `tests/sqlx/src/fixtures/cipherstash.rs:44-53`.)


Devs should almost never use the CS_* variables. Just login with npx stash auth login and auth/keys are stored in the profile (~/.cipherstash).

@coderdan The variables are required for testing and CI atm

It should say use env vars for CI and profile for local then.

coderdan · 2026-06-16T23:30:22Z

@@ -0,0 +1,351 @@
+# Fixture Generator Walkthrough


This looks like its targeted at devs who work on EQL (explaining its internals) rather than for devs who use EQL.

Might be worth moving to an internals folder and section and keep the main docs focused on consumption.

coderdan · 2026-06-16T23:31:36Z

+  `tests/sqlx/fixtures/v3_ste_vec.sql`, which is gitignored (`.gitignore:228`) and
+  rebuilt each run by its own generator (`fixtures::v3_ste_vec::generate()`).
+
+The plaintext **values** are single-sourced in the Rust catalog


What does "single-sourced" mean?

coderdan · 2026-06-16T23:46:44Z

+
+## 2. End-to-end pipeline
+
+```mermaid


Are any of the terms in this diagram defined somewhere? Might be worth doing so in copy above the diagram or linking to the relevant docs.

Defintions for: "ScalarSpec", "Catalog", "kind", "domain", "temporal_values" etc would be appreciated as I'm still not quite sure what they all mean!

@coderdan Good call. These are the internal types - this is a walkthrough of the internal implementation implementation.

This may not end up committed - was triggered by discussion in breakout yesterday.

tobyhede added 9 commits June 16, 2026 13:26

feat(v3): N-block ORE comparator + ordered numeric & timestamptz

aaec244

Derive the ORE block count N from term length instead of hardcoding 8, and wire the ordered numeric scalar while promoting timestamptz to ordered.

style(v3): cargo fmt sem.rs has_ore_block_256 test case

807b793

tobyhede added 3 commits June 16, 2026 18:16

docs(walkthroughs): fix review findings (counts, ste_vec framing, lin…

2c0f90b

…e refs)

docs: correct stale v3_ste_vec 'committed exception' claim in CLAUDE.…

07fa023

…md + walkthroughs

tobyhede force-pushed the docs/generator-walkthroughs branch from c249b44 to 07fa023 Compare June 16, 2026 08:16

coderdan reviewed Jun 16, 2026

View reviewed changes

Base automatically changed from eql-v3-ore-block-256 to eql-v3-ore-block-rename June 17, 2026 05:18

tobyhede force-pushed the eql-v3-ore-block-rename branch from f30ca5d to 7614212 Compare June 17, 2026 11:53

Base automatically changed from eql-v3-ore-block-rename to eql_v3 June 17, 2026 12:20

tobyhede closed this Jun 18, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs(v3): code generator + fixture generator walkthroughs (+ eql-types numeric/timestamptz)#283

docs(v3): code generator + fixture generator walkthroughs (+ eql-types numeric/timestamptz)#283
tobyhede wants to merge 12 commits into
eql_v3from
docs/generator-walkthroughs

tobyhede commented Jun 16, 2026

Uh oh!

coderabbitai Bot commented Jun 16, 2026 •

edited

Loading

Review skipped

Uh oh!

coderdan Jun 16, 2026

Uh oh!

tobyhede Jun 17, 2026

Uh oh!

coderdan Jun 17, 2026

Uh oh!

coderdan Jun 16, 2026

Uh oh!

coderdan Jun 16, 2026

Uh oh!

coderdan Jun 16, 2026

Uh oh!

tobyhede Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tobyhede commented Jun 16, 2026

Summary

Docs

eql-types

Review status

Test plan

Uh oh!

coderabbitai Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

coderdan Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

tobyhede Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

coderdan Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

coderdan Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

coderdan Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

coderdan Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

tobyhede Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented Jun 16, 2026 •

edited

Loading