Commit 3a86939
authored
feat(migrate): EQL v3 support (#649)
* feat(migrate): detect a column's EQL version (v2 vs v3)
First slice of EQL v3 support (#648): detectColumnEqlVersion() inspects an
encrypted column's Postgres domain type — public.eql_v2_encrypted -> v2, a
concrete eql_v3_* domain -> v3, anything else (plaintext / not found) -> null.
This is the keystone the version-aware lifecycle branches on. Unit-tested with a
mocked pg client; no behaviour change to existing v2 paths.
Refs #648
Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
* feat(migrate,cli): EQL v3 rollout lifecycle — phases 2-5 (#648)
Completes #649. The backfill engine was already version-agnostic (it never
touches the v2 config machine in eql.ts — that's CLI-called only), and
isEncryptedPayload already accepts both v3 wire shapes, so the work lands
where the v2 coupling actually lived:
migrate:
- countEncrypted(client, table, column) — the v3 verification primitive
(v3 has no eql_v2.count_encrypted_with_active_config; a plain count of
the populated domain column is the equivalent, the domain CHECK already
guarantees validity).
- Manifest gains an optional eqlVersion (2|3) field, recorded at backfill
time, so status/plan branch without a DB round-trip.
- backfill-v3.integration.test.ts pins the v3 contract against a real
Postgres: flat-scalar and SteVec payloads through the leak guard, the
$N::jsonb write, countEncrypted, idempotency. vitest fileParallelism
disabled for the package — the two integration suites share the
singleton cipherstash.cs_migrations schema and raced when parallel.
cli (all auto-detected via detectColumnEqlVersion, no new flags):
- encrypt backfill: detects the version, logs the lifecycle, records
eqlVersion + the v3 target phase ('dropped' — no cut-over) in the
manifest, and prints v3 next steps (switch-by-name, then drop).
- encrypt cutover: v3 short-circuits with "not applicable" guidance
(exit 0) before any v2 config-machine call.
- encrypt drop: v3 runs from 'backfilled' and drops the ORIGINAL column
(no <col>_plaintext exists under v3); v2 unchanged, both pinned by tests.
- encrypt status: v2-only drift flags (not-registered,
plaintext-col-missing) suppressed for v3 columns; EQL column shows 'v3'.
- Fixed a pre-existing exit-code bug the new tests exposed: cutover/drop
precondition guards `return` from inside `try`, so the trailing
`if (exitCode) process.exit(...)` was unreachable — precondition
failures exited 0. The exit now lives in `finally`.
Verified live against postgres-eql + EQL v3 installed by this branch's CLI:
real EncryptionV3 ciphertext backfilled into a concrete eql_v3_text domain
column (CHECK satisfied), detectColumnEqlVersion → v3, countEncrypted
exact, decrypt round-trip exact.
Docs: migrate README rewritten around the two lifecycles; stash-cli and
stash-encryption skills updated (the #648 v2-only notes are gone).
Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
* fix(migrate,cli): resolve EQL columns from domain types; verify before drop
Code-review round on #649. The EQL v3 types are self-describing — that
was the point of v3 — so version and encrypted-column resolution now
come from the Postgres domain types, with the <col>_encrypted naming
demoted to an unenforced convention:
- @cipherstash/migrate: classifyEqlDomain (the one home for the rule),
listEncryptedColumns, resolveEncryptedColumn (hint → convention →
sole-EQL-column). detectColumnEqlVersion resolves tables case-exactly
(format('%I') before to_regclass) — a bare to_regclass case-folded
Prisma-style "User" tables to 'user' and silently wedged v3 columns
into the v2 lifecycle. EqlVersion is numeric (2|3), matching the
manifest and installer; no more string↔number translation.
- The manifest records encryptedColumn at backfill time; cutover and
drop resolve manifest-hint-first instead of guessing the name, so a
custom --encrypted-column no longer deadlocks the v3 lifecycle.
- encrypt drop (v3) now verifies live coverage before generating the
migration: refuses while any row has plaintext set and ciphertext
NULL (countUnencrypted) — the verification the README promised but
nothing performed. Dropping the original column is the irreversible
step; the phase gate alone only proved a backfill once finished.
- encrypt cutover (v3) is phase-aware: mid-backfill it exits 1 telling
the user to finish the backfill, instead of exit-0 guidance to switch
the app onto a half-populated column. The v2 path guards the
eql_v2_configuration query so v3-only databases get an explanation,
not a raw relation-does-not-exist error.
- encrypt status classifies from the observed domain type (manifest as
fallback) — a v3 column whose manifest entry predates detection no
longer collects permanent false not-registered flags. stash status's
quest ladder and the init agent handoff teach the v3 next step (4-rung
ladder, switch-by-name) instead of steering v3 users into the no-op
cutover.
- Docs trued up: manifest docstrings, migrate README (drop's built-in
coverage gate), skills' phase-ladder intros (both lifecycles), and the
stale workflow comments.
Unit + real-catalog integration coverage for the resolver (mixed-case
tables, custom names, ambiguity); CLI tests pin the coverage gate, the
phase-aware cutover, and resolved-name usage. 507 CLI + 49 migrate +
58 pty-e2e green.
Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
* test(migrate): domain-typed target column; tighten eql_v3_ prefix
Two review-thread fixes on #649: the v3 backfill integration table now
types email_encrypted with a domain carrying the real eql_v3_* storage
CHECK shape (both scalar and SteVec arms), so the $N::jsonb write
exercises the implicit jsonb→domain cast + CHECK enforcement instead of
plain jsonb; and classifyEqlDomain matches the 'eql_v3_' prefix with
the underscore so a hypothetical eql_v30_* generation can't classify
as v3 (negative pins added).
Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
* docs(skills): cutover on v3 is phase-aware — exit 0 only when backfilled
Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
* fix(cli,migrate): harden EQL column resolution per #649 review
Review follow-ups from Toby, James, and CodeRabbit:
- Resolution provenance: pickEncryptedColumn (new pure core of
resolveEncryptedColumn) tags every match `via: hint | convention |
sole`. A sole-EQL-column match only proves uniqueness, not the
plaintext<->ciphertext pairing, so `encrypt drop` — the one
irreversible step — now refuses it with instructions instead of
gating coverage on a possibly unrelated column. Display paths keep
using it, so convention-free resolution still works everywhere else.
- Fail closed on ambiguity: cutover and drop error listing the
candidate EQL columns when none is identifiable, instead of falling
through to the v2 machinery; the post-cutover v2 same-name state is
recognised and still falls through to the v2 preconditions.
- The generated v3 drop migration re-verifies coverage at APPLY time:
LOCK TABLE, re-count plaintext-only rows, RAISE EXCEPTION without
dropping if any appeared since generation; the DROP runs via EXECUTE
in the same DO block so check-and-drop stay atomic under
non-transactional runners.
- resolveColumnLifecycle no longer swallows manifest read failures
(ENOENT stays null via readManifest; malformed JSON/schema/permission
errors propagate) and fetches the catalog once instead of up to three
times.
- cutover on an already-dropped v3 column says "lifecycle complete"
(exit 0) instead of "finish the backfill".
- stash status derives eqlVersion from the encrypted column's domain
type (manifest encryptedColumn hint over the naming convention),
falling back to the manifest only when the DB isn't visible —
matching encrypt status.
- Stale drop.ts JSDoc rewritten version-aware; migrate README fence
language (MD040); stash-cli skill blockquote joined (MD028).
Test-cast note: the `as unknown as ClientBase` factories in migrate
tests stay (ClientBase's overloaded query signature can't be met by a
structural fixture) but most resolution tests now target the pure
pickEncryptedColumn and need no client at all.
Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w1 parent d5b2d5b commit 3a86939
24 files changed
Lines changed: 1838 additions & 82 deletions
File tree
- .changeset
- packages
- cli/src
- bin
- cli
- commands
- encrypt
- __tests__
- lib
- init/lib
- __tests__
- status
- migrate
- src
- __tests__
- skills
- stash-cli
- stash-encryption
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
122 | 122 | | |
123 | 123 | | |
124 | 124 | | |
125 | | - | |
| 125 | + | |
126 | 126 | | |
127 | 127 | | |
128 | 128 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
498 | 498 | | |
499 | 499 | | |
500 | 500 | | |
501 | | - | |
| 501 | + | |
502 | 502 | | |
503 | 503 | | |
504 | 504 | | |
| |||
0 commit comments