Skip to content

Commit 3a86939

Browse files
authored
feat(migrate): EQL v3 support (#649)
* feat(migrate): detect a column's EQL version (v2 vs v3) First slice of EQL v3 support (#648): detectColumnEqlVersion() inspects an encrypted column's Postgres domain type — public.eql_v2_encrypted -> v2, a concrete eql_v3_* domain -> v3, anything else (plaintext / not found) -> null. This is the keystone the version-aware lifecycle branches on. Unit-tested with a mocked pg client; no behaviour change to existing v2 paths. Refs #648 Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w * feat(migrate,cli): EQL v3 rollout lifecycle — phases 2-5 (#648) Completes #649. The backfill engine was already version-agnostic (it never touches the v2 config machine in eql.ts — that's CLI-called only), and isEncryptedPayload already accepts both v3 wire shapes, so the work lands where the v2 coupling actually lived: migrate: - countEncrypted(client, table, column) — the v3 verification primitive (v3 has no eql_v2.count_encrypted_with_active_config; a plain count of the populated domain column is the equivalent, the domain CHECK already guarantees validity). - Manifest gains an optional eqlVersion (2|3) field, recorded at backfill time, so status/plan branch without a DB round-trip. - backfill-v3.integration.test.ts pins the v3 contract against a real Postgres: flat-scalar and SteVec payloads through the leak guard, the $N::jsonb write, countEncrypted, idempotency. vitest fileParallelism disabled for the package — the two integration suites share the singleton cipherstash.cs_migrations schema and raced when parallel. cli (all auto-detected via detectColumnEqlVersion, no new flags): - encrypt backfill: detects the version, logs the lifecycle, records eqlVersion + the v3 target phase ('dropped' — no cut-over) in the manifest, and prints v3 next steps (switch-by-name, then drop). - encrypt cutover: v3 short-circuits with "not applicable" guidance (exit 0) before any v2 config-machine call. - encrypt drop: v3 runs from 'backfilled' and drops the ORIGINAL column (no <col>_plaintext exists under v3); v2 unchanged, both pinned by tests. - encrypt status: v2-only drift flags (not-registered, plaintext-col-missing) suppressed for v3 columns; EQL column shows 'v3'. - Fixed a pre-existing exit-code bug the new tests exposed: cutover/drop precondition guards `return` from inside `try`, so the trailing `if (exitCode) process.exit(...)` was unreachable — precondition failures exited 0. The exit now lives in `finally`. Verified live against postgres-eql + EQL v3 installed by this branch's CLI: real EncryptionV3 ciphertext backfilled into a concrete eql_v3_text domain column (CHECK satisfied), detectColumnEqlVersion → v3, countEncrypted exact, decrypt round-trip exact. Docs: migrate README rewritten around the two lifecycles; stash-cli and stash-encryption skills updated (the #648 v2-only notes are gone). Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w * fix(migrate,cli): resolve EQL columns from domain types; verify before drop Code-review round on #649. The EQL v3 types are self-describing — that was the point of v3 — so version and encrypted-column resolution now come from the Postgres domain types, with the <col>_encrypted naming demoted to an unenforced convention: - @cipherstash/migrate: classifyEqlDomain (the one home for the rule), listEncryptedColumns, resolveEncryptedColumn (hint → convention → sole-EQL-column). detectColumnEqlVersion resolves tables case-exactly (format('%I') before to_regclass) — a bare to_regclass case-folded Prisma-style "User" tables to 'user' and silently wedged v3 columns into the v2 lifecycle. EqlVersion is numeric (2|3), matching the manifest and installer; no more string↔number translation. - The manifest records encryptedColumn at backfill time; cutover and drop resolve manifest-hint-first instead of guessing the name, so a custom --encrypted-column no longer deadlocks the v3 lifecycle. - encrypt drop (v3) now verifies live coverage before generating the migration: refuses while any row has plaintext set and ciphertext NULL (countUnencrypted) — the verification the README promised but nothing performed. Dropping the original column is the irreversible step; the phase gate alone only proved a backfill once finished. - encrypt cutover (v3) is phase-aware: mid-backfill it exits 1 telling the user to finish the backfill, instead of exit-0 guidance to switch the app onto a half-populated column. The v2 path guards the eql_v2_configuration query so v3-only databases get an explanation, not a raw relation-does-not-exist error. - encrypt status classifies from the observed domain type (manifest as fallback) — a v3 column whose manifest entry predates detection no longer collects permanent false not-registered flags. stash status's quest ladder and the init agent handoff teach the v3 next step (4-rung ladder, switch-by-name) instead of steering v3 users into the no-op cutover. - Docs trued up: manifest docstrings, migrate README (drop's built-in coverage gate), skills' phase-ladder intros (both lifecycles), and the stale workflow comments. Unit + real-catalog integration coverage for the resolver (mixed-case tables, custom names, ambiguity); CLI tests pin the coverage gate, the phase-aware cutover, and resolved-name usage. 507 CLI + 49 migrate + 58 pty-e2e green. Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w * test(migrate): domain-typed target column; tighten eql_v3_ prefix Two review-thread fixes on #649: the v3 backfill integration table now types email_encrypted with a domain carrying the real eql_v3_* storage CHECK shape (both scalar and SteVec arms), so the $N::jsonb write exercises the implicit jsonb→domain cast + CHECK enforcement instead of plain jsonb; and classifyEqlDomain matches the 'eql_v3_' prefix with the underscore so a hypothetical eql_v30_* generation can't classify as v3 (negative pins added). Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w * docs(skills): cutover on v3 is phase-aware — exit 0 only when backfilled Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w * fix(cli,migrate): harden EQL column resolution per #649 review Review follow-ups from Toby, James, and CodeRabbit: - Resolution provenance: pickEncryptedColumn (new pure core of resolveEncryptedColumn) tags every match `via: hint | convention | sole`. A sole-EQL-column match only proves uniqueness, not the plaintext<->ciphertext pairing, so `encrypt drop` — the one irreversible step — now refuses it with instructions instead of gating coverage on a possibly unrelated column. Display paths keep using it, so convention-free resolution still works everywhere else. - Fail closed on ambiguity: cutover and drop error listing the candidate EQL columns when none is identifiable, instead of falling through to the v2 machinery; the post-cutover v2 same-name state is recognised and still falls through to the v2 preconditions. - The generated v3 drop migration re-verifies coverage at APPLY time: LOCK TABLE, re-count plaintext-only rows, RAISE EXCEPTION without dropping if any appeared since generation; the DROP runs via EXECUTE in the same DO block so check-and-drop stay atomic under non-transactional runners. - resolveColumnLifecycle no longer swallows manifest read failures (ENOENT stays null via readManifest; malformed JSON/schema/permission errors propagate) and fetches the catalog once instead of up to three times. - cutover on an already-dropped v3 column says "lifecycle complete" (exit 0) instead of "finish the backfill". - stash status derives eqlVersion from the encrypted column's domain type (manifest encryptedColumn hint over the naming convention), falling back to the manifest only when the DB isn't visible — matching encrypt status. - Stale drop.ts JSDoc rewritten version-aware; migrate README fence language (MD040); stash-cli skill blockquote joined (MD028). Test-cast note: the `as unknown as ClientBase` factories in migrate tests stay (ClientBase's overloaded query signature can't be met by a structural fixture) but most resolution tests now target the pure pickEncryptedColumn and need no client at all. Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
1 parent d5b2d5b commit 3a86939

24 files changed

Lines changed: 1838 additions & 82 deletions

File tree

.changeset/migrate-eql-v3.md

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
---
2+
'@cipherstash/migrate': minor
3+
'stash': minor
4+
---
5+
6+
EQL v3 support for the encryption rollout lifecycle (#648). The `stash
7+
encrypt *` commands (and `@cipherstash/migrate` underneath) now resolve a
8+
column's EQL version and its encrypted counterpart from the **Postgres domain
9+
types** — the EQL v3 types are self-describing, so the `<col>_encrypted`
10+
naming is a convention only, never enforced or relied upon — and follow the
11+
right lifecycle, no new flags:
12+
13+
- **`encrypt backfill`** works on v3 columns unchanged (the engine was always
14+
version-agnostic; pass an `EncryptionV3` client and real v3 envelopes land
15+
in the concrete `eql_v3_*` domain column — verified live against a real
16+
database, including the domain CHECK and a decrypt round-trip). The
17+
manifest records the detected version, the encrypted column's name, and the
18+
v3 target phase, and the command prints v3-appropriate next steps.
19+
- **`encrypt cutover`** on a backfilled v3 column reports "not applicable"
20+
(exit 0) with guidance: v3 has no rename cut-over — the application
21+
switches to the encrypted column by name. Before backfill completes it
22+
exits 1 and says to finish the backfill instead of instructing the switch.
23+
On a database with no `eql_v2_configuration` table (a v3-only install) the
24+
v2 path now explains that instead of surfacing a raw Postgres error.
25+
- **`encrypt drop`** is version-aware: v3 runs from the `backfilled` phase,
26+
**verifies live coverage** (refuses to generate the migration while any row
27+
still has the plaintext set and the encrypted column NULL — the
28+
`countUnencrypted` check), and drops the ORIGINAL plaintext column (there
29+
is no `<col>_plaintext` under v3); v2 behaviour is unchanged. The generated
30+
v3 migration **re-verifies coverage at apply time** — it locks the table,
31+
re-counts, and aborts without dropping if plaintext-only rows appeared
32+
after generation. And because dropping is the one irreversible step, it
33+
requires a positively asserted plaintext↔ciphertext pairing (the
34+
manifest's recorded `encryptedColumn` or the naming convention): a match
35+
found only by being the table's sole EQL column is refused with
36+
instructions, and an ambiguous table (several EQL columns, none
37+
identifiable) fails closed listing the candidates — as does `cutover`.
38+
- **`encrypt status`** classifies each column from the observed domain type
39+
(manifest as fallback), shows `v3` in the EQL column, and no longer raises
40+
the v2-only `not-registered` / `plaintext-col-missing` drift flags for v3
41+
columns. `stash status`'s quest ladder and the `stash init` agent handoff
42+
prompt teach the version-appropriate next step (no more "run cutover" on
43+
v3 columns).
44+
- New `@cipherstash/migrate` exports: `classifyEqlDomain`,
45+
`resolveEncryptedColumn`, `pickEncryptedColumn`, `listEncryptedColumns`
46+
(domain-type resolution — case-exact for quoted/mixed-case table names),
47+
`countEncrypted` / `countUnencrypted` (coverage counts), and manifest
48+
`eqlVersion` + `encryptedColumn` fields. `EqlVersion` is numeric (`2 | 3`),
49+
matching the manifest and the installer. Resolved columns carry `via:
50+
'hint' | 'convention' | 'sole'` so callers can tell a positively asserted
51+
pairing from a by-elimination guess.
52+
- Fixed: `encrypt cutover`/`encrypt drop` precondition failures now actually
53+
exit 1 — the early-return guards previously skipped the exit-code path
54+
entirely, so failed preconditions exited 0. (This also applies to v2
55+
preconditions: scripted pipelines that relied on the erroneous exit 0 will
56+
now see the documented exit 1.)
57+
58+
The `stash-cli` and `stash-encryption` skills and the `@cipherstash/migrate`
59+
README document the two lifecycles (v2: backfill → cutover → drop;
60+
v3: backfill → switch-by-name → drop).

packages/cli/src/bin/main.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -122,7 +122,7 @@ Commands:
122122
encrypt status Show per-column migration status (phase, progress, drift)
123123
encrypt plan Diff intent (.cipherstash/migrations.json) vs observed state
124124
encrypt backfill Resumably encrypt plaintext into the encrypted column
125-
encrypt cutover Rename swap encrypted → primary column
125+
encrypt cutover Rename swap encrypted → primary column (EQL v2 only)
126126
encrypt drop Generate a migration to drop the plaintext column
127127
128128
env (experimental) Print production env vars for deployment

packages/cli/src/cli/registry.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -498,7 +498,7 @@ export const registry: CommandGroup[] = [
498498
},
499499
{
500500
name: 'encrypt cutover',
501-
summary: 'Rename swap encrypted → primary column',
501+
summary: 'Rename swap encrypted → primary column (EQL v2 only)',
502502
flags: [
503503
TABLE_FLAG,
504504
COLUMN_FLAG,

0 commit comments

Comments
 (0)