Skip to content

Commit 17b46e7

Browse files
committed
feat(migrate,cli): EQL v3 rollout lifecycle — phases 2-5 (#648)
Completes #649. The backfill engine was already version-agnostic (it never touches the v2 config machine in eql.ts — that's CLI-called only), and isEncryptedPayload already accepts both v3 wire shapes, so the work lands where the v2 coupling actually lived: migrate: - countEncrypted(client, table, column) — the v3 verification primitive (v3 has no eql_v2.count_encrypted_with_active_config; a plain count of the populated domain column is the equivalent, the domain CHECK already guarantees validity). - Manifest gains an optional eqlVersion (2|3) field, recorded at backfill time, so status/plan branch without a DB round-trip. - backfill-v3.integration.test.ts pins the v3 contract against a real Postgres: flat-scalar and SteVec payloads through the leak guard, the $N::jsonb write, countEncrypted, idempotency. vitest fileParallelism disabled for the package — the two integration suites share the singleton cipherstash.cs_migrations schema and raced when parallel. cli (all auto-detected via detectColumnEqlVersion, no new flags): - encrypt backfill: detects the version, logs the lifecycle, records eqlVersion + the v3 target phase ('dropped' — no cut-over) in the manifest, and prints v3 next steps (switch-by-name, then drop). - encrypt cutover: v3 short-circuits with "not applicable" guidance (exit 0) before any v2 config-machine call. - encrypt drop: v3 runs from 'backfilled' and drops the ORIGINAL column (no <col>_plaintext exists under v3); v2 unchanged, both pinned by tests. - encrypt status: v2-only drift flags (not-registered, plaintext-col-missing) suppressed for v3 columns; EQL column shows 'v3'. - Fixed a pre-existing exit-code bug the new tests exposed: cutover/drop precondition guards `return` from inside `try`, so the trailing `if (exitCode) process.exit(...)` was unreachable — precondition failures exited 0. The exit now lives in `finally`. Verified live against postgres-eql + EQL v3 installed by this branch's CLI: real EncryptionV3 ciphertext backfilled into a concrete eql_v3_text domain column (CHECK satisfied), detectColumnEqlVersion → v3, countEncrypted exact, decrypt round-trip exact. Docs: migrate README rewritten around the two lifecycles; stash-cli and stash-encryption skills updated (the #648 v2-only notes are gone). Claude-Session: https://claude.ai/code/session_01MxTTPaPP16m6br7Hoab94w
1 parent 4f2542d commit 17b46e7

16 files changed

Lines changed: 564 additions & 26 deletions

File tree

.changeset/migrate-eql-v3.md

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
---
2+
'@cipherstash/migrate': minor
3+
'stash': minor
4+
---
5+
6+
EQL v3 support for the encryption rollout lifecycle (#648). The `stash
7+
encrypt *` commands (and `@cipherstash/migrate` underneath) now auto-detect a
8+
column's EQL version from its Postgres domain type and follow the right
9+
lifecycle — no new flags:
10+
11+
- **`encrypt backfill`** works on v3 columns unchanged (the engine was always
12+
version-agnostic; pass an `EncryptionV3` client and real v3 envelopes land
13+
in the concrete `eql_v3_*` domain column — verified live against a real
14+
database, including the domain CHECK and a decrypt round-trip). The
15+
manifest records the detected version and the v3 target phase, and the
16+
command prints v3-appropriate next steps.
17+
- **`encrypt cutover`** on a v3 column reports "not applicable" (exit 0) with
18+
guidance: v3 has no rename cut-over — the application switches to
19+
`<col>_encrypted` by name.
20+
- **`encrypt drop`** is version-aware: v3 runs from the `backfilled` phase
21+
and drops the ORIGINAL plaintext column (there is no `<col>_plaintext`
22+
under v3); v2 behaviour is unchanged.
23+
- **`encrypt status`** no longer raises the v2-only `not-registered` /
24+
`plaintext-col-missing` drift flags for v3 columns (v3 has no
25+
`eql_v2_configuration` and no rename) and shows `v3` in the EQL column.
26+
- New `@cipherstash/migrate` exports: `countEncrypted` (the v3
27+
backfill-verification primitive) and a manifest `eqlVersion` field.
28+
- Fixed: `encrypt cutover`/`encrypt drop` precondition failures now actually
29+
exit 1 — the early-return guards previously skipped the exit-code path
30+
entirely, so failed preconditions exited 0.
31+
32+
The `stash-cli` and `stash-encryption` skills and the `@cipherstash/migrate`
33+
README document the two lifecycles (v2: backfill → cutover → drop;
34+
v3: backfill → switch-by-name → drop).

packages/cli/src/bin/main.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -122,7 +122,7 @@ Commands:
122122
encrypt status Show per-column migration status (phase, progress, drift)
123123
encrypt plan Diff intent (.cipherstash/migrations.json) vs observed state
124124
encrypt backfill Resumably encrypt plaintext into the encrypted column
125-
encrypt cutover Rename swap encrypted → primary column
125+
encrypt cutover Rename swap encrypted → primary column (EQL v2 only)
126126
encrypt drop Generate a migration to drop the plaintext column
127127
128128
env (experimental) Print production env vars for deployment

packages/cli/src/cli/registry.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -498,7 +498,7 @@ export const registry: CommandGroup[] = [
498498
},
499499
{
500500
name: 'encrypt cutover',
501-
summary: 'Rename swap encrypted → primary column',
501+
summary: 'Rename swap encrypted → primary column (EQL v2 only)',
502502
flags: [
503503
TABLE_FLAG,
504504
COLUMN_FLAG,
Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
import { beforeEach, describe, expect, it, vi } from 'vitest'
2+
3+
// The EQL-v3 lifecycle branches (#648/#649): v3 has no cut-over rename —
4+
// `encrypt cutover` must short-circuit with guidance instead of running the
5+
// v2 config machine, and `encrypt drop` must target the ORIGINAL plaintext
6+
// column (there is no `<col>_plaintext`) gated on `backfilled` rather than
7+
// `cut-over`. The v2 paths are pinned alongside as regression guards.
8+
9+
const queryMock = vi.hoisted(() => vi.fn(async () => ({ rows: [] })))
10+
vi.mock('pg', () => ({
11+
default: {
12+
Client: class {
13+
connect = vi.fn(async () => {})
14+
end = vi.fn(async () => {})
15+
query = queryMock
16+
},
17+
},
18+
}))
19+
20+
const migrateMocks = vi.hoisted(() => ({
21+
detectColumnEqlVersion: vi.fn(async () => 'v3' as 'v2' | 'v3' | null),
22+
progress: vi.fn(
23+
async () => ({ phase: 'backfilled' }) as { phase: string } | null,
24+
),
25+
appendEvent: vi.fn(async () => {}),
26+
setManifestTargetPhase: vi.fn(async () => {}),
27+
renameEncryptedColumns: vi.fn(async () => {}),
28+
migrateConfig: vi.fn(async () => {}),
29+
activateConfig: vi.fn(async () => {}),
30+
reloadConfig: vi.fn(async () => {}),
31+
}))
32+
vi.mock('@cipherstash/migrate', () => migrateMocks)
33+
34+
vi.mock('@clack/prompts', () => ({
35+
intro: vi.fn(),
36+
outro: vi.fn(),
37+
confirm: vi.fn(async () => true),
38+
isCancel: vi.fn(() => false),
39+
log: {
40+
info: vi.fn(),
41+
success: vi.fn(),
42+
warn: vi.fn(),
43+
error: vi.fn(),
44+
step: vi.fn(),
45+
},
46+
note: vi.fn(),
47+
}))
48+
vi.mock('@/config/index.js', () => ({
49+
loadStashConfig: vi.fn(async () => ({ databaseUrl: 'postgres://test' })),
50+
}))
51+
vi.mock('@/commands/db/detect.js', () => ({
52+
detectDrizzle: vi.fn(() => false),
53+
}))
54+
vi.mock('@/commands/init/utils.js', () => ({
55+
detectPackageManager: vi.fn(() => 'npm'),
56+
runnerCommand: vi.fn((_pm: string, ref: string) => `npx ${ref}`),
57+
}))
58+
const scaffoldMock = vi.hoisted(() =>
59+
vi.fn(async ({ name }: { name: string }) => ({
60+
path: `drizzle/${name}.sql`,
61+
})),
62+
)
63+
vi.mock('../drizzle-helper.js', () => ({
64+
scaffoldDrizzleMigration: scaffoldMock,
65+
}))
66+
const writeFileMock = vi.hoisted(() => vi.fn())
67+
vi.mock('node:fs', () => ({
68+
default: { mkdirSync: vi.fn(), writeFileSync: writeFileMock },
69+
}))
70+
71+
import * as p from '@clack/prompts'
72+
import { cutoverCommand } from '../cutover.js'
73+
import { dropCommand } from '../drop.js'
74+
75+
describe('encrypt cutover — EQL version awareness', () => {
76+
beforeEach(() => {
77+
vi.clearAllMocks()
78+
migrateMocks.progress.mockResolvedValue({ phase: 'backfilled' })
79+
})
80+
81+
it('short-circuits on a v3 column: guidance, no rename, no config machine', async () => {
82+
migrateMocks.detectColumnEqlVersion.mockResolvedValueOnce('v3')
83+
84+
await cutoverCommand({ table: 'users', column: 'email' })
85+
86+
expect(p.log.info).toHaveBeenCalledWith(
87+
expect.stringContaining('not applicable to EQL v3'),
88+
)
89+
expect(p.log.info).toHaveBeenCalledWith(
90+
expect.stringContaining('stash encrypt drop'),
91+
)
92+
expect(migrateMocks.renameEncryptedColumns).not.toHaveBeenCalled()
93+
expect(migrateMocks.migrateConfig).not.toHaveBeenCalled()
94+
expect(migrateMocks.activateConfig).not.toHaveBeenCalled()
95+
expect(migrateMocks.appendEvent).not.toHaveBeenCalled()
96+
})
97+
98+
it('still runs the v2 flow for a v2 column (regression pin)', async () => {
99+
migrateMocks.detectColumnEqlVersion.mockResolvedValueOnce('v2')
100+
// pending-config check returns true
101+
queryMock.mockImplementation(async (sql: string) =>
102+
typeof sql === 'string' && sql.includes('eql_v2_configuration')
103+
? { rows: [{ exists: true }] }
104+
: { rows: [] },
105+
)
106+
107+
await cutoverCommand({ table: 'users', column: 'email' })
108+
109+
expect(migrateMocks.renameEncryptedColumns).toHaveBeenCalled()
110+
expect(migrateMocks.migrateConfig).toHaveBeenCalled()
111+
expect(migrateMocks.activateConfig).toHaveBeenCalled()
112+
})
113+
})
114+
115+
describe('encrypt drop — EQL version awareness', () => {
116+
beforeEach(() => {
117+
vi.clearAllMocks()
118+
})
119+
120+
it('v3: requires phase backfilled and drops the ORIGINAL column', async () => {
121+
migrateMocks.detectColumnEqlVersion.mockResolvedValueOnce('v3')
122+
migrateMocks.progress.mockResolvedValueOnce({ phase: 'backfilled' })
123+
124+
await dropCommand({ table: 'users', column: 'email' })
125+
126+
// Non-drizzle fallback writes the SQL file — inspect the generated DDL.
127+
const sql = writeFileMock.mock.calls[0]?.[1] as string
128+
expect(sql).toContain('DROP COLUMN "email"')
129+
expect(sql).not.toContain('email_plaintext')
130+
expect(migrateMocks.appendEvent).toHaveBeenCalledWith(
131+
expect.anything(),
132+
expect.objectContaining({ event: 'dropped', phase: 'dropped' }),
133+
)
134+
})
135+
136+
it('v3: rejects when not yet backfilled', async () => {
137+
migrateMocks.detectColumnEqlVersion.mockResolvedValueOnce('v3')
138+
migrateMocks.progress.mockResolvedValueOnce({ phase: 'backfilling' })
139+
const exitSpy = vi
140+
.spyOn(process, 'exit')
141+
.mockImplementation((() => undefined) as never)
142+
143+
await dropCommand({ table: 'users', column: 'email' })
144+
145+
expect(p.log.error).toHaveBeenCalledWith(
146+
expect.stringContaining("Must be 'backfilled'"),
147+
)
148+
expect(writeFileMock).not.toHaveBeenCalled()
149+
expect(exitSpy).toHaveBeenCalledWith(1)
150+
exitSpy.mockRestore()
151+
})
152+
153+
it('v2: unchanged — requires cut-over and drops <col>_plaintext (regression pin)', async () => {
154+
migrateMocks.detectColumnEqlVersion.mockResolvedValueOnce('v2')
155+
migrateMocks.progress.mockResolvedValueOnce({ phase: 'cut-over' })
156+
157+
await dropCommand({ table: 'users', column: 'email' })
158+
159+
const sql = writeFileMock.mock.calls[0]?.[1] as string
160+
expect(sql).toContain('DROP COLUMN "email_plaintext"')
161+
})
162+
})

packages/cli/src/commands/encrypt/backfill.ts

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import {
22
appendEvent,
3+
detectColumnEqlVersion,
34
type ManifestColumn,
45
progress,
56
runBackfill,
@@ -136,6 +137,22 @@ export async function backfillCommand(options: BackfillCommandOptions) {
136137
const encryptedColumn =
137138
options.encryptedColumn ?? `${options.column}_encrypted`
138139

140+
// v2 or v3 changes the rest of the LIFECYCLE (v3 has no cut-over — the
141+
// ladder is backfill → switch-by-name → drop), so detect it up front,
142+
// record it in the manifest, and tell the user which path they're on.
143+
// `null` means the target column doesn't exist or isn't an EQL domain —
144+
// let the existing checks below produce their specific errors.
145+
const eqlVersion = await detectColumnEqlVersion(
146+
db,
147+
options.table,
148+
encryptedColumn,
149+
)
150+
if (eqlVersion) {
151+
p.log.info(
152+
`${options.table}.${encryptedColumn} is EQL ${eqlVersion}${eqlVersion === 'v3' ? ' — lifecycle is backfill → switch the app to the encrypted column by name → drop (no cut-over rename).' : ''}`,
153+
)
154+
}
155+
139156
// Phase guard: backfill requires the application to already be writing
140157
// to both columns, otherwise rows inserted *during* the backfill land
141158
// in plaintext only and create silent migration drift. Records the
@@ -186,6 +203,7 @@ export async function backfillCommand(options: BackfillCommandOptions) {
186203
schemaColumnKey,
187204
plaintextColumn,
188205
options.pkColumn,
206+
eqlVersion,
189207
)
190208
await upsertManifestColumn(options.table, manifestEntry)
191209
p.log.success(
@@ -254,6 +272,12 @@ export async function backfillCommand(options: BackfillCommandOptions) {
254272
return
255273
}
256274

275+
if (eqlVersion === 'v3') {
276+
p.note(
277+
`EQL v3 has no cut-over. Next:\n 1. Point your application at ${encryptedColumn} (schema + queries), deploy, verify reads.\n 2. Generate the plaintext drop: stash encrypt drop --table ${options.table} --column ${plaintextColumn}`,
278+
'Next steps (EQL v3)',
279+
)
280+
}
257281
p.outro(
258282
`Backfill complete. ${result.rowsProcessed.toLocaleString()} rows encrypted.`,
259283
)
@@ -499,6 +523,7 @@ function buildManifestEntry(
499523
schemaColumnKey: string,
500524
plaintextColumn: string,
501525
pkColumn: string | undefined,
526+
eqlVersion: 'v2' | 'v3' | null,
502527
): ManifestColumn {
503528
// SDK `cast_as` ('string', 'number', …) and EQL `castAs` ('text',
504529
// 'double', …) are different vocabularies; translate via the same
@@ -520,9 +545,12 @@ function buildManifestEntry(
520545
column: plaintextColumn,
521546
castAs,
522547
indexes,
523-
targetPhase: 'cut-over',
548+
// v2's ladder ends with the rename cut-over; v3 has none — its end
549+
// state is the plaintext column dropped.
550+
targetPhase: eqlVersion === 'v3' ? 'dropped' : 'cut-over',
524551
...(pkColumn ? { pkColumn } : {}),
525-
}
552+
...(eqlVersion ? { eqlVersion: eqlVersion === 'v3' ? 3 : 2 } : {}),
553+
} as ManifestColumn
526554
}
527555

528556
// Drop the wrapping default so unknown values fail validation instead of

packages/cli/src/commands/encrypt/cutover.ts

Lines changed: 27 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import {
22
activateConfig,
33
appendEvent,
4+
detectColumnEqlVersion,
45
migrateConfig,
56
progress,
67
reloadConfig,
@@ -62,6 +63,26 @@ export async function cutoverCommand(options: CutoverCommandOptions) {
6263
try {
6364
await client.connect()
6465

66+
// Cut-over is an EQL v2 concept: v2 hides the swap behind
67+
// `eql_v2.rename_encrypted_columns()` + a Proxy config promotion. A v3
68+
// column has neither — the application switches to the encrypted column
69+
// BY NAME, and the plaintext column is dropped later. Detect before any
70+
// phase/config checks so v3 users get the real answer, not a confusing
71+
// precondition error.
72+
const encryptedColumn = `${options.column}_encrypted`
73+
const version = await detectColumnEqlVersion(
74+
client,
75+
options.table,
76+
encryptedColumn,
77+
)
78+
if (version === 'v3') {
79+
p.log.info(
80+
`Cut-over is not applicable to EQL v3 columns. ${options.table}.${encryptedColumn} is EQL v3: there is no rename step — point your application at ${encryptedColumn} (update your schema/queries), verify reads, then generate the plaintext drop with:\n stash encrypt drop --table ${options.table} --column ${options.column}`,
81+
)
82+
p.outro('Nothing to do for EQL v3.')
83+
return
84+
}
85+
6586
const state = await progress(client, options.table, options.column)
6687
if (state?.phase !== 'backfilled') {
6788
p.log.error(
@@ -183,8 +204,13 @@ export async function cutoverCommand(options: CutoverCommandOptions) {
183204
exitCode = 1
184205
} finally {
185206
await client.end()
207+
// In `finally` (not after the try/catch) deliberately: the precondition
208+
// guards above `return` from inside `try`, which skips any code placed
209+
// after the block — so a trailing `if (exitCode) process.exit(...)`
210+
// was unreachable on exactly the failure paths it existed for, and
211+
// guard failures exited 0.
212+
if (exitCode) process.exit(exitCode)
186213
}
187-
if (exitCode) process.exit(exitCode)
188214
}
189215

190216
/**

0 commit comments

Comments
 (0)