Skip to content

Commit 9c21323

Browse files
committed
fix(cli): integrate cutover/drop with drizzle-kit; phase-aware status
Three issues from the spike's lifecycle smoke-test, all in the encrypt CLI. Bundled into one commit because they touch adjacent files. Issue 1: `encrypt drop` wrote a self-named timestamped migration (`20260504112456_drop_*.sql`) that drizzle-kit migrate refused to pick up — no journal entry, wrong prefix. Same shape `db install --drizzle` already gets right by shelling out to `drizzle-kit generate --custom`. Fix: detect drizzle and route through a new `packages/cli/src/commands/encrypt/drizzle-helper.ts` that wraps `drizzle-kit generate --custom --name=...`, locates the generated file, and writes the drop SQL into it. The migration now lands with a journal entry; `drizzle-kit migrate` applies it like any other migration. Non-drizzle projects keep the timestamped-file fallback (Prisma / raw-SQL paths planned). Issue 2: `encrypt cutover` ran `eql_v2.rename_encrypted_columns()` live and never told drizzle. Drizzle's `meta/_journal.json` and snapshot stayed pinned to the pre-rename shape, so the next `drizzle-kit generate` against the source produced a confused diff trying to recreate the old layout. Fix: after cutover succeeds (transaction committed), scaffold a follow-up custom drizzle migration containing idempotent `ALTER TABLE … RENAME COLUMN` statements wrapped in a `DO` block that checks whether `<col>_encrypted` still exists. On the source DB the rename already ran, so the block is a no-op and Drizzle's journal still records the migration; on a fresh restore the block performs the rename. Same file, both behaviours, reproducible. Non-drizzle projects skip the resync step (logged-only warning if scaffolding fails). Issue 3: `encrypt status` rendered `rowsProcessed/rowsTotal (pct%)` uniformly across every phase. The same fraction means different things at different points in the lifecycle, and `0/0 (100%)` for a `backfilled` column that needed no encrypting reads as nonsense. Fix: phase-aware framing for the PROGRESS column. `schema-added` shows `—`. `dual-writing` shows `(awaiting backfill)`. `backfilling` keeps the fraction. `backfilled` / `cut-over` / `dropped` show a plain completion marker instead of a degenerate ratio. Same data, phase-appropriate label. Wire `--migrations-dir <path>` through to cutover for projects with non-default drizzle out dirs. 166 tests pass; biome clean. Coverage check during dual-writing (the bug report's bonus suggestion — show "rows-with-both-columns / total" rather than just "awaiting backfill") needs a live SELECT against the user's table, not just the cs_migrations data we already have. Tracked as a follow-up; today's status surfaces phase awareness without new queries.
1 parent 8544a36 commit 9c21323

5 files changed

Lines changed: 262 additions & 32 deletions

File tree

packages/cli/src/bin/stash.ts

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -286,7 +286,12 @@ async function runEncryptCommand(
286286
const { cutoverCommand } = await requireStack(
287287
() => import('../commands/encrypt/cutover.js'),
288288
)
289-
await cutoverCommand({ table, column, proxyUrl: values['proxy-url'] })
289+
await cutoverCommand({
290+
table,
291+
column,
292+
proxyUrl: values['proxy-url'],
293+
migrationsDir: values['migrations-dir'],
294+
})
290295
break
291296
}
292297
case 'drop': {

packages/cli/src/commands/encrypt/cutover.ts

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
import { detectDrizzle } from '@/commands/db/detect.js'
12
import { loadStashConfig } from '@/config/index.js'
23
import {
34
activateConfig,
@@ -9,6 +10,7 @@ import {
910
} from '@cipherstash/migrate'
1011
import * as p from '@clack/prompts'
1112
import pg from 'pg'
13+
import { scaffoldDrizzleMigration } from './drizzle-helper.js'
1214

1315
/**
1416
* Options accepted by `stash encrypt cutover`. Swaps the plaintext and
@@ -35,6 +37,12 @@ export interface CutoverCommandOptions {
3537
* Also readable from `CIPHERSTASH_PROXY_URL` in the environment.
3638
*/
3739
proxyUrl?: string
40+
/**
41+
* Drizzle migrations directory (passed to `drizzle-kit generate
42+
* --custom`). Defaults to `./drizzle`. Only used when the project is
43+
* Drizzle — non-Drizzle projects skip the snapshot-resync step.
44+
*/
45+
migrationsDir?: string
3846
}
3947

4048
/**
@@ -104,6 +112,36 @@ export async function cutoverCommand(options: CutoverCommandOptions) {
104112
`Renamed ${options.column}${options.column}_plaintext and ${options.column}_encrypted → ${options.column}; pending config promoted to active.`,
105113
)
106114

115+
// Drizzle snapshot resync. The rename above ran outside drizzle-kit's
116+
// authority — the snapshot at `<out>/meta/<idx>_snapshot.json` still
117+
// describes the pre-rename column shape. If we don't acknowledge the
118+
// change in Drizzle's metadata, the next `drizzle-kit generate` will
119+
// produce a confused diff trying to re-create the old layout.
120+
//
121+
// Scaffolding a custom migration with idempotent rename SQL solves
122+
// both problems: it adds the journal entry + snapshot diff that
123+
// Drizzle expects, and the SQL itself is a no-op on the source DB
124+
// (the pre-rename column doesn't exist any more) but applies
125+
// correctly when migrating a fresh database.
126+
if (detectDrizzle(process.cwd())) {
127+
try {
128+
const renameSql = buildRenameMigrationSql(options.table, options.column)
129+
const result = await scaffoldDrizzleMigration({
130+
name: `cutover_${options.table}_${options.column}`,
131+
outDir: options.migrationsDir ?? 'drizzle',
132+
sql: renameSql,
133+
})
134+
p.log.success(
135+
`Drizzle snapshot updated: ${result.path} (idempotent — no-op on this DB, applies on a fresh restore).`,
136+
)
137+
} catch (err) {
138+
const reason = err instanceof Error ? err.message : String(err)
139+
p.log.warn(
140+
`Could not scaffold the Drizzle rename migration: ${reason}\nDrizzle's snapshot may be out of sync with the live schema. Run \`drizzle-kit pull\` to resync, or scaffold the rename migration manually.`,
141+
)
142+
}
143+
}
144+
107145
const proxyUrl = options.proxyUrl ?? process.env.CIPHERSTASH_PROXY_URL
108146
if (proxyUrl) {
109147
const proxy = new pg.Client({ connectionString: proxyUrl })
@@ -130,3 +168,30 @@ export async function cutoverCommand(options: CutoverCommandOptions) {
130168
await client.end()
131169
}
132170
}
171+
172+
/**
173+
* Build the SQL body for the post-cutover Drizzle migration. Wrapped in a
174+
* `DO` block that checks whether `<col>_encrypted` still exists — on the
175+
* source database the rename already ran (so the column is gone and the
176+
* block does nothing), but on a fresh restore the rename hasn't run yet
177+
* (so the block performs the swap). Same migration file, both behaviours,
178+
* idempotent.
179+
*/
180+
function buildRenameMigrationSql(table: string, column: string): string {
181+
return `-- Generated by stash encrypt cutover.
182+
-- Records the rename that eql_v2.rename_encrypted_columns() performed
183+
-- so Drizzle's snapshot stays in sync. Idempotent: a no-op on the DB
184+
-- where cutover already ran; applies on a fresh restore.
185+
DO $$
186+
BEGIN
187+
IF EXISTS (
188+
SELECT 1 FROM information_schema.columns
189+
WHERE table_name = '${table}'
190+
AND column_name = '${column}_encrypted'
191+
) THEN
192+
ALTER TABLE "${table}" RENAME COLUMN "${column}" TO "${column}_plaintext";
193+
ALTER TABLE "${table}" RENAME COLUMN "${column}_encrypted" TO "${column}";
194+
END IF;
195+
END $$;
196+
`
197+
}
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
import { execSync } from 'node:child_process'
2+
import { existsSync } from 'node:fs'
3+
import { readdir, writeFile } from 'node:fs/promises'
4+
import { join, resolve } from 'node:path'
5+
6+
/**
7+
* Scaffold a custom Drizzle Kit migration file with a known name and write
8+
* the supplied SQL into it. Mirrors the dance `db install --drizzle` already
9+
* does — `drizzle-kit generate --custom` creates the file and records the
10+
* journal entry / snapshot, then we overwrite the empty body with our SQL.
11+
*
12+
* Used by `encrypt drop` (to ship the plaintext-column drop migration in a
13+
* shape `drizzle-kit migrate` actually picks up) and `encrypt cutover` (to
14+
* record the live rename so Drizzle's snapshot reflects post-cutover
15+
* reality).
16+
*
17+
* Throws if `drizzle-kit` isn't on PATH or the generated file can't be
18+
* located afterwards. Callers should fall back to the self-named-file
19+
* approach for non-Drizzle projects.
20+
*/
21+
export async function scaffoldDrizzleMigration(opts: {
22+
/**
23+
* Migration name passed to `drizzle-kit generate --custom --name=<name>`.
24+
* Drizzle prefixes with the next sequential index (`0003_<name>.sql`).
25+
*/
26+
name: string
27+
/** Drizzle's `out` directory, defaults to `./drizzle` if unset. */
28+
outDir: string
29+
/** SQL to write into the generated file. */
30+
sql: string
31+
}): Promise<{ path: string }> {
32+
const outDir = resolve(opts.outDir)
33+
if (!existsSync(outDir)) {
34+
throw new Error(
35+
`Drizzle output directory not found: ${outDir}\nMake sure drizzle-kit is configured correctly (check drizzle.config.ts's \`out\`).`,
36+
)
37+
}
38+
39+
// `drizzle-kit generate --custom` scaffolds an empty migration file with
40+
// the right prefix and records the journal/snapshot entry. Stderr is
41+
// captured so a missing-drizzle-kit error surfaces cleanly.
42+
try {
43+
execSync(`npx drizzle-kit generate --custom --name=${opts.name}`, {
44+
stdio: 'pipe',
45+
encoding: 'utf-8',
46+
})
47+
} catch (error) {
48+
const stderr =
49+
error !== null &&
50+
typeof error === 'object' &&
51+
'stderr' in error &&
52+
typeof error.stderr === 'string'
53+
? error.stderr.trim()
54+
: undefined
55+
throw new Error(
56+
`Failed to scaffold Drizzle migration: ${stderr ?? (error instanceof Error ? error.message : String(error))}`,
57+
)
58+
}
59+
60+
const generatedPath = await findLatestNamedMigration(outDir, opts.name)
61+
await writeFile(generatedPath, opts.sql, 'utf-8')
62+
63+
return { path: generatedPath }
64+
}
65+
66+
/**
67+
* Find the most recent `0NNN_<name>.sql` in the Drizzle out directory.
68+
* Drizzle's flat numbered prefix means string-sort is the right order.
69+
*/
70+
async function findLatestNamedMigration(
71+
outDir: string,
72+
name: string,
73+
): Promise<string> {
74+
const entries = await readdir(outDir)
75+
const matching = entries
76+
.filter((entry) => entry.endsWith('.sql') && entry.includes(name))
77+
.sort()
78+
if (matching.length === 0) {
79+
throw new Error(
80+
`Could not find a migration matching "${name}" in ${outDir} after running drizzle-kit generate.`,
81+
)
82+
}
83+
return join(outDir, matching[matching.length - 1] as string)
84+
}

packages/cli/src/commands/encrypt/drop.ts

Lines changed: 48 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
import fs from 'node:fs'
22
import path from 'node:path'
3+
import { detectDrizzle } from '@/commands/db/detect.js'
34
import { loadStashConfig } from '@/config/index.js'
45
import {
56
appendEvent,
@@ -8,6 +9,7 @@ import {
89
} from '@cipherstash/migrate'
910
import * as p from '@clack/prompts'
1011
import pg from 'pg'
12+
import { scaffoldDrizzleMigration } from './drizzle-helper.js'
1113

1214
/**
1315
* Options accepted by `stash encrypt drop`. Generates a migration file
@@ -33,9 +35,17 @@ export interface DropCommandOptions {
3335

3436
/**
3537
* CLI handler for `stash encrypt drop`. Requires the column to be in
36-
* phase `cut-over`; otherwise errors out. Writes a timestamped
37-
* `ALTER TABLE … DROP COLUMN <col>_plaintext` statement, appends a
38-
* `dropped` event, and prints instructions for applying the migration.
38+
* phase `cut-over`; otherwise errors out.
39+
*
40+
* For Drizzle projects, scaffolds the migration via `drizzle-kit generate
41+
* --custom` so the file lands with the correct sequential prefix and a
42+
* journal/snapshot entry — which is what `drizzle-kit migrate` actually
43+
* picks up. Hand-rolling the file (the prior behaviour) wrote a
44+
* timestamped `<ts>_drop_*.sql` that Drizzle ignored.
45+
*
46+
* For non-Drizzle projects, falls back to the self-named file behaviour
47+
* — a clearly named SQL file the user reviews and applies with their own
48+
* tooling (Prisma migrate, psql, etc.).
3949
*/
4050
export async function dropCommand(options: DropCommandOptions) {
4151
p.intro('npx @cipherstash/cli encrypt drop')
@@ -54,41 +64,56 @@ export async function dropCommand(options: DropCommandOptions) {
5464
process.exit(1)
5565
}
5666

57-
const migrationsDir = path.resolve(
58-
process.cwd(),
59-
options.migrationsDir ?? 'drizzle',
60-
)
61-
fs.mkdirSync(migrationsDir, { recursive: true })
67+
const dropSql = `-- Generated by stash encrypt drop\n-- Drops the plaintext column now that ${options.table}.${options.column} is encrypted.\n\nALTER TABLE "${options.table}" DROP COLUMN "${options.column}_plaintext";\n`
6268

63-
const ts = new Date()
64-
.toISOString()
65-
.replace(/[-:.TZ]/g, '')
66-
.slice(0, 14)
67-
const fileName = `${ts}_drop_${options.table}_${options.column}_plaintext.sql`
68-
const filePath = path.join(migrationsDir, fileName)
69+
const cwd = process.cwd()
70+
const migrationsDir = options.migrationsDir ?? 'drizzle'
71+
const isDrizzle = detectDrizzle(cwd)
72+
let filePath: string
73+
let nextStep: string
6974

70-
const sql = `-- Generated by @cipherstash/cli encrypt drop\n-- Drops the plaintext column now that ${options.table}.${options.column} is encrypted.\n\nALTER TABLE "${options.table}" DROP COLUMN "${options.column}_plaintext";\n`
71-
fs.writeFileSync(filePath, sql, 'utf-8')
75+
if (isDrizzle) {
76+
// Scaffold via drizzle-kit so the migration is journaled + snapshotted.
77+
// Without this, the file ships but `drizzle-kit migrate` never picks it
78+
// up because the journal doesn't reference it.
79+
const result = await scaffoldDrizzleMigration({
80+
name: `drop_${options.table}_${options.column}_plaintext`,
81+
outDir: migrationsDir,
82+
sql: dropSql,
83+
})
84+
filePath = result.path
85+
nextStep =
86+
'Apply with your usual Drizzle migrate command:\n npx drizzle-kit migrate'
87+
} else {
88+
// Non-Drizzle fallback: emit a timestamped SQL file the user
89+
// applies with whichever migration tool they're running.
90+
const dirAbs = path.resolve(cwd, migrationsDir)
91+
fs.mkdirSync(dirAbs, { recursive: true })
92+
const ts = new Date()
93+
.toISOString()
94+
.replace(/[-:.TZ]/g, '')
95+
.slice(0, 14)
96+
const fileName = `${ts}_drop_${options.table}_${options.column}_plaintext.sql`
97+
filePath = path.join(dirAbs, fileName)
98+
fs.writeFileSync(filePath, dropSql, 'utf-8')
99+
nextStep = `Review the migration, then apply with your migration tool:\n - prisma migrate deploy\n - psql -f ${fileName}`
100+
}
72101

73102
await appendEvent(client, {
74103
tableName: options.table,
75104
columnName: options.column,
76105
event: 'dropped',
77106
phase: 'dropped',
78-
details: { migrationFile: filePath },
107+
details: { migrationFile: filePath, drizzleScaffolded: isDrizzle },
79108
})
80109

81110
// Bump the manifest's target phase so `encrypt plan` reflects the
82111
// user's commitment to fully removing the plaintext column. No-op
83-
// when the column wasn't tracked in the manifest yet (e.g. migrations
84-
// begun before the manifest-on-backfill behaviour shipped).
112+
// when the column wasn't tracked in the manifest yet.
85113
await setManifestTargetPhase(options.table, options.column, 'dropped')
86114

87115
p.log.success(`Migration written to ${filePath}`)
88-
p.note(
89-
`Review the migration, then apply it with your migration tool:\n - drizzle-kit migrate\n - prisma migrate deploy\n - psql -f ${fileName}`,
90-
'Next',
91-
)
116+
p.note(nextStep, 'Next')
92117
p.outro('Drop migration generated.')
93118
} catch (error) {
94119
p.log.error(

packages/cli/src/commands/encrypt/status.ts

Lines changed: 59 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -205,14 +205,13 @@ function renderRow(input: {
205205
? eqlColumn.indexes.join(', ') || '(none)'
206206
: intentIndexes?.join(', ') || '—'
207207

208-
let progress = '—'
209-
if (state && state.rowsTotal !== null && state.rowsTotal !== undefined) {
210-
const pct =
211-
state.rowsTotal > 0
212-
? Math.floor(((state.rowsProcessed ?? 0) / state.rowsTotal) * 100)
213-
: 100
214-
progress = `${state.rowsProcessed ?? 0}/${state.rowsTotal} (${pct}%)`
215-
}
208+
// The PROGRESS column means different things in different phases. The
209+
// raw rowsProcessed/rowsTotal numbers in cs_migrations come from the
210+
// backfill engine, but rendering them as a uniform fraction across
211+
// every phase produces nonsense at the boundaries (e.g. `0/0 (100%)`
212+
// for a `backfilled` column that needed no encrypting because dual-
213+
// writes covered every row from seeding). Frame per phase.
214+
const progress = formatProgress(phase, state)
216215

217216
const flags: string[] = []
218217
if (intentIndexes && !eqlColumn) flags.push('not-registered')
@@ -234,6 +233,58 @@ function renderRow(input: {
234233
}
235234
}
236235

236+
/**
237+
* Phase-aware framing for the PROGRESS column.
238+
*
239+
* schema-added — no backfill has run, no progress data yet.
240+
* dual-writing — backfill hasn't started either; the meaningful
241+
* measurement here is "is dual-write code populating
242+
* every new row?", which requires a live coverage
243+
* query against the user's table. We don't run that
244+
* here (the row would have to do its own SELECT) so
245+
* we surface "(awaiting backfill)" — see follow-ups.
246+
* backfilling — show backfill progress: rowsProcessed/rowsTotal.
247+
* Percentage based on rows already done.
248+
* backfilled — show "(complete)" instead of a degenerate ratio.
249+
* `0/0` here means "nothing needed encrypting
250+
* because dual-writes already covered every row" —
251+
* not a failure, but unintuitive as a fraction.
252+
* cut-over — physical rename complete, encrypted column live.
253+
* dropped — plaintext column gone, lifecycle complete.
254+
*/
255+
function formatProgress(
256+
phase: MigrationPhase | '—',
257+
state: {
258+
rowsProcessed: number | null
259+
rowsTotal: number | null
260+
} | null,
261+
): string {
262+
switch (phase) {
263+
case 'schema-added':
264+
return '—'
265+
case 'dual-writing':
266+
return '(awaiting backfill)'
267+
case 'backfilling': {
268+
if (!state || state.rowsTotal === null || state.rowsTotal === undefined) {
269+
return '—'
270+
}
271+
const pct =
272+
state.rowsTotal > 0
273+
? Math.floor(((state.rowsProcessed ?? 0) / state.rowsTotal) * 100)
274+
: 100
275+
return `${state.rowsProcessed ?? 0}/${state.rowsTotal} (${pct}%)`
276+
}
277+
case 'backfilled':
278+
return '(backfill complete)'
279+
case 'cut-over':
280+
return '(cut over)'
281+
case 'dropped':
282+
return '(dropped)'
283+
default:
284+
return '—'
285+
}
286+
}
287+
237288
function formatTable(rows: Row[]): string {
238289
const headers: Row = {
239290
table: 'TABLE',

0 commit comments

Comments
 (0)