Skip to content

Commit 3675408

Browse files
committed
feat(db): add DSQL-compatible migration runner with tests
The existing migrate.ts had hash-based verification that broke on formatter changes, lacked validation for most DSQL-incompatible ALTER TABLE patterns, and did not support .ts/.mjs batch migrations. - Extract pure functions (transform/validate/validateStatement) into dsql-compat.ts; check-dsql-compat.ts becomes a thin CLI wrapper - Remove hash verification from _migrations table (name-based skip) - Add validation for SET/DROP NOT NULL, SET/DROP DEFAULT, DROP CONSTRAINT, TRUNCATE - Fix REFERENCES removal to preserve column definitions (strip only the REFERENCES clause, not the entire line) - Fix statement-breakpoint replacement to produce blank lines (\n\n) for correct statement splitting - Add .ts/.mjs migration support (export default async function) - Update CDK Construct to memorySize 2048 / timeout 15min - Add unit tests (43) and DSQL integration tests (16 passed, 2 skipped due to oxlint no-restricted-syntax not yet supported) - Colocate tests with source files (foo.test.ts next to foo.ts) - Add packages/db/README.md with usage, constraints, and caveats - Update AGENTS.md with ALTER TABLE constraints, unfixable error workflow, and test colocation convention
1 parent bd5d6ac commit 3675408

38 files changed

Lines changed: 2392 additions & 127 deletions

AGENTS.md

Lines changed: 8 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -55,32 +55,17 @@ DSQL constraints:
5555
- No JSON/JSONB — use TEXT
5656
- CREATE INDEX must use ASYNC keyword
5757
- 1 DDL per transaction
58+
- ALTER TABLE only supports: ADD COLUMN, RENAME COLUMN/TABLE/CONSTRAINT, SET SCHEMA, OWNER TO, and IDENTITY operations. Everything else (DROP COLUMN, ALTER COLUMN TYPE, SET/DROP NOT NULL, SET/DROP DEFAULT, DROP CONSTRAINT) requires table recreation.
5859

5960
### Database migration
6061

61-
Migrations are SQL files in `packages/db/migrations/`. The migration runner is invoked automatically during `cdk deploy` via CDK Trigger. For manual invocation, use the `MigrationCommand` from CDK outputs.
62+
See `packages/db/README.md` for full usage. Key rules:
6263

63-
Schema changes:
64-
65-
```bash
66-
# 1. Edit schema
67-
# packages/db/src/schema.ts
68-
69-
# 2. Generate migration SQL (auto-transforms for DSQL compatibility)
70-
pnpm --filter @repo/db run generate
71-
# Auto-transforms: statement-breakpoint → blank lines, CREATE INDEX → ASYNC, FK removal
72-
# Errors on unfixable patterns: ALTER COLUMN TYPE, DROP COLUMN, SERIAL
73-
74-
# 3. Review the generated SQL in packages/db/migrations/
75-
# - Add IF NOT EXISTS for idempotency
76-
77-
# 4. Apply to dev cluster
78-
pnpm --filter @repo/db run migrate
79-
80-
# 5. Commit schema + migration + snapshot together
81-
```
82-
83-
Do not use `drizzle-kit push` or `drizzle-kit migrate` — DSQL requires 1 DDL per transaction. The custom runner (`pnpm run migrate`) handles this.
64+
- `pnpm --filter @repo/db run generate` — generates and auto-transforms SQL for DSQL.
65+
- `pnpm --filter @repo/db run migrate` — applies migrations (1 DDL per transaction).
66+
- Do not use `drizzle-kit push` or `drizzle-kit migrate` — they violate DSQL's 1 DDL/transaction constraint.
67+
- When `generate` errors on unfixable patterns (DROP COLUMN, ALTER COLUMN TYPE, etc.): run `git checkout -- migrations/`, then `drizzle-kit generate --custom --name=<name>`, and write table recreation SQL or a `.ts` batch migration manually.
68+
- `.ts` / `.mjs` migrations exist for batch data migrations (e.g. table recreation with >3,000 rows). They `export default async function(client: PoolClient)`.
8469

8570
### Lambda environment
8671

@@ -105,6 +90,7 @@ Server → client push uses AppSync Events. Server-side: `sendEvent(channelName,
10590
- UI components: use [shadcn/ui](https://ui.shadcn.com/). Do not introduce alternative component libraries.
10691
- Logs: use JSON structured output.
10792
- Dependencies: esbuild and Next.js bundle everything, so only packages with native binaries needed at Lambda runtime belong in `dependencies`. Everything else goes in `devDependencies`.
93+
- Tests: colocate with source (`foo.test.ts` next to `foo.ts`). Use `.integ.test.ts` suffix for tests requiring external resources. Test runner is vitest.
10894

10995
## Do not
11096

apps/cdk/lib/constructs/dsql-migrator/index.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,9 +24,9 @@ export class DsqlMigrator extends Construct {
2424
ignoreMode: IgnoreMode.DOCKER,
2525
}),
2626
architecture: Architecture.ARM_64,
27-
timeout: Duration.minutes(5),
27+
timeout: Duration.minutes(15),
2828
environment: database.getLambdaEnvironment(),
29-
memorySize: 256,
29+
memorySize: 2048,
3030
logGroup: new LogGroup(this, 'Logs', {
3131
retention: RetentionDays.ONE_WEEK,
3232
removalPolicy: RemovalPolicy.DESTROY,

apps/cdk/test/__snapshots__/serverless-fullstack-webapp-starter-kit-without-domain.test.ts.snap

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1507,15 +1507,15 @@ exports[`Snapshot test 2`] = `
15071507
"Ref": "DsqlMigratorLogsD0AD5027",
15081508
},
15091509
},
1510-
"MemorySize": 256,
1510+
"MemorySize": 2048,
15111511
"PackageType": "Image",
15121512
"Role": {
15131513
"Fn::GetAtt": [
15141514
"DsqlMigratorHandlerServiceRole911689B1",
15151515
"Arn",
15161516
],
15171517
},
1518-
"Timeout": 300,
1518+
"Timeout": 900,
15191519
},
15201520
"Type": "AWS::Lambda::Function",
15211521
},

apps/cdk/test/__snapshots__/serverless-fullstack-webapp-starter-kit.test.ts.snap

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1423,15 +1423,15 @@ exports[`Snapshot test 2`] = `
14231423
"Ref": "DsqlMigratorLogsD0AD5027",
14241424
},
14251425
},
1426-
"MemorySize": 256,
1426+
"MemorySize": 2048,
14271427
"PackageType": "Image",
14281428
"Role": {
14291429
"Fn::GetAtt": [
14301430
"DsqlMigratorHandlerServiceRole911689B1",
14311431
"Arn",
14321432
],
14331433
},
1434-
"Timeout": 300,
1434+
"Timeout": 900,
14351435
},
14361436
"Type": "AWS::Lambda::Function",
14371437
},

drizzle-dsql-migrator-plan.md

Lines changed: 587 additions & 0 deletions
Large diffs are not rendered by default.

packages/db/README.md

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# @repo/db
2+
3+
Database schema, DSQL-compatible migration runner, and Drizzle ORM client for Aurora DSQL.
4+
5+
## Schema changes
6+
7+
```bash
8+
# 1. Edit src/schema.ts
9+
10+
# 2. Generate migration SQL (auto-transforms for DSQL)
11+
pnpm run generate
12+
13+
# 3. Review generated SQL in migrations/
14+
15+
# 4. Apply to dev cluster
16+
pnpm run migrate
17+
18+
# 5. Commit schema + migration + snapshot together
19+
```
20+
21+
`generate` runs `drizzle-kit generate` then auto-transforms the output:
22+
23+
- `CREATE INDEX``CREATE INDEX ASYNC`
24+
- Inline `REFERENCES` / `FOREIGN KEY` clauses removed
25+
- `--> statement-breakpoint` → blank line (statement separator)
26+
27+
If the generated SQL contains unfixable patterns (e.g. `DROP COLUMN`, `ALTER COLUMN TYPE`), the script errors with instructions to use `drizzle-kit generate --custom` for manual table recreation.
28+
29+
## DSQL constraints enforced
30+
31+
**At lint time** (oxlint):
32+
33+
- `serial`, `json`, `jsonb` imports from `drizzle-orm/pg-core` are blocked
34+
35+
**At generate time** (check-dsql-compat):
36+
37+
- `ALTER COLUMN TYPE`, `DROP COLUMN`, `SET/DROP NOT NULL`, `SET/DROP DEFAULT`, `DROP CONSTRAINT`, `SERIAL`, `TRUNCATE`
38+
39+
**At migration runtime** (validateStatement):
40+
41+
- All of the above, plus `CREATE INDEX` without `ASYNC`, `REFERENCES`, `FOREIGN KEY`
42+
43+
## Migration file formats
44+
45+
- **`.sql`** — Split on blank lines (`\n\n`). Each statement runs in its own `BEGIN`/`COMMIT`.
46+
- **`.ts` / `.mjs`** — Must `export default async function(client: PoolClient)`. Used for batch data migrations exceeding 3,000 rows. In Lambda, `.ts` files must be pre-transpiled to `.mjs` via the Dockerfile.
47+
48+
## When `generate` fails with unfixable errors
49+
50+
The script cannot auto-fix destructive schema changes. Follow the steps printed in the error:
51+
52+
```bash
53+
git checkout -- migrations/ # discard generated files
54+
pnpm exec drizzle-kit generate --custom --name=<migration-name> # empty migration + updated snapshot
55+
# Write table recreation SQL (.sql) or batch migration (.ts) in the generated file
56+
pnpm run migrate
57+
```
58+
59+
## Do not
60+
61+
- **Do not use `drizzle-kit push` or `drizzle-kit migrate`** — they run all DDL in one transaction. DSQL requires 1 DDL per transaction. Use `pnpm run migrate` (custom runner).
62+
- **Do not edit applied migration files** expecting re-execution — the runner skips by name, not by content hash.
63+
- **Do not mix DDL and DML in the same transaction** in `.ts` migrations — DSQL rejects this.
64+
65+
## `.ts` migration caveats
66+
67+
- Lambda execution limit is 15 minutes. Migrations exceeding this need Step Functions (out of scope).
68+
- DSQL transaction limit is 3,000 rows. Batch inserts must commit in chunks.
69+
- In Lambda, `.ts` files are pre-transpiled to `.mjs` in the Dockerfile. The runner picks up `.mjs` at runtime.
70+
71+
## oxlint limitation
72+
73+
`no-restricted-syntax` (`.references()` detection in schema files) is configured in `oxlintrc.json` but **not yet supported by oxlint** (as of v1.56.0). The SQL-level `REFERENCES` / `FOREIGN KEY` validation in `check-dsql-compat` and `migrate` serves as the fallback.
74+
75+
## Environment
76+
77+
Create `.env` in this package (gitignored):
78+
79+
```
80+
DSQL_ENDPOINT=<cluster>.dsql.<region>.on.aws
81+
AWS_REGION=<region>
82+
```
83+
84+
Or use `scripts/dsql.sh create --region <region>` from the repo root to provision a dev cluster and write `.env` automatically.
85+
86+
## Testing
87+
88+
```bash
89+
pnpm run test:unit # unit tests (no DB required)
90+
pnpm run test:integ # integration tests (DSQL cluster required)
91+
DSQL_ENDPOINT=... AWS_REGION=... \
92+
pnpm run test:integ # explicit env
93+
```

packages/db/migrations/meta/0001_snapshot.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,4 +94,4 @@
9494
"schemas": {},
9595
"tables": {}
9696
}
97-
}
97+
}

packages/db/package.json

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,9 @@
1111
"scripts": {
1212
"generate": "drizzle-kit generate && tsx src/check-dsql-compat.ts",
1313
"migrate": "tsx --env-file=.env src/cli.ts",
14+
"test:unit": "vitest run --exclude '**/*.integ.test.ts'",
15+
"test:integ": "vitest run --config vitest.integ.config.ts",
16+
"test:watch": "vitest --exclude '**/*.integ.test.ts'",
1417
"format": "oxfmt --write .",
1518
"format:ci": "oxfmt --check .",
1619
"lint": "oxlint --config ../../oxlintrc.json --fix .",
@@ -27,6 +30,7 @@
2730
"@types/pg": "^8",
2831
"drizzle-kit": "^0.31.10",
2932
"tsx": "^4",
30-
"typescript": "^5"
33+
"typescript": "^5",
34+
"vitest": "^3"
3135
}
3236
}
Lines changed: 15 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,66 +1,43 @@
1-
// Transform drizzle-kit generated SQL for DSQL compatibility, then validate.
2-
// Automatically fixes what can be fixed, errors on what cannot.
3-
//
4-
// Auto-transforms:
5-
// - "--> statement-breakpoint" → blank line (runner splits on \n\n)
6-
// - "CREATE INDEX" → "CREATE INDEX ASYNC"
7-
// - Removes REFERENCES / FOREIGN KEY clauses
8-
//
9-
// Errors (cannot auto-fix):
10-
// - ALTER COLUMN TYPE
11-
// - DROP COLUMN
12-
// - SERIAL types
1+
#!/usr/bin/env tsx
2+
// CLI script: transform drizzle-kit generated SQL for DSQL compatibility, then validate.
3+
// Pure logic lives in dsql-compat.ts.
134
import fs from 'node:fs';
145
import path from 'node:path';
156
import { fileURLToPath } from 'node:url';
7+
import { transformSql, validateSql } from './dsql-compat';
168

179
const __dirname = path.dirname(fileURLToPath(import.meta.url));
1810
const migrationsDir = path.join(__dirname, '..', 'migrations');
1911

2012
const files = fs.readdirSync(migrationsDir).filter((f) => f.endsWith('.sql'));
2113
let transformed = 0;
22-
let errors = 0;
14+
let hasErrors = false;
2315

2416
for (const file of files) {
2517
const filePath = path.join(migrationsDir, file);
2618
const original = fs.readFileSync(filePath, 'utf8');
27-
let sql = original;
28-
29-
// Auto-transform: statement-breakpoint → blank line
30-
sql = sql.replace(/--> statement-breakpoint\n/g, '\n');
31-
32-
// Auto-transform: CREATE INDEX → CREATE INDEX ASYNC (skip if already ASYNC)
33-
sql = sql.replace(/CREATE\s+INDEX(?!\s+ASYNC)/gi, 'CREATE INDEX ASYNC');
34-
35-
// Auto-transform: remove lines with REFERENCES or FOREIGN KEY
36-
sql = sql
37-
.split('\n')
38-
.filter((line) => !/REFERENCES\s+/i.test(line) && !/FOREIGN\s+KEY/i.test(line))
39-
.join('\n');
19+
const sql = transformSql(original);
4020

4121
if (sql !== original) {
4222
fs.writeFileSync(filePath, sql);
4323
console.log(`TRANSFORMED: ${file}`);
4424
transformed++;
4525
}
4626

47-
// Validate: errors that cannot be auto-fixed
48-
const unfixable: { pattern: RegExp; message: string }[] = [
49-
{ pattern: /ALTER\s+.*\s+TYPE\s+/i, message: 'ALTER COLUMN TYPE not supported' },
50-
{ pattern: /DROP\s+COLUMN/i, message: 'DROP COLUMN not supported' },
51-
{ pattern: /\bSERIAL\b/i, message: 'SERIAL types not supported (use UUID)' },
52-
];
53-
for (const { pattern, message } of unfixable) {
54-
if (pattern.test(sql)) {
55-
console.error(`ERROR: ${file}${message}`);
56-
errors++;
27+
const errors = validateSql(sql);
28+
if (errors.length > 0) {
29+
for (const e of errors) {
30+
console.error(`ERROR: ${file}${e.pattern}: ${e.message}`);
5731
}
32+
hasErrors = true;
5833
}
5934
}
6035

6136
if (transformed > 0) console.log(`\n${transformed} file(s) auto-transformed for DSQL compatibility.`);
62-
if (errors > 0) {
63-
console.error(`\n${errors} unfixable error(s). These require manual migration scripts.`);
37+
if (hasErrors) {
38+
console.error(
39+
`\nUnfixable error(s) detected. Steps:\n 1. Run: git checkout -- migrations/\n 2. Run: pnpm --filter @repo/db exec drizzle-kit generate --custom --name=<migration-name>\n 3. Write table recreation SQL/TS in the generated file\n 4. Run: pnpm --filter @repo/db run migrate`,
40+
);
6441
process.exit(1);
6542
}
6643
console.log('All migration files DSQL-compatible.');
Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
import { describe, expect, test, beforeAll, afterAll } from 'vitest';
2+
import fs from 'node:fs';
3+
import path from 'node:path';
4+
import os from 'node:os';
5+
import { transformSql, validateSql } from './dsql-compat';
6+
7+
let tmpDir: string;
8+
9+
beforeAll(() => {
10+
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'compat-test-'));
11+
});
12+
13+
afterAll(() => {
14+
fs.rmSync(tmpDir, { recursive: true });
15+
});
16+
17+
function runCheckCompat(migrationsDir: string): { exitCode: number; errors: string[] } {
18+
const files = fs.readdirSync(migrationsDir).filter((f) => f.endsWith('.sql'));
19+
let transformed = 0;
20+
const errors: string[] = [];
21+
22+
for (const file of files) {
23+
const filePath = path.join(migrationsDir, file);
24+
const original = fs.readFileSync(filePath, 'utf8');
25+
const sql = transformSql(original);
26+
if (sql !== original) {
27+
fs.writeFileSync(filePath, sql);
28+
transformed++;
29+
}
30+
for (const e of validateSql(sql)) {
31+
errors.push(`${file}: ${e.pattern}`);
32+
}
33+
}
34+
35+
return { exitCode: errors.length > 0 ? 1 : 0, errors };
36+
}
37+
38+
describe('check-dsql-compat integration', () => {
39+
test('C1: transforms drizzle-kit output', () => {
40+
const sqlFile = path.join(tmpDir, '0001.sql');
41+
fs.writeFileSync(
42+
sqlFile,
43+
'CREATE TABLE "T" ("id" text PRIMARY KEY, "userId" text NOT NULL REFERENCES "User"("id"));--> statement-breakpoint\nCREATE INDEX "T_userId_idx" ON "T" ("userId");',
44+
);
45+
46+
const { exitCode } = runCheckCompat(tmpDir);
47+
expect(exitCode).toBe(0);
48+
49+
const result = fs.readFileSync(sqlFile, 'utf8');
50+
expect(result).not.toContain('statement-breakpoint');
51+
expect(result).not.toContain('REFERENCES');
52+
expect(result).toContain('CREATE INDEX ASYNC');
53+
});
54+
55+
test('C2: no-transform file unchanged', () => {
56+
// Clean up previous files
57+
for (const f of fs.readdirSync(tmpDir)) fs.unlinkSync(path.join(tmpDir, f));
58+
59+
const sqlFile = path.join(tmpDir, '0002.sql');
60+
const content = 'ALTER TABLE "T" ADD COLUMN "name" text;\n';
61+
fs.writeFileSync(sqlFile, content);
62+
63+
runCheckCompat(tmpDir);
64+
65+
expect(fs.readFileSync(sqlFile, 'utf8')).toBe(content);
66+
});
67+
68+
test('C3: unfixable pattern returns exit 1', () => {
69+
// Clean up previous files
70+
for (const f of fs.readdirSync(tmpDir)) fs.unlinkSync(path.join(tmpDir, f));
71+
72+
const sqlFile = path.join(tmpDir, '0003.sql');
73+
fs.writeFileSync(sqlFile, 'ALTER TABLE "T" ALTER COLUMN "c" TYPE varchar(100);\n');
74+
75+
const { exitCode, errors } = runCheckCompat(tmpDir);
76+
expect(exitCode).toBe(1);
77+
expect(errors.some((e) => e.includes('ALTER COLUMN TYPE'))).toBe(true);
78+
});
79+
});

0 commit comments

Comments
 (0)