Skip to content

Commit 343fb51

Browse files
pyramationpyramation
committed
docs: add code-generation skill, expand fixtures skill and AGENTS.md with full script reference
1 parent 700445e commit 343fb51

3 files changed

Lines changed: 246 additions & 33 deletions

File tree

‎.agents/skills/code-generation/SKILL.md‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# Code Generation & Type Inference
2+
3+
This skill documents the code generation pipelines in pgsql-parser: protobuf-based TypeScript generation, type inference from SQL fixtures, and keyword list generation from PostgreSQL source.
4+
5+
## Overview
6+
7+
Several packages generate TypeScript code from external sources rather than being hand-written. These generated files should **not** be edited by hand — instead, re-run the generation scripts after changing inputs.
8+
9+
## 1. Protobuf-Based Code Generation (`build:proto`)
10+
11+
Four packages generate TypeScript from the PostgreSQL protobuf definition at `__fixtures__/proto/17-latest.proto`:
12+
13+
| Package | Script | What it generates |
14+
|---------|--------|-------------------|
15+
| `@pgsql/utils` | `npm run build:proto` | AST helper functions (`src/`), wrapped helpers (`wrapped.ts`), runtime schema (`runtime-schema.ts`) |
16+
| `@pgsql/traverse` | `npm run build:proto` | Visitor-pattern traversal utilities |
17+
| `@pgsql/transform` | `npm run build:proto` | Multi-version AST transformer utilities |
18+
| `pg-ast` | `npm run build:proto` | Low-level AST type helpers |
19+
20+
Each package has a `scripts/pg-proto-parser.ts` that configures `PgProtoParser` with package-specific options (which features to enable, output paths, type sources).
21+
22+
**When to re-run:** After updating `__fixtures__/proto/17-latest.proto` (e.g., when upgrading to a new PostgreSQL version).
23+
24+
```bash
25+
# Re-generate for a specific package
26+
cd packages/utils && npm run build:proto
27+
28+
# Or build all (build:proto runs as part of build)
29+
pnpm run build
30+
```
31+
32+
Note: `build:proto` is called automatically as part of `npm run build` in these packages, so a full `pnpm run build` from root covers everything.
33+
34+
### Proto-Parser Test Utils
35+
36+
The `pg-proto-parser` package also has its own generation script:
37+
38+
```bash
39+
cd packages/proto-parser && npm run generate:test-utils
40+
```
41+
42+
This generates test utility functions from a `13-latest.proto` fixture into `test-utils/utils/`.
43+
44+
## 2. Type Inference & Narrowed Type Generation (`pgsql-types`)
45+
46+
The `pgsql-types` package has a two-step pipeline that discovers actual AST usage patterns from SQL fixtures and generates narrowed TypeScript types:
47+
48+
### Step 1: Infer field metadata
49+
50+
```bash
51+
cd packages/pgsql-types && npm run infer
52+
```
53+
54+
Runs `scripts/infer-field-metadata.ts`:
55+
- Reads all `.sql` files from `__fixtures__/kitchen-sink/` and `__fixtures__/postgres/`
56+
- Parses each statement and walks the AST
57+
- For every `Node`-typed field, records which concrete node tags actually appear
58+
- Writes `src/field-metadata.json` with nullable/tag/array info per field
59+
60+
### Step 2: Generate narrowed types
61+
62+
```bash
63+
cd packages/pgsql-types && npm run generate
64+
```
65+
66+
Runs `scripts/generate-types.ts`:
67+
- Reads `src/field-metadata.json` (must run `infer` first)
68+
- Generates `src/types.ts` with narrowed union types instead of generic `Node`
69+
- Example: instead of `whereClause?: Node`, generates `whereClause?: { BoolExpr: BoolExpr } | { A_Expr: A_Expr } | ...`
70+
71+
**When to re-run:** After adding new SQL fixtures (which may introduce new node type combinations) or after updating the runtime schema.
72+
73+
Note: `infer` is called automatically as part of `npm run build` in pgsql-types.
74+
75+
## 3. Keyword List Generation (`@pgsql/quotes`)
76+
77+
```bash
78+
cd packages/quotes && npm run keywords -- /path/to/postgres/src/include/parser/kwlist.h
79+
```
80+
81+
Runs `scripts/keywords.ts`:
82+
- Reads PostgreSQL's `kwlist.h` header file (from a local PostgreSQL source checkout)
83+
- Parses `PG_KEYWORD(...)` macros to extract keywords and their categories
84+
- Generates `src/kwlist.ts` with typed keyword sets (RESERVED, UNRESERVED, COL_NAME, TYPE_FUNC_NAME)
85+
86+
**When to re-run:** When upgrading to a new PostgreSQL version that adds/removes/reclassifies keywords.
87+
88+
**Requires:** A local checkout of the PostgreSQL source code to provide the `kwlist.h` file. The script will prompt for the path interactively if not provided as an argument.
89+
90+
## 4. Version-Specific Deparser Generation (`pgsql-deparser`)
91+
92+
```bash
93+
cd packages/deparser && ts-node scripts/generate-version-deparsers.ts
94+
```
95+
96+
Generates `versions/{13,14,15,16}/src/index.ts` files that wire up version-specific AST transformers (e.g., `PG13ToPG17Transformer`) to the main v17 deparser. This allows deparsing ASTs from older PostgreSQL versions.
97+
98+
**When to re-run:** When adding support for a new PostgreSQL version or changing the transformer class names.
99+
100+
## Quick Reference
101+
102+
| Workflow | Command | Input | Output |
103+
|----------|---------|-------|--------|
104+
| Proto codegen (all) | `pnpm run build` | `__fixtures__/proto/17-latest.proto` | Generated TS in each package's `src/` |
105+
| Proto codegen (one pkg) | `cd packages/<pkg> && npm run build:proto` | Same | Same |
106+
| Type inference | `cd packages/pgsql-types && npm run infer` | `__fixtures__/kitchen-sink/**/*.sql` | `src/field-metadata.json` |
107+
| Type generation | `cd packages/pgsql-types && npm run generate` | `src/field-metadata.json` | `src/types.ts` |
108+
| Keyword generation | `cd packages/quotes && npm run keywords -- <kwlist.h>` | PostgreSQL `kwlist.h` | `src/kwlist.ts` |
109+
| Version deparsers | `cd packages/deparser && ts-node scripts/generate-version-deparsers.ts` | Transformer configs | `versions/*/src/index.ts` |

‎.agents/skills/testing-fixtures/SKILL.md‎

Lines changed: 53 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ __fixtures__/
2424
plpgsql/ # PL/pgSQL fixture SQL files
2525
plpgsql-generated/
2626
generated.json # Auto-generated: PL/pgSQL fixtures
27+
2728
packages/deparser/
2829
__tests__/
2930
kitchen-sink/ # Auto-generated test files from kitchen-sink fixtures
@@ -36,6 +37,13 @@ packages/deparser/
3637
test-utils/
3738
index.ts # Core test utilities (expectParseDeparse, FixtureTestUtils, TestUtils)
3839
PrettyTest.ts # Pretty-print test helper
40+
packages/plpgsql-deparser/
41+
scripts/
42+
make-fixtures.ts # Extracts PL/pgSQL statements -> plpgsql-generated/generated.json
43+
packages/transform/
44+
scripts/
45+
make-kitchen-sink.ts # Generates transform kitchen-sink tests
46+
test-ast.ts # AST round-trip validation for transform
3947
```
4048

4149
## How Fixtures Work
@@ -136,29 +144,61 @@ This generates two tests per case (pretty and non-pretty) and uses Jest snapshot
136144
3. Run `npx jest` to verify all tests pass
137145
4. Commit the `.sql` file, `generated.json`, and any new generated test files in `__tests__/kitchen-sink/`
138146

147+
## PL/pgSQL Fixtures (`plpgsql-deparser`)
148+
149+
The PL/pgSQL deparser has its own fixture pipeline:
150+
151+
```bash
152+
cd packages/plpgsql-deparser && npm run fixtures
153+
```
154+
155+
This runs `scripts/make-fixtures.ts` which:
156+
- Reads SQL files from `__fixtures__/plpgsql/`
157+
- Parses with `libpg-query`, filters to `CreateFunctionStmt` (language plpgsql) and `DoStmt`
158+
- Validates each statement parses as PL/pgSQL via `parsePlPgSQLSync()`
159+
- Writes to `__fixtures__/plpgsql-generated/generated.json`
160+
161+
## Transform Kitchen-Sink (`transform`)
162+
163+
The transform package has its own kitchen-sink and AST test scripts:
164+
165+
```bash
166+
cd packages/transform
167+
npm run kitchen-sink # generate transform-specific kitchen-sink tests
168+
npm run test:ast # run AST round-trip validation
169+
```
170+
139171
## Package Scripts Reference
140172

141-
From `packages/deparser/package.json`:
173+
### `packages/deparser` (primary fixture pipeline)
142174

143175
| Script | Command | Description |
144176
|--------|---------|-------------|
145177
| `fixtures` | `ts-node scripts/make-fixtures.ts` | Regenerate `generated.json` |
146178
| `fixtures:kitchen-sink` | `ts-node scripts/make-kitchen-sink.ts` | Regenerate kitchen-sink test files |
147179
| `kitchen-sink` | `npm run fixtures && npm run fixtures:kitchen-sink` | Both steps combined |
148-
| `fixtures:ast` | `ts-node scripts/make-fixtures-ast.ts` | Generate AST JSON fixtures (from legacy) |
149-
| `fixtures:sql` | `ts-node scripts/make-fixtures-sql.ts` | Generate SQL fixtures via native deparse (from legacy) |
150-
| `fixtures:upstream-diff` | `ts-node scripts/make-upstream-diff.ts` | Generate diff of upstream vs deparsed |
180+
| `fixtures:ast` | `ts-node scripts/make-fixtures-ast.ts` | Generate AST JSON fixtures |
181+
| `fixtures:sql` | `ts-node scripts/make-fixtures-sql.ts` | Generate SQL fixtures via native deparse |
182+
| `fixtures:upstream-diff` | `ts-node scripts/make-upstream-diff.ts` | Generate diff comparing upstream (libpg-query) vs our deparser output |
151183
| `test` | `jest` | Run all tests |
152184
| `test:watch` | `jest --watch` | Run tests in watch mode |
153185

154-
## Build & Lint
186+
### `packages/plpgsql-deparser`
155187

156-
```bash
157-
# From repo root
158-
pnpm install
159-
pnpm run build
188+
| Script | Command | Description |
189+
|--------|---------|-------------|
190+
| `fixtures` | `ts-node scripts/make-fixtures.ts` | Extract PL/pgSQL fixtures to `plpgsql-generated/generated.json` |
160191

161-
# From packages/deparser
162-
npm run build # tsc (CJS + ESM) + asset copy
163-
npm run lint # eslint --fix
164-
```
192+
### `packages/transform`
193+
194+
| Script | Command | Description |
195+
|--------|---------|-------------|
196+
| `kitchen-sink` | `ts-node scripts/make-kitchen-sink.ts` | Generate transform kitchen-sink tests |
197+
| `test:ast` | `ts-node scripts/test-ast.ts` | AST round-trip validation |
198+
199+
### `packages/parser`
200+
201+
| Script | Command | Description |
202+
|--------|---------|-------------|
203+
| `prepare-versions` | `ts-node scripts/prepare-versions.ts` | Generate version-specific sub-packages from `config/versions.json` |
204+
| `test:ast` | `ts-node scripts/test-ast.ts` | AST round-trip validation |

‎AGENTS.md‎

Lines changed: 84 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -12,47 +12,111 @@ A pnpm monorepo for PostgreSQL AST parsing, deparsing, and code generation. All
1212
| `pgsql-deparser` | `packages/deparser` | Convert AST back to SQL (pure TypeScript) |
1313
| `plpgsql-parser` | `packages/plpgsql-parser` | Parse PL/pgSQL to AST |
1414
| `plpgsql-deparser` | `packages/plpgsql-deparser` | Convert PL/pgSQL AST back to SQL |
15-
| `@pgsql/types` | `packages/pgsql-types` | TypeScript type definitions for PostgreSQL AST nodes |
15+
| `pgsql-types` | `packages/pgsql-types` | Narrowed TypeScript types inferred from SQL fixtures |
16+
| `@pgsql/types` | (published from proto-parser codegen) | Core TypeScript type definitions for PostgreSQL AST nodes |
1617
| `@pgsql/utils` | `packages/utils` | Type-safe AST node creation utilities |
1718
| `@pgsql/traverse` | `packages/traverse` | Visitor-pattern AST traversal |
18-
| `@pgsql/transform` | `packages/transform` | Multi-version AST transformer (PG 13→17) |
19-
| `@pgsql/quotes` | `packages/quotes` | SQL identifier/string quoting utilities |
19+
| `@pgsql/transform` | `packages/transform` | Multi-version AST transformer (PG 13-17) |
20+
| `@pgsql/quotes` | `packages/quotes` | SQL identifier/string quoting and keyword classification |
2021
| `@pgsql/cli` | `packages/pgsql-cli` | CLI tool for parse/deparse operations |
2122
| `pg-proto-parser` | `packages/proto-parser` | Generate TypeScript from PostgreSQL protobuf definitions |
22-
| `pg-ast` | `packages/pg-ast` | Low-level AST types |
23+
| `pg-ast` | `packages/pg-ast` | Low-level AST helpers |
2324

2425
## Setup
2526

2627
```bash
2728
pnpm install
28-
pnpm run build # builds all packages
29+
pnpm run build # builds all packages (includes code generation)
2930
pnpm run test # runs all package tests
3031
pnpm run lint # lints all packages
3132
```
3233

33-
## Per-Package Commands
34+
## Skills
3435

35-
Each package supports:
36-
- `npm run build` — TypeScript compilation (CJS + ESM) + asset copy
37-
- `npm run test` — Jest tests
38-
- `npm run lint` — ESLint with auto-fix
39-
- `npm run test:watch` — Jest in watch mode
36+
Detailed workflow documentation lives in `.agents/skills/`:
4037

41-
## Testing
38+
| Skill | Path | Covers |
39+
|-------|------|--------|
40+
| **Testing & Fixtures** | `.agents/skills/testing-fixtures/SKILL.md` | Fixture-based testing pipeline, adding new test fixtures, kitchen-sink workflow, PL/pgSQL fixtures, transform tests |
41+
| **Code Generation** | `.agents/skills/code-generation/SKILL.md` | Protobuf codegen (`build:proto`), type inference/generation (`pgsql-types`), keyword generation (`@pgsql/quotes`), version-specific deparsers |
4242

43-
Tests use Jest. The deparser packages use a **fixture-based testing system** — see `.agents/skills/testing-fixtures/SKILL.md` for full details.
43+
## Root Scripts
4444

45-
Quick reference:
46-
```bash
47-
cd packages/deparser
48-
npm run kitchen-sink # regenerate fixtures + test files
49-
npx jest # run all tests
50-
```
45+
| Script | Command | Description |
46+
|--------|---------|-------------|
47+
| `build` | `pnpm -r run build` | Build all packages (TypeScript compilation + code generation) |
48+
| `clean` | `pnpm -r run clean` | Clean all package `dist/` directories |
49+
| `test` | `pnpm -r run test` | Run Jest tests across all packages |
50+
| `lint` | `pnpm -r run lint` | ESLint with auto-fix across all packages |
51+
| `deps` | `pnpm up -r -i -L` | Interactive dependency update across workspace |
52+
| `bump-versions` | `ts-node scripts/bump-versions.ts` | Interactive version bumper — fetches latest npm versions, prompts for bump type per PG version |
53+
| `update-workspace` | `makage update-workspace` | Update pnpm workspace configuration |
54+
55+
## Per-Package Standard Scripts
56+
57+
Every package supports these scripts:
58+
59+
| Script | Command | Description |
60+
|--------|---------|-------------|
61+
| `build` | `tsc && tsc -p tsconfig.esm.json` + extras | TypeScript compilation (CJS + ESM) + asset copy. Some packages run `build:proto` first |
62+
| `build:dev` | Same as build but with `--declarationMap` | Development build with source maps for declaration files |
63+
| `clean` | `makage clean dist` | Remove `dist/` directory |
64+
| `copy` | `makage assets` | Copy non-TS assets to `dist/` |
65+
| `lint` | `eslint . --fix` | ESLint with auto-fix |
66+
| `test` | `jest` | Run Jest tests |
67+
| `test:watch` | `jest --watch` | Run Jest in watch mode |
68+
| `prepublishOnly` | `npm run build` | Ensure build before publish |
69+
70+
## Package-Specific Scripts
71+
72+
### Fixture & Testing Scripts (see testing-fixtures skill)
73+
74+
| Package | Script | Description |
75+
|---------|--------|-------------|
76+
| `deparser` | `npm run kitchen-sink` | Regenerate fixtures + test files (most common command) |
77+
| `deparser` | `npm run fixtures` | Regenerate `generated.json` only |
78+
| `deparser` | `npm run fixtures:kitchen-sink` | Regenerate test files only |
79+
| `deparser` | `npm run fixtures:ast` | Generate AST JSON fixtures |
80+
| `deparser` | `npm run fixtures:sql` | Generate SQL fixtures via native deparse |
81+
| `deparser` | `npm run fixtures:upstream-diff` | Compare upstream (libpg-query) vs our deparser output |
82+
| `plpgsql-deparser` | `npm run fixtures` | Extract PL/pgSQL fixtures |
83+
| `transform` | `npm run kitchen-sink` | Generate transform kitchen-sink tests |
84+
| `transform` | `npm run test:ast` | AST round-trip validation |
85+
| `parser` | `npm run test:ast` | AST round-trip validation |
86+
87+
### Code Generation Scripts (see code-generation skill)
88+
89+
| Package | Script | Description |
90+
|---------|--------|-------------|
91+
| `utils` | `npm run build:proto` | Generate AST helpers from protobuf |
92+
| `traverse` | `npm run build:proto` | Generate traversal utilities from protobuf |
93+
| `transform` | `npm run build:proto` | Generate transformer utilities from protobuf |
94+
| `pg-ast` | `npm run build:proto` | Generate AST types from protobuf |
95+
| `pgsql-types` | `npm run infer` | Infer field metadata from SQL fixtures |
96+
| `pgsql-types` | `npm run generate` | Generate narrowed types from metadata |
97+
| `proto-parser` | `npm run generate:test-utils` | Generate test utilities from protobuf |
98+
| `quotes` | `npm run keywords` | Generate keyword list from PostgreSQL `kwlist.h` |
99+
100+
### Version Management
101+
102+
| Package | Script | Description |
103+
|---------|--------|-------------|
104+
| `parser` | `npm run prepare-versions` | Generate version-specific sub-packages from `config/versions.json` |
105+
| (root) | `npm run bump-versions` | Interactive CLI to bump `pgsql-parser` / `pgsql-deparser` versions per PG version |
106+
107+
Version configuration lives in `config/versions.json` — maps PG versions (13-17) to their `libpg-query`, `pgsql-parser`, `pgsql-deparser`, and `@pgsql/types` versions plus npm dist-tags.
108+
109+
### CLI Development
110+
111+
| Package | Script | Description |
112+
|---------|--------|-------------|
113+
| `pgsql-cli` | `npm run dev` | Run CLI in dev mode via `ts-node src/index` |
51114

52115
## Code Conventions
53116

54117
- TypeScript throughout, compiled to both CJS and ESM
55118
- `@pgsql/types` provides all AST node types — use them for type safety
56119
- `@pgsql/quotes` handles SQL identifier quoting — use `QuoteUtils` methods
57120
- Test files go in `__tests__/` within each package
58-
- Fixture SQL files go in `__fixtures__/kitchen-sink/` (see skills for details)
121+
- Fixture SQL files go in `__fixtures__/kitchen-sink/` (see testing-fixtures skill)
122+
- Generated files (marked `DO NOT EDIT BY HAND`) should be regenerated via scripts, not edited manually

0 commit comments

Comments
 (0)