Skip to content

Commit 0266000

Browse files
pyramationpyramation
authored
Merge pull request #289 from constructive-io/devin/1775200098-fix-extract-epoch-syntax
fix: strip quotes and uppercase EXTRACT field name in deparser
2 parents cdd732b + 343fb51 commit 0266000

7 files changed

Lines changed: 505 additions & 1 deletion

File tree

‎.agents/skills/code-generation/SKILL.md‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# Code Generation & Type Inference
2+
3+
This skill documents the code generation pipelines in pgsql-parser: protobuf-based TypeScript generation, type inference from SQL fixtures, and keyword list generation from PostgreSQL source.
4+
5+
## Overview
6+
7+
Several packages generate TypeScript code from external sources rather than being hand-written. These generated files should **not** be edited by hand — instead, re-run the generation scripts after changing inputs.
8+
9+
## 1. Protobuf-Based Code Generation (`build:proto`)
10+
11+
Four packages generate TypeScript from the PostgreSQL protobuf definition at `__fixtures__/proto/17-latest.proto`:
12+
13+
| Package | Script | What it generates |
14+
|---------|--------|-------------------|
15+
| `@pgsql/utils` | `npm run build:proto` | AST helper functions (`src/`), wrapped helpers (`wrapped.ts`), runtime schema (`runtime-schema.ts`) |
16+
| `@pgsql/traverse` | `npm run build:proto` | Visitor-pattern traversal utilities |
17+
| `@pgsql/transform` | `npm run build:proto` | Multi-version AST transformer utilities |
18+
| `pg-ast` | `npm run build:proto` | Low-level AST type helpers |
19+
20+
Each package has a `scripts/pg-proto-parser.ts` that configures `PgProtoParser` with package-specific options (which features to enable, output paths, type sources).
21+
22+
**When to re-run:** After updating `__fixtures__/proto/17-latest.proto` (e.g., when upgrading to a new PostgreSQL version).
23+
24+
```bash
25+
# Re-generate for a specific package
26+
cd packages/utils && npm run build:proto
27+
28+
# Or build all (build:proto runs as part of build)
29+
pnpm run build
30+
```
31+
32+
Note: `build:proto` is called automatically as part of `npm run build` in these packages, so a full `pnpm run build` from root covers everything.
33+
34+
### Proto-Parser Test Utils
35+
36+
The `pg-proto-parser` package also has its own generation script:
37+
38+
```bash
39+
cd packages/proto-parser && npm run generate:test-utils
40+
```
41+
42+
This generates test utility functions from a `13-latest.proto` fixture into `test-utils/utils/`.
43+
44+
## 2. Type Inference & Narrowed Type Generation (`pgsql-types`)
45+
46+
The `pgsql-types` package has a two-step pipeline that discovers actual AST usage patterns from SQL fixtures and generates narrowed TypeScript types:
47+
48+
### Step 1: Infer field metadata
49+
50+
```bash
51+
cd packages/pgsql-types && npm run infer
52+
```
53+
54+
Runs `scripts/infer-field-metadata.ts`:
55+
- Reads all `.sql` files from `__fixtures__/kitchen-sink/` and `__fixtures__/postgres/`
56+
- Parses each statement and walks the AST
57+
- For every `Node`-typed field, records which concrete node tags actually appear
58+
- Writes `src/field-metadata.json` with nullable/tag/array info per field
59+
60+
### Step 2: Generate narrowed types
61+
62+
```bash
63+
cd packages/pgsql-types && npm run generate
64+
```
65+
66+
Runs `scripts/generate-types.ts`:
67+
- Reads `src/field-metadata.json` (must run `infer` first)
68+
- Generates `src/types.ts` with narrowed union types instead of generic `Node`
69+
- Example: instead of `whereClause?: Node`, generates `whereClause?: { BoolExpr: BoolExpr } | { A_Expr: A_Expr } | ...`
70+
71+
**When to re-run:** After adding new SQL fixtures (which may introduce new node type combinations) or after updating the runtime schema.
72+
73+
Note: `infer` is called automatically as part of `npm run build` in pgsql-types.
74+
75+
## 3. Keyword List Generation (`@pgsql/quotes`)
76+
77+
```bash
78+
cd packages/quotes && npm run keywords -- /path/to/postgres/src/include/parser/kwlist.h
79+
```
80+
81+
Runs `scripts/keywords.ts`:
82+
- Reads PostgreSQL's `kwlist.h` header file (from a local PostgreSQL source checkout)
83+
- Parses `PG_KEYWORD(...)` macros to extract keywords and their categories
84+
- Generates `src/kwlist.ts` with typed keyword sets (RESERVED, UNRESERVED, COL_NAME, TYPE_FUNC_NAME)
85+
86+
**When to re-run:** When upgrading to a new PostgreSQL version that adds/removes/reclassifies keywords.
87+
88+
**Requires:** A local checkout of the PostgreSQL source code to provide the `kwlist.h` file. The script will prompt for the path interactively if not provided as an argument.
89+
90+
## 4. Version-Specific Deparser Generation (`pgsql-deparser`)
91+
92+
```bash
93+
cd packages/deparser && ts-node scripts/generate-version-deparsers.ts
94+
```
95+
96+
Generates `versions/{13,14,15,16}/src/index.ts` files that wire up version-specific AST transformers (e.g., `PG13ToPG17Transformer`) to the main v17 deparser. This allows deparsing ASTs from older PostgreSQL versions.
97+
98+
**When to re-run:** When adding support for a new PostgreSQL version or changing the transformer class names.
99+
100+
## Quick Reference
101+
102+
| Workflow | Command | Input | Output |
103+
|----------|---------|-------|--------|
104+
| Proto codegen (all) | `pnpm run build` | `__fixtures__/proto/17-latest.proto` | Generated TS in each package's `src/` |
105+
| Proto codegen (one pkg) | `cd packages/<pkg> && npm run build:proto` | Same | Same |
106+
| Type inference | `cd packages/pgsql-types && npm run infer` | `__fixtures__/kitchen-sink/**/*.sql` | `src/field-metadata.json` |
107+
| Type generation | `cd packages/pgsql-types && npm run generate` | `src/field-metadata.json` | `src/types.ts` |
108+
| Keyword generation | `cd packages/quotes && npm run keywords -- <kwlist.h>` | PostgreSQL `kwlist.h` | `src/kwlist.ts` |
109+
| Version deparsers | `cd packages/deparser && ts-node scripts/generate-version-deparsers.ts` | Transformer configs | `versions/*/src/index.ts` |

‎.agents/skills/testing-fixtures/SKILL.md‎

Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
# pgsql-parser Fixtures & Testing System
2+
3+
This skill documents how the fixture-based testing pipeline works in pgsql-parser and how to add new test fixtures.
4+
5+
## Architecture Overview
6+
7+
The testing pipeline has three layers:
8+
9+
1. **SQL fixture files** — Raw SQL statements in `__fixtures__/kitchen-sink/`
10+
2. **Generated fixture JSON** — `__fixtures__/generated/generated.json` (individual statements keyed by path)
11+
3. **Generated test files** — `packages/deparser/__tests__/kitchen-sink/*.test.ts`
12+
13+
## Directory Structure
14+
15+
```
16+
__fixtures__/
17+
kitchen-sink/ # Source SQL fixture files
18+
latest/postgres/ # PostgreSQL regression test extracts
19+
misc/ # Miscellaneous test cases (issues, features)
20+
original/ # Original upstream PostgreSQL test files
21+
pretty/ # Pretty-print specific test cases
22+
generated/
23+
generated.json # Auto-generated: individual statements keyed by relative path
24+
plpgsql/ # PL/pgSQL fixture SQL files
25+
plpgsql-generated/
26+
generated.json # Auto-generated: PL/pgSQL fixtures
27+
28+
packages/deparser/
29+
__tests__/
30+
kitchen-sink/ # Auto-generated test files from kitchen-sink fixtures
31+
misc/ # Hand-written test files for specific features
32+
pretty/ # Pretty-print tests using PrettyTest utility
33+
scripts/
34+
make-fixtures.ts # Splits SQL files -> generated.json
35+
make-kitchen-sink.ts # Generates kitchen-sink test files
36+
statement-splitter.ts # Extracts individual statements from multi-statement SQL
37+
test-utils/
38+
index.ts # Core test utilities (expectParseDeparse, FixtureTestUtils, TestUtils)
39+
PrettyTest.ts # Pretty-print test helper
40+
packages/plpgsql-deparser/
41+
scripts/
42+
make-fixtures.ts # Extracts PL/pgSQL statements -> plpgsql-generated/generated.json
43+
packages/transform/
44+
scripts/
45+
make-kitchen-sink.ts # Generates transform kitchen-sink tests
46+
test-ast.ts # AST round-trip validation for transform
47+
```
48+
49+
## How Fixtures Work
50+
51+
### Step 1: Write SQL fixture file
52+
53+
Create a `.sql` file in `__fixtures__/kitchen-sink/`. Organize by category:
54+
- `misc/` for bug fixes, feature-specific tests, issue reproductions
55+
- `pretty/` for pretty-print formatting tests
56+
- `latest/postgres/` for PostgreSQL regression test extracts
57+
- `original/` for original upstream PostgreSQL test files
58+
59+
Each SQL statement in the file becomes a separate test case. Use comments for context:
60+
61+
```sql
62+
-- Brief description of what's being tested
63+
-- Ref: constructive-io/pgsql-parser#289
64+
SELECT EXTRACT(EPOCH FROM now());
65+
SELECT EXTRACT(YEAR FROM TIMESTAMP '2001-02-16 20:38:40');
66+
```
67+
68+
### Step 2: Regenerate fixtures
69+
70+
From `packages/deparser/`:
71+
72+
```bash
73+
# Generate generated.json from all kitchen-sink SQL files
74+
npm run fixtures
75+
76+
# Generate kitchen-sink test files from generated.json
77+
npm run fixtures:kitchen-sink
78+
79+
# Or do both at once:
80+
npm run kitchen-sink
81+
```
82+
83+
**What `make-fixtures.ts` does:**
84+
- Reads all `__fixtures__/kitchen-sink/**/*.sql` files
85+
- Splits each file into individual statements using `statement-splitter.ts`
86+
- Validates each statement parses correctly via `libpg-query`
87+
- Writes all statements to `__fixtures__/generated/generated.json` as `{"relative/path-N.sql": "SQL statement"}`
88+
89+
**What `make-kitchen-sink.ts` does:**
90+
- Reads all `__fixtures__/kitchen-sink/**/*.sql` files
91+
- For each file, generates a test file in `packages/deparser/__tests__/kitchen-sink/`
92+
- Test file name: `{category}-{name}.test.ts` (slashes become hyphens)
93+
- Each test file uses `FixtureTestUtils.runFixtureTests()` with the list of statement keys
94+
95+
### Step 3: Run tests
96+
97+
```bash
98+
# Run all deparser tests
99+
cd packages/deparser && npx jest
100+
101+
# Run only kitchen-sink tests
102+
npx jest __tests__/kitchen-sink/
103+
104+
# Run a specific fixture test
105+
npx jest __tests__/kitchen-sink/misc-extract.test.ts
106+
```
107+
108+
## Test Verification
109+
110+
The `FixtureTestUtils.runFixtureTests()` method (via `TestUtils.expectAstMatch()`) performs this verification for each statement:
111+
112+
1. **Parse** the original SQL -> AST
113+
2. **Deparse** (non-pretty) the AST -> SQL string
114+
3. **Reparse** the deparsed SQL -> new AST
115+
4. **Compare** the cleaned ASTs — they must match
116+
5. **Repeat** steps 2-4 with `pretty: true`
117+
118+
This ensures full round-trip fidelity: `parse(sql) -> deparse(ast) -> parse(deparsed) === original AST`
119+
120+
The `expectParseDeparse()` utility in `test-utils/index.ts` does the same thing for hand-written tests.
121+
122+
## Pretty-Print Tests
123+
124+
Pretty-print tests use a different pattern via `PrettyTest` class in `test-utils/PrettyTest.ts`:
125+
126+
```typescript
127+
import { PrettyTest } from '../../test-utils/PrettyTest';
128+
129+
const testCases = [
130+
'pretty/misc-1.sql',
131+
'pretty/misc-2.sql',
132+
];
133+
134+
const prettyTest = new PrettyTest(testCases);
135+
prettyTest.generateTests();
136+
```
137+
138+
This generates two tests per case (pretty and non-pretty) and uses Jest snapshots. Pretty tests read from `generated.json` and compare output via `toMatchSnapshot()`.
139+
140+
## Adding a New Fixture (Quick Reference)
141+
142+
1. Create/edit a `.sql` file in `__fixtures__/kitchen-sink/{category}/`
143+
2. Run `cd packages/deparser && npm run kitchen-sink`
144+
3. Run `npx jest` to verify all tests pass
145+
4. Commit the `.sql` file, `generated.json`, and any new generated test files in `__tests__/kitchen-sink/`
146+
147+
## PL/pgSQL Fixtures (`plpgsql-deparser`)
148+
149+
The PL/pgSQL deparser has its own fixture pipeline:
150+
151+
```bash
152+
cd packages/plpgsql-deparser && npm run fixtures
153+
```
154+
155+
This runs `scripts/make-fixtures.ts` which:
156+
- Reads SQL files from `__fixtures__/plpgsql/`
157+
- Parses with `libpg-query`, filters to `CreateFunctionStmt` (language plpgsql) and `DoStmt`
158+
- Validates each statement parses as PL/pgSQL via `parsePlPgSQLSync()`
159+
- Writes to `__fixtures__/plpgsql-generated/generated.json`
160+
161+
## Transform Kitchen-Sink (`transform`)
162+
163+
The transform package has its own kitchen-sink and AST test scripts:
164+
165+
```bash
166+
cd packages/transform
167+
npm run kitchen-sink # generate transform-specific kitchen-sink tests
168+
npm run test:ast # run AST round-trip validation
169+
```
170+
171+
## Package Scripts Reference
172+
173+
### `packages/deparser` (primary fixture pipeline)
174+
175+
| Script | Command | Description |
176+
|--------|---------|-------------|
177+
| `fixtures` | `ts-node scripts/make-fixtures.ts` | Regenerate `generated.json` |
178+
| `fixtures:kitchen-sink` | `ts-node scripts/make-kitchen-sink.ts` | Regenerate kitchen-sink test files |
179+
| `kitchen-sink` | `npm run fixtures && npm run fixtures:kitchen-sink` | Both steps combined |
180+
| `fixtures:ast` | `ts-node scripts/make-fixtures-ast.ts` | Generate AST JSON fixtures |
181+
| `fixtures:sql` | `ts-node scripts/make-fixtures-sql.ts` | Generate SQL fixtures via native deparse |
182+
| `fixtures:upstream-diff` | `ts-node scripts/make-upstream-diff.ts` | Generate diff comparing upstream (libpg-query) vs our deparser output |
183+
| `test` | `jest` | Run all tests |
184+
| `test:watch` | `jest --watch` | Run tests in watch mode |
185+
186+
### `packages/plpgsql-deparser`
187+
188+
| Script | Command | Description |
189+
|--------|---------|-------------|
190+
| `fixtures` | `ts-node scripts/make-fixtures.ts` | Extract PL/pgSQL fixtures to `plpgsql-generated/generated.json` |
191+
192+
### `packages/transform`
193+
194+
| Script | Command | Description |
195+
|--------|---------|-------------|
196+
| `kitchen-sink` | `ts-node scripts/make-kitchen-sink.ts` | Generate transform kitchen-sink tests |
197+
| `test:ast` | `ts-node scripts/test-ast.ts` | AST round-trip validation |
198+
199+
### `packages/parser`
200+
201+
| Script | Command | Description |
202+
|--------|---------|-------------|
203+
| `prepare-versions` | `ts-node scripts/prepare-versions.ts` | Generate version-specific sub-packages from `config/versions.json` |
204+
| `test:ast` | `ts-node scripts/test-ast.ts` | AST round-trip validation |

0 commit comments

Comments
 (0)