Skip to content

Commit c5781e6

Browse files
committed
refactor: replace pure TS scanner with WASM-based scanner from libpg-query
Replace the custom TypeScript comment scanner with PostgreSQL's real lexer via libpg-query's WASM _wasm_scan function. This eliminates the risk of bugs from reimplementing PostgreSQL's lexer in TypeScript. Key changes: - scanner.ts now loads the WASM module directly and calls _wasm_scan with proper JSON escaping (works around upstream bug where control characters in token text are not escaped) - Dependency changed from libpg-query to @libpg-query/parser (full build with scan support) - Unified loadModule() initializes both parse/deparse and scanner WASM - All 36 tests passing including multi-line block comments and dollar-quoted string handling
1 parent 7fd2072 commit c5781e6

6 files changed

Lines changed: 251 additions & 156 deletions

File tree

packages/parse/__tests__/scanner.test.ts

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
1-
import { scanComments, ScannedElement } from '../src/scanner';
1+
import { scanComments, ScannedElement, initWasm } from '../src/scanner';
22

33
describe('scanComments', () => {
4+
beforeAll(async () => {
5+
await initWasm();
6+
});
47
describe('line comments', () => {
58
it('extracts a simple line comment', () => {
69
const sql = '-- hello world\nSELECT 1;';

packages/parse/package.json

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,8 +40,8 @@
4040
"round-trip"
4141
],
4242
"dependencies": {
43+
"@libpg-query/parser": "^17.6.3",
4344
"@pgsql/types": "^17.6.2",
44-
"libpg-query": "17.7.3",
4545
"pgsql-deparser": "workspace:*"
4646
},
4747
"devDependencies": {

packages/parse/src/index.ts

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,14 @@ export { deparseEnhanced, deparseEnhancedSync, Deparser, DeparserOptions } from
2222
// Re-export standard deparse for non-enhanced use
2323
export { deparse, deparseSync } from 'pgsql-deparser';
2424

25-
// Re-export loadModule from libpg-query
26-
export { loadModule } from 'libpg-query';
25+
// Unified loadModule that initializes both the library's WASM
26+
// (for parse/deparse) and our scanner's WASM (for _wasm_scan).
27+
import { loadModule as libLoadModule } from '@libpg-query/parser';
28+
import { initWasm } from './scanner';
29+
30+
export async function loadModule(): Promise<void> {
31+
await Promise.all([libLoadModule(), initWasm()]);
32+
}
2733

2834
// Types
2935
export {

packages/parse/src/parse.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
* parse result's stmts array.
55
*/
66

7-
import { parse as libParse, parseSync as libParseSync } from 'libpg-query';
7+
import { parse as libParse, parseSync as libParseSync } from '@libpg-query/parser';
88
import { ParseResult, RawStmt } from '@pgsql/types';
99
import { scanComments, ScannedElement } from './scanner';
1010
import { EnhancedParseResult, EnhancedStmt } from './types';

0 commit comments

Comments
 (0)