Skip to content

feat: switch tree-sitter from native N-API to web-tree-sitter (WASM) #76

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
HrushiBorhade merged 2 commits into from
Mar 24, 2026
Merged

feat: switch tree-sitter from native N-API to web-tree-sitter (WASM) #76

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e985ae9
feat: switch tree-sitter from native N-API to web-tree-sitter (WASM)
HrushiBorhade Mar 24, 2026
dfeb294
fix: parser memory leak on parse failure + use Pino logger consistently
HrushiBorhade Mar 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 1 addition & 8 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,14 +28,7 @@
"pnpm": {
"onlyBuiltDependencies": [
"better-sqlite3",
"esbuild",
"tree-sitter",
"tree-sitter-css",
"tree-sitter-go",
"tree-sitter-javascript",
"tree-sitter-python",
"tree-sitter-rust",
"tree-sitter-typescript"
"esbuild"
]
}
}
9 changes: 2 additions & 7 deletions packages/core/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,13 +34,8 @@
"fast-glob": "^3.3.3",
"pino": "^10.3.1",
"pino-pretty": "^13.1.3",
"tree-sitter": "^0.21.1",
"tree-sitter-css": "^0.21.1",
"tree-sitter-go": "^0.21.2",
"tree-sitter-javascript": "^0.21.4",
"tree-sitter-python": "^0.21.0",
"tree-sitter-rust": "^0.21.0",
"tree-sitter-typescript": "^0.23.2",
"tree-sitter-wasms": "^0.1.13",
"web-tree-sitter": "0.22.6",
"zod": "^4.3.6"
},
"devDependencies": {
Expand Down
145 changes: 83 additions & 62 deletions packages/core/src/chunker/ast.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,41 +1,56 @@
import Parser from 'tree-sitter';
import TypeScript from 'tree-sitter-typescript';
import JavaScript from 'tree-sitter-javascript';
import Python from 'tree-sitter-python';
import Rust from 'tree-sitter-rust';
import Go from 'tree-sitter-go';
import CSS from 'tree-sitter-css';
import { resolve } from 'node:path';
import { createRequire } from 'node:module';
import type { Chunk } from './types.ts';
import { fallbackChunk } from './fallback.ts';
import { createLogger } from '../utils/logger.ts';

const log = createLogger('chunker');

// --- Parser cache (one parser per language, reused) ---

const parsers = new Map<string, Parser>();
// --- WASM initialization ---
// web-tree-sitter types use `declare module` pattern which doesn't play well
// with nodenext resolution. We use a typed wrapper to keep things clean.

// eslint-disable-next-line @typescript-eslint/no-explicit-any
const GRAMMARS: Record<string, any> = {
typescript: TypeScript.typescript,
tsx: TypeScript.tsx,
javascript: JavaScript,
python: Python,
rust: Rust,
go: Go,
css: CSS,
let ParserClass: any;
let initialized = false;

async function ensureInit(): Promise<void> {
if (initialized) return;
const mod = await import('web-tree-sitter');
ParserClass = mod.default;
await ParserClass.init();
initialized = true;
}
Comment on lines +17 to +23

@coderabbitai coderabbitai Bot Mar 24, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Add environment-aware parser selection instead of always forcing WASM.

Current flow unconditionally initializes WASM parser for all environments. This drops the CLI native-N-API path required by project policy and can regress CLI performance/startup characteristics.

As per coding guidelines **/chunker/**/*.{ts,tsx}: Use tree-sitter with native N-API bindings for AST chunking in CLI, but switch to web-tree-sitter (WASM) for Trigger.dev Lambda/container environments.

Also applies to: 85-88

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@packages/core/src/chunker/ast.ts` around lines 17 - 23, ensureInit currently
always imports web-tree-sitter (WASM); change it to select the parser based on
environment by first trying to load the native Node N-API binding (e.g.,
require/import 'tree-sitter' or the native package your project uses) for
CLI/node environments and only fall back to import('web-tree-sitter') for
Lambda/container/trigger environments; update the logic inside ensureInit
(referencing ParserClass, initialized and the ParserClass.init() call) to
attempt native load, call ParserClass.init() if needed, and set
initialized=true, with a clear fallback to the WASM import when native loading
fails or an environment variable indicates a container/lambda runtime.


// --- Grammar loading (WASM files from tree-sitter-wasms package) ---

const require = createRequire(import.meta.url);
const wasmsDir = resolve(require.resolve('tree-sitter-wasms/package.json'), '..', 'out');

const GRAMMAR_FILES: Record<string, string> = {
typescript: 'tree-sitter-typescript.wasm',
tsx: 'tree-sitter-tsx.wasm',
javascript: 'tree-sitter-javascript.wasm',
python: 'tree-sitter-python.wasm',
rust: 'tree-sitter-rust.wasm',
go: 'tree-sitter-go.wasm',
css: 'tree-sitter-css.wasm',
};

function getParser(language: string): Parser | undefined {
if (parsers.has(language)) return parsers.get(language)!;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const languages = new Map<string, any>();

const grammar = GRAMMARS[language];
if (!grammar) return undefined;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
async function getLanguage(language: string): Promise<any | undefined> {
if (languages.has(language)) return languages.get(language)!;

const file = GRAMMAR_FILES[language];
if (!file) return undefined;

const parser = new Parser();
parser.setLanguage(grammar);
parsers.set(language, parser);
return parser;
const wasmPath = resolve(wasmsDir, file);
const lang = await ParserClass.Language.load(wasmPath);
languages.set(language, lang);
return lang;
}

// --- Top-level node types to extract per language ---
Expand Down Expand Up @@ -65,59 +80,65 @@ const AST_NODE_TYPES: Record<string, Set<string>> = {
css: new Set(['rule_set', 'media_statement', 'keyframes_statement']),
};

// --- AST chunking ---
// --- AST chunking (async — requires WASM init + language loading) ---

async function chunkAST(source: string, filePath: string, language: string): Promise<Chunk[]> {
await ensureInit();

function chunkAST(source: string, filePath: string, language: string): Chunk[] {
const parser = getParser(language);
if (!parser) {
const lang = await getLanguage(language);
if (!lang) {
log.warn(`No tree-sitter grammar for "${language}", using fallback: ${filePath}`);
return [fallbackChunk(source, filePath, language)];
}

const nodeTypes = AST_NODE_TYPES[language];
if (!nodeTypes) {
console.warn(
`[chunker] No AST node types configured for "${language}", using fallback: ${filePath}`,
);
log.warn(`No AST node types configured for "${language}", using fallback: ${filePath}`);
return [fallbackChunk(source, filePath, language)];
}

let tree: Parser.Tree;
const parser = new ParserClass();
parser.setLanguage(lang);

try {
tree = parser.parse(source);
const tree = parser.parse(source);

try {
const chunks: Chunk[] = [];

for (const node of tree.rootNode.children) {
if (!nodeTypes.has(node.type)) continue;

const content = node.text.trim();
if (content.length === 0) continue;

chunks.push({
content,
filePath,
lineStart: node.startPosition.row + 1,
lineEnd: node.endPosition.row + 1,
language,
type: 'ast',
});
}

if (chunks.length === 0 && source.trim().length > 0) {
log.warn(`AST parsed but found 0 matching nodes for "${language}" in: ${filePath}`);
return [fallbackChunk(source, filePath, language)];
}

return chunks;
} finally {
tree.delete();
}
} catch (err: unknown) {
log.error(
`tree-sitter parse failed for ${filePath}: ${err instanceof Error ? err.message : err}`,
);
return [fallbackChunk(source, filePath, language)];
} finally {
parser.delete();
}

const chunks: Chunk[] = [];

for (const node of tree.rootNode.children) {
if (!nodeTypes.has(node.type)) continue;

const content = node.text.trim();
if (content.length === 0) continue;

chunks.push({
content,
filePath,
lineStart: node.startPosition.row + 1,
lineEnd: node.endPosition.row + 1,
language,
type: 'ast',
});
}

if (chunks.length === 0 && source.trim().length > 0) {
console.warn(
`[chunker] AST parsed but found 0 matching nodes for "${language}" in: ${filePath}`,
);
return [fallbackChunk(source, filePath, language)];
}

return chunks;
}

export { chunkAST };
Loading
Loading