Skip to content

Commit 9fe16e0

Browse files
feat: POC native ast plugins (#47)
Final PR in the #44#45#46 native-AST stack. Adds canonical Rust (syn), Python (ruff), and Go (go/parser) WASM plugins under plugins/ast/, plus the ast-plugins CI workflow that builds, smoke-tests against the host ABI, and publishes them on plugins-v* tags. Reviewed safe: strictly additive (0 deletions), no TS/src/dist/package.json changes, npm tarball still excludes plugins/ (verified via npm pack), plugins do no I/O/network/process/fs (pure AST parsers in sandboxed WASM), deps reputably pinned (syn, ruff@SHA, Go stdlib), CI least-privilege with release gated to maintainer tags. All functional checks pass; only failure is the known fork-PR read-only-token artifact.
1 parent e996382 commit 9fe16e0

11 files changed

Lines changed: 2793 additions & 0 deletions

File tree

.github/workflows/ast-plugins.yml

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
---
2+
name: ast-plugins
3+
4+
# Builds the native-AST WASM plugins (Rust/syn, Python/ruff, Go/go-parser) from
5+
# the source under plugins/ast, smoke-tests each against the host ABI, and — on a
6+
# `plugins-v*` tag — publishes the .wasm files (+ SHA256SUMS) as release assets.
7+
#
8+
# The plugins are a self-contained, polyglot "closed box": the Node/TS package
9+
# neither builds nor depends on them (they're excluded from the npm tarball).
10+
# This workflow is the ONLY thing that compiles them, so it runs on every PR that
11+
# touches plugins/ast to guard against source rot. wasm is platform-independent,
12+
# so one artifact per plugin serves every OS/arch — no build matrix needed.
13+
#
14+
# Compatibility is governed by the ABI `contractVersion()`, not `codesight`'s npm
15+
# version, so plugins are released independently via their own `plugins-v*` tags.
16+
#
17+
# Note: Cargo.lock is committed and `cargo build --locked` enforces it, so Rust
18+
# builds are reproducible (the ruff git rev is pinned; the lockfile freezes all
19+
# transitive deps) and CI fails if the lockfile is stale.
20+
21+
on:
22+
pull_request:
23+
paths:
24+
- plugins/ast/**
25+
- .github/workflows/ast-plugins.yml
26+
27+
push:
28+
tags:
29+
- plugins-v*
30+
31+
workflow_dispatch:
32+
33+
permissions:
34+
contents: read
35+
36+
jobs:
37+
build:
38+
name: Build & smoke-test plugins
39+
runs-on: ubuntu-latest
40+
steps:
41+
- uses: actions/checkout@v4
42+
43+
- name: Set up Rust (wasm32-unknown-unknown)
44+
uses: dtolnay/rust-toolchain@stable
45+
with:
46+
targets: wasm32-unknown-unknown
47+
48+
- name: Cache cargo build
49+
uses: swatinem/rust-cache@v2
50+
with:
51+
workspaces: plugins/ast
52+
53+
- name: Set up Go
54+
uses: actions/setup-go@v5
55+
with:
56+
go-version: "1.24"
57+
cache: false # stdlib-only module, no go.sum to key a dependency cache on
58+
59+
- uses: pnpm/action-setup@v4
60+
with:
61+
version: 9
62+
63+
- uses: actions/setup-node@v4
64+
with:
65+
node-version: "20"
66+
67+
- name: Install Node deps & build host
68+
run: |
69+
pnpm install
70+
pnpm build
71+
72+
# warnings = "deny" is wired into each crate, so any warning fails here.
73+
- name: Build Rust plugins (syn / ruff → wasm)
74+
working-directory: plugins/ast
75+
run: cargo build --locked --release --target wasm32-unknown-unknown
76+
77+
# -ldflags="-s -w" drops the symbol table and DWARF (Go has no `strip = true`
78+
# equivalent); the wasm-opt pass below shrinks all three further.
79+
- name: Build Go plugin (go/parser → wasm)
80+
working-directory: plugins/ast/golang
81+
run: GOOS=wasip1 GOARCH=wasm go build -buildmode=c-shared -ldflags="-s -w" -o codesight-go-ast.wasm .
82+
83+
# Stage under the host's discovery filename template (^codesight-<lang>-ast.wasm$).
84+
# cargo emits underscores (codesight_rust_ast.wasm) and forbids hyphens in lib
85+
# target names, so the rename lives here in CI — never in the repo build.
86+
- name: Stage artifacts
87+
run: >-
88+
mkdir -p dist-plugins;
89+
cp
90+
plugins/ast/target/wasm32-unknown-unknown/release/codesight_rust_ast.wasm
91+
dist-plugins/codesight-rust-ast.wasm;
92+
cp
93+
plugins/ast/target/wasm32-unknown-unknown/release/codesight_python_ast.wasm
94+
dist-plugins/codesight-python-ast.wasm;
95+
cp
96+
plugins/ast/golang/codesight-go-ast.wasm
97+
dist-plugins/codesight-go-ast.wasm;
98+
ls -la dist-plugins
99+
100+
- name: Install wasm-opt (Binaryen)
101+
run: sudo apt-get update && sudo apt-get install -y binaryen
102+
103+
# WASM-specific whole-module size pass. -Oz optimizes for size. The toolchains
104+
# emit no target_features section but do use bulk-memory (memory.copy/fill), so
105+
# we enable exactly the post-MVP features they need — all long supported by the
106+
# Node host's V8. The smoke test below runs against these optimized binaries,
107+
# so we validate exactly what ships.
108+
- name: Optimize wasm (wasm-opt -Oz)
109+
working-directory: dist-plugins
110+
run: |
111+
for f in *.wasm; do
112+
before=$(wc -c < "$f")
113+
wasm-opt -Oz \
114+
--enable-bulk-memory --enable-sign-ext \
115+
--enable-mutable-globals --enable-nontrapping-float-to-int \
116+
"$f" -o "$f"
117+
echo "$f: ${before} -> $(wc -c < "$f") bytes"
118+
done
119+
120+
# Load each optimized plugin through the real host and round-trip a fixture to
121+
# catch host<->plugin drift bidirectionally (mirrors the reference-plugin guard).
122+
- name: Smoke-test ABI against the host
123+
env:
124+
PLUGIN_DIR: dist-plugins
125+
run: |
126+
cat > "${RUNNER_TEMP}/smoke.mjs" <<'JS'
127+
import assert from 'node:assert/strict';
128+
import { pathToFileURL } from 'node:url';
129+
130+
const host = pathToFileURL(`${process.cwd()}/dist/wasm/plugin-host.js`).href;
131+
const { loadPlugin } = await import(host);
132+
const dir = process.env.PLUGIN_DIR;
133+
134+
const fixtures = {
135+
rust: '#[get("/health")]\nfn handler() {}',
136+
python: 'from fastapi import FastAPI\napp = FastAPI()\n@app.get("/health")\ndef handler(): ...',
137+
go: 'package main\nfunc register(r *gin.Engine) { r.GET("/health", nil) }',
138+
};
139+
140+
let failed = 0;
141+
for (const [lang, src] of Object.entries(fixtures)) {
142+
try {
143+
const plugin = loadPlugin(lang, [dir]);
144+
assert.ok(plugin, `${lang}: failed to load`);
145+
assert.equal(plugin.metadata?.languageId, lang, `${lang}: describe() languageId mismatch`);
146+
const routes = plugin.routes?.(src) ?? [];
147+
assert.ok(routes.length > 0, `${lang}: expected at least one route from the fixture`);
148+
console.log(`ok: ${lang} — ${routes.length} route(s), languageId=${plugin.metadata.languageId}`);
149+
} catch (error) {
150+
console.error(`FAIL: ${error.message}`);
151+
failed = 1;
152+
}
153+
}
154+
process.exit(failed);
155+
JS
156+
node "${RUNNER_TEMP}/smoke.mjs"
157+
158+
- name: Generate checksums
159+
working-directory: dist-plugins
160+
run: sha256sum *.wasm | tee SHA256SUMS
161+
162+
- uses: actions/upload-artifact@v4
163+
with:
164+
name: ast-plugins
165+
path: dist-plugins/
166+
if-no-files-found: error
167+
168+
release:
169+
name: Publish release assets
170+
needs: build
171+
if: startsWith(github.ref, 'refs/tags/plugins-v')
172+
runs-on: ubuntu-latest
173+
permissions:
174+
contents: write
175+
steps:
176+
- uses: actions/download-artifact@v4
177+
with:
178+
name: ast-plugins
179+
path: dist-plugins
180+
181+
- name: Publish to the GitHub Release
182+
uses: softprops/action-gh-release@v2
183+
with:
184+
files: |
185+
dist-plugins/*.wasm
186+
dist-plugins/SHA256SUMS

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,8 @@ dist/
44
.DS_Store
55
docs/superpowers/
66
package-lock.json
7+
plugins/**/*.wasm
8+
plugins/**/target/
79
tests/fixtures/**/.codesight/
810
tests/fixtures/**/CODESIGHT.md
911
# Fully test-generated fixture directories

docs/wasm-plugins.md

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ This document is the contract a plugin must satisfy. It covers:
1818
- [Fallback & strict semantics](#fallback--strict-semantics)
1919
- [Plugin skeleton](#plugin-skeleton)
2020
- [Building & testing locally](#building--testing-locally)
21+
- [Releases](#releases)
2122
- [Versioning](#versioning)
2223

2324
---
@@ -373,6 +374,57 @@ function trapped on that file.
373374

374375
---
375376

377+
## Releases
378+
379+
`codesight`'s npm package ships **no** plugins — but the project maintains a small
380+
set of reference implementations under [`plugins/ast/`](../plugins/ast) and publishes
381+
them as **prebuilt release assets**, so you don't have to set up a Rust/Go toolchain
382+
to use them:
383+
384+
| Asset | Language | Parser |
385+
|------------------------------|----------|-------------------------------------------------------------------|
386+
| `codesight-rust-ast.wasm` | Rust | [`syn`](https://docs.rs/syn) |
387+
| `codesight-python-ast.wasm` | Python | [`ruff`](https://github.com/astral-sh/ruff) (`ruff_python_ast`) |
388+
| `codesight-go-ast.wasm` | Go | stdlib `go/parser` |
389+
390+
They are optional and opt-in; with none installed, `codesight` behaves exactly as
391+
its built-in extractors always have.
392+
393+
**Built in CI, not in this package.** The [`ast-plugins`](../.github/workflows/ast-plugins.yml)
394+
workflow compiles each plugin from source, smoke-tests it against the host ABI, and
395+
attaches the `.wasm` files plus a `SHA256SUMS` manifest to a GitHub Release. Because
396+
WebAssembly is platform-independent, one artifact per plugin runs on every OS and
397+
architecture — there are no per-platform downloads.
398+
399+
**Independent versioning.** Plugins are released on their own `plugins-v*` tags,
400+
decoupled from `codesight`'s npm version. Compatibility is governed solely by the ABI
401+
[`contractVersion()`](#versioning): any plugin built for the host's contract works,
402+
and one built for an older contract is cleanly skipped.
403+
404+
**Installing a released plugin:**
405+
406+
```bash
407+
mkdir -p ~/.codesight/plugins
408+
cd ~/.codesight/plugins
409+
410+
# Download the asset(s) you want from the GitHub Release — already named to match the
411+
# discovery template (no renaming needed) — plus the checksum manifest.
412+
curl -LO https://github.com/Houseofmvps/codesight/releases/download/<tag>/codesight-rust-ast.wasm
413+
curl -LO https://github.com/Houseofmvps/codesight/releases/download/<tag>/SHA256SUMS
414+
415+
# Verify integrity before trusting the binary.
416+
sha256sum --ignore-missing -c SHA256SUMS
417+
418+
# Enable it (an explicit language list is authoritative for that language's files).
419+
codesight --native-ast=rust ./my-project
420+
```
421+
422+
Dropped into any directory on the [discovery waterfall](#discovery--naming), the
423+
plugin is picked up automatically; `--plugin-dir <dir>` points at an arbitrary
424+
location instead.
425+
426+
---
427+
376428
## Versioning
377429

378430
The current contract version is **1**. Breaking changes to the ABI or JSON shapes

0 commit comments

Comments
 (0)