WASM Support

[Armaldio]

Hi,

Could you add WASM support for sem to be used as library and publish it to npm?
This would allow easy use in browser-based apps.

Note: only the diff engine here seems relevant as git and multithreading have no usage inside the browser

Thanks!

[rs545837]

Cool, I will look into it, and revert here.

[rs545837]

Hey @Armaldio, this is possible but not straightforward.

sem-core depends on 18 native tree-sitter grammars (compiled C), plus git2 and rayon for git integration and parallelism. For WASM, we'd need to swap native tree-sitter for web-tree-sitter, disable git2 and rayon, and expose just the diff engine via wasm-bindgen.

What would work in WASM:

• Entity extraction (extract_entities)
• Entity matching/diffing (compute_semantic_diff)
• Structural hashing

What wouldn't:

• Git integration, parallel processing, file system access

Here's roughly how we'd approach it:

1. Add a wasm feature flag to sem-core/Cargo.toml that gates out git2, rayon, and the native tree-sitter grammars
2. Behind that flag, swap in web-tree-sitter for grammar loading. Each grammar becomes a .wasm file loaded at runtime instead of compiled-in C
3. Create a thin sem-wasm crate that depends on sem-core with the wasm flag, uses wasm-bindgen to expose extract_entities(content, file_path) -> JSON and compute_semantic_diff(before, after, file_path) -> JSON
4. Build with wasm-pack build --target web, publish to npm
5. On the JS side, the consumer would need to load the grammar .wasm files for whichever languages they care about (no need to bundle all 18)

The binary size concern is real. Letting users load only the grammars they need (step 5) keeps it manageable.

It's on our radar but not immediate priority. If you want to take a stab at it, happy to help guide the implementation.

[MattCozendey]

I'd like to pitch in that this PR: #72 may indirectly help you since it wraps the package into essentially a wrapped binary that you could install with npm/bun/pnpm without using WASM. Other repos have done this, like scss or biome irrc.

[nikelborm]

Why would filesystem access not work? As far as I know, WASM supports the filesystem WASI interface and can use a few available storage backends like OPFS or native fs access

[rs545837]

You're right WASI and browser APIs (OPFS, File System Access API) do support filesystem access. For sem's browser use case though, you'd likely pass content strings directly rather than read files from disk, so filesystem access isn't really the blocker.

The main challenge is swapping native tree-sitter grammars for web-tree-sitter and gating out git2/rayon.

[nikelborm]

Maybe you can also explore one approach, such as https://webcontainers.io/. A simple wrapper on top of it, even though maybe with some overhead, could work. There are quite a few things like webcontainers.io, with various depths of simulation, some of them simulate a whole Linux, or even full QEMU with anything under it, like running Windows directly in the tab of a browser

[nikelborm]

A popular game https://ohmygit.org/ builds on top of this kind of stack

[nikelborm]

Although I understand that there's still a benefit in developing a WASM integration, because the web is not the only consumer of WASM, and it would make it very easy to integrate sem in any language or any platform