Zero copy functionality vs just PostMessage across workers?

[wysisoft]

For the foreseeable future I will be adding nodejs functionality on workers separate from where nodepod booted.

For example, VSCode's multi-file search occurs on its own worker, and I am currently just messaging from this worker back to the main thread where nodepod "lives" for file searching, it works but its manual messaging.

I see in the docs there is some zero-copy functionality for VFS, should I look into this instead? If so, can you comment on the limitations like 16,000 files, etc?

It would be cool long term if nodepod had a client with a GetExisting method, and I specify some id and nodepod client just hooks into the existing instance that booted on another thread, and I just use these thin clients across a bunch of threads

[R1ck404]

Honest answer, the docs oversold it. That zero-copy VFS stuff is used internally between nodepods own workers, it's not exposed publicly, so from a sibling worker you can't actually hook into it today. My bad, I'll fix the docs.

For when it does get exposed, the limits are real though. 16,384 files max, 64 MB total buffer, 248 byte path limit, and if you go over it just fails silently (no eviction). Fine for source trees, not fine if node_modules is in scope.

Your GetExisting idea is the right shape and pretty doable. The reader side is already stateless and safe for multiple workers, we just need to expose the SAB handles from the booted instance and add an attach() factory. Manual messaging is your best bet until then. I'll open a tracking issue later.

[wysisoft]

That would be so cool to boot nodepod with id 1234, then I can imagine almost a compilation step where all source code require(fs) becomes new nodepodClient(1234).fs, etc

[wysisoft]

Is this something you got on your end or do you want me to limp something together as a proof of concept? I am not very familiar with all the moving pieces, and will probably just try to use some kind of codegen tooling to create the client from the sdk ts code or comlink or something

[R1ck404]

Ill work on this when i have time, this is however not a top priority for me. ill fix any bugs first and after that if i have time ill work on this.

[wysisoft]

You know it’s not a lot of code on my side to do this narrowly for my needs. A general purpose nodepod polyfill that I can apply to arbitrary js code would be really cool feature, I’ll leave that to you probably unless it falls in my lap

[R1ck404]

update, found some time for this. got a branch going at feat/attach-shared-fs with the reader side working. theres a runnable demo at examples/shared-fs-attach/.

api looks like this. on main:

const nodepod = await Nodepod.boot({ /* ... */ });

const sibling = new Worker('/my-worker.js', { type: 'module' });
sibling.postMessage({ type: 'attach', buffer: nodepod.sharedFSBuffer });

on the sibling:

import { Nodepod } from "@scelar/nodepod";

let fs;
self.addEventListener('message', async (ev) => {
  if (ev.data.type === 'attach') {
    fs = Nodepod.attachFS(ev.data.buffer);

    // async, fs.promises shaped
    const text   = await fs.readFile('/src/index.ts', 'utf8');
    const bytes  = await fs.readFile('/bin/data');        // Uint8Array
    const files  = await fs.readdir('/src');
    const stat   = await fs.stat('/package.json');
    const exists = await fs.exists('/tsconfig.json');
  }
});

one postMessage to hand over the buffer, after that every read hits shared memory directly so no per-call IPC. version counter + waitForChange are there too if you want to block until something changes.

writes throw ENOTSUP for now since the canonical MemoryVolume lives on the booting thread and a sibling writing straight to the SAB would desync it. if you need writes youd still route them back over a MessagePort. might add a write-port option later but read-only covers the multi-file search case you described.

caps are unchanged, still 16384 entries / 64 MB / 248 byte paths, so node_modules scale isnt there yet, bumping those is a separate thing.

40 tests green including one cross-thread test with node:worker_threads that proves the SAB actually bridges. also found and fixed a latent bug in VFSBridge.watch where main-thread writes were landing in the SAB under relative paths, which nothing read before so it had gone unnoticed.

ill polish a bit more then open a PR against main.

[R1ck404]

1.6.0 is out on npm with this in it.

tested the demo in the browser and the cross-thread case in vitest, both green. my test vfs is small though, handful of files, so i havent stressed the caps.

can you try it against your actual multi-file search setup? if it works ill close this. if you hit the 16384 entries / 64mb / 248 path walls before its useful, or you end up needing writes from the sibling, drop a new issue for those and well tackle them separately.

[wysisoft]

is the 16384 limits just for VFS sharing or for VFS overall? I'll add that I need a real nodepodClient for my current effort, so I'm creating a nodepodClient over postmessage

…

On Wed, Apr 22, 2026 at 3:57 AM Rick ***@***.***> wrote: *R1ck404* left a comment (ScelarOrg/Nodepod#31) <#31 (comment)> 1.6.0 is out on npm with this in it. tested the demo in the browser and the cross-thread case in vitest, both green. my test vfs is small though, handful of files, so i havent stressed the caps. can you try it against your actual multi-file search setup? if it works ill close this. if you hit the 16384 entries / 64mb / 248 path walls before its useful, or you end up needing writes from the sibling, drop a new issue for those and well tackle them separately. — Reply to this email directly, view it on GitHub <#31?email_source=notifications&email_token=ADTZR3ER4GQ3XNW3XUY5T3L4XCJPHA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTIMRZGUZDKNBQGAZ2M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4295254003>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADTZR3DPFZ3VOHZ6BL2ZKAT4XCJPHAVCNFSM6AAAAACX55DVBKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DEOJVGI2TIMBQGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[R1ck404]

Just the sharing layer, not the VFS overall.

The 16384 cap lives in src/threading/shared-vfs.ts — it's MAX_ENTRIES on the SharedArrayBuffer-backed file table that workers read from synchronously. The SAB layout has to be pre-allocated (header + fixed entry table + data region), so the slot count is baked in at construction. At 264 bytes per entry that's already ~4.2 MB just for the table.

The main VFS (MemoryVolume in src/memory-volume.ts) has no such limit — it's just plain Maps and grows unbounded. Same story for the async bridge path.

So anything over 16384 entries still works fine, it just falls off the sync fast-path and goes through the async bridge instead. If we ever bump into it for real we can either raise the constant or switch to a growable hash table layout, but 16384 felt like a reasonable starting point.