feat(pool): add pool.memoryLimit to recycle workers by RSS

[pkasarda]

Draft — opening for visibility, no CC, no reviewers.

What

Adds pool.memoryLimit to recycle a reused worker once its RSS exceeds a configurable threshold. Mirrors Vitest's VmOptions.memoryLimit so the API shape is familiar.

defineConfig({
  isolate: false,
  pool: { memoryLimit: '1GB' },
});

Accepted values:

• number >= 1 — bytes
• number in (0, 1] — fraction of total system memory
• string with a unit suffix — "512MB", "1.5GB", "20%", "1GiB", … (case-insensitive)

Unknown units / unparseable strings / non-positive numbers disable the cap rather than erroring out, so a misspelled config option doesn't tank CI.

Why

isolate: false reuses worker processes across files for speed, but vendor module caches, JSDOM nodes, React fiber trees, and accumulated closures grow the heap monotonically. On a long jsdom-heavy suite we measured a single worker climbing from 71 MB at file 1 to 4 GB at file 65, with per-test wall time degrading 2-3× under GC pressure. Without an upper bound, users either give up isolate: false entirely or hit OOM.

Mechanism (zero protocol changes)

The worker already reports process.memoryUsage().rss after each task — the existing MemoryGate spawn gate uses it to defer new spawns under memory pressure. This PR consumes the same signal:

• PoolRunner captures response.memory.rss from each runFinished / collectFinished message.
• isUsable() returns false when the last sample exceeds the parsed byte cap.
• The pool's existing release path disposes the worker instead of returning it to the idle pool; the next dispatch spawns a fresh one.
• isolate: true opts out — workers are single-use, so the cap is dead code there.

What doesn't change

• isolate: true semantics — unchanged.
• isolate: false without memoryLimit — unchanged (unbounded reuse).
• Worker startup, IPC protocol, memory reporting — unchanged.

Files

• packages/core/src/pool/parseMemoryLimit.ts — new, parses the user-facing union to bytes
• packages/core/src/pool/poolRunner.ts — lastReportedRssBytes, memoryLimitBytes opt, isUsable() cap check
• packages/core/src/pool/pool.ts — wires bytes through to PoolRunner only when isolate: false
• packages/core/src/pool/index.ts — calls parseMemoryLimit once at pool construction
• packages/core/src/pool/types.ts — PoolOptions.memoryLimitBytes
• packages/core/src/types/config.ts — RstestPoolOptions.memoryLimit
• packages/core/tests/pool/parseMemoryLimit.test.ts — 31 cases covering byte / fraction / unit / percent / whitespace / invalid inputs
• e2e/memory-limit/ — fixture with memoryLimit: 2 (2 bytes) + maxWorkers: 1 + 4 files. Driver asserts unique process.pid per file, proving the cap recycles end-to-end.

Testing

• pnpm --filter @rstest/core test — 273/273 (was 242 — +31 new parseMemoryLimit cases)
• e2e/memory-limit — 1/1
• Full e2e — 628/630 (2 pre-existing browser-mode flakes unrelated)

Status

Draft. Opening for visibility — not requesting review yet. No CC. Will mark ready once we've validated on a workspace running under isolate: false at scale.