feat: threading and multiprocessing adapters in WASM

[peter-gy]

Original POC PR description

Drafting this as a POC on stubbing threading and multiprocessing in WASM notebooks.
This first slice adds a minimal threading patch for Pyodide:

• bootstrap WASM runtime patches before marimo imports runtime modules that capture
  threading.local
• patch the small threading surface needed for the POC: Thread, Event, local, current thread, ident, enumerate/count
• give started WASM threads a synthetic identity
• make threading.local use that identity
• repair marimo runtime context storage if it was imported before the patch
• let mo.Thread hand async targets back to the patched runner

The main thing to review is the lifecycle: install early, install once, keep one interpreter-wide state object, and make marimo runtime context follow the synthetic thread identity.

Review order

Probably easiest to read in this order:

1. marimo/__init__.py and marimo/_pyodide/bootstrap.py
2. marimo/_runtime/_wasm/__init__.py
3. marimo/_runtime/_wasm/_concurrency/_install.py
4. marimo/_runtime/_wasm/_concurrency/_state.py
5. marimo/_runtime/_wasm/_concurrency/_threading.py
6. marimo/_runtime/_wasm/_concurrency/_wait.py
7. marimo/_runtime/threads.py
8. tests/_runtime/test_wasm_threading_poc.py

WASM Demo

Screen.Recording.2026-06-09.at.16.56.56.mov

Not in this PR / Follow-up Work

Leaving these out for now, but the intended approaches are:

• ThreadPoolExecutor: build on the same synthetic thread identity model, with
  submitted callables running through asyncio tasks and returning normal-looking
  Futures.
• concurrent.futures.wait / as_completed: adapt future completion to the Pyodide
  wait bridge so timeout and completion behavior stay close to stdlib call sites.
• multiprocessing.Process: keep the process API callable in Pyodide, but execute the
  target in the current interpreter and report lifecycle state through start, join,
  is_alive, and exitcode.
• multiprocessing.Queue: use an in-memory queue implementation that works with the
  process-shaped APIs above.
• multiprocessing.Pool: map pool work onto the same current-interpreter execution
  model, mostly for common notebook/library code that imports Pool.
• ProcessPoolExecutor: layer futures over the process-shaped runner, so code using
  the API can run without crashing in Pyodide.
• stream proxy repair: if stdout/stderr proxies captured a pre-patch threading.local,
  swap them to the WASM local and sync state on restore.
• browser matrix / Pyodide acceptance tests: run the examples in a real WASM notebook
  path, including mo.Thread, stdlib threading, futures, and multiprocessing-shaped
  code.
• docs and lint-rule wording: explain which APIs marimo patches in WASM and which APIs
  still have Pyodide-specific semantics.

Belongs to PR family of #9413 and #9480.

This PR makes the common Python concurrency APIs callable in WASM notebooks by installing browser-backed adapters for threading, futures, and process-shaped multiprocessing. The adapters run inside the current Pyodide interpreter on the browser event loop while preserving the API shape and lifecycle semantics that notebook and library code commonly expects.

Summary

Adds WASM adapters for:

• threading.Thread, threading.Event, threading.local, current thread identity, and thread enumeration
• concurrent.futures.ThreadPoolExecutor, Future.result, callbacks, wait, and as_completed
• process-shaped multiprocessing.Process, Queue, SimpleQueue, and Pool
• concurrent.futures.ProcessPoolExecutor
• mo.Thread output routing from WASM workers back to the spawning cell

The branch also updates WASM lint rules and docs so supported process-shaped APIs are no longer flagged as unavailable, while we still call out unsupported native process primitives.

Runtime model

As per the POC (e1c9570), all work stays in one Pyodide interpreter. Started threads, executor workers, and process-shaped targets are scheduled onto the browser-backed asyncio loop with synthetic thread and process identities.

Blocking-looking waits use the Pyodide JSPI bridge when needed, so calls like thread.join(), Event.wait(), Future.result(), wait(), as_completed(), Queue.get(), and pool result waits can yield to browser runtime work instead of failing immediately.

APIs that require OS processes, shared memory, pipes, managers, fork/forkserver contexts, or native synchronization remain unsupported.

Support Levels

WASM notebooks run Python inside one browser-hosted Pyodide interpreter. Some concurrency APIs can preserve their Python API shape there, but their execution model differs from server-backed CPython. This PR introduces four support levels to make those differences explicit:

• api-compatible: the tested API shape and result behavior match the local Python contract for that operation.
• serialized: the API shape is available, but submitted work runs one task at a time in the current Pyodide interpreter.
• cooperative-only: waits, cancellation, and termination progress when Python yields back to Pyodide's event loop. Already-running Python code is not preempted.
• blocked: marimo rejects the API because the browser cannot provide the native process, synchronization, or shared-memory primitive it requires.

Changes

• Installs WASM concurrency patches during Pyodide bootstrap before marimo runtime modules capture thread-local state.
• Keeps one interpreter-wide WASM runtime state for synthetic thread identity, current process ownership, live work tracking, and shutdown.
• Adds a serialized WASM executor for ThreadPoolExecutor and process-pool-shaped APIs.
• Adds process-shaped lifecycle support for multiprocessing.Process: start, join, is_alive, exitcode, current_process, parent_process, and active_children.
• Adds in-memory Queue and SimpleQueue adapters that preserve the standard put/get shape inside the browser interpreter.
• Adds multiprocessing.Pool support for apply, map, starmap, imap, async results, callbacks, close/join, and termination semantics that make sense in Pyodide.
• Adds ProcessPoolExecutor over the same process-shaped executor layer.
• Repairs pre-imported marimo runtime context storage and stdout/stderr stream proxies so mo.Thread output and progress updates route back to the correct cell.
• Extends WASM lint tests and docs to distinguish supported adapters from APIs that still cannot work in Pyodide.

Testing

• Added runtime coverage for WASM threading, futures, multiprocessing process, queue, pool, and process pool adapters
• Added Pyodide browser matrix coverage for the public concurrency surfaces
• Added stream and message-type coverage for WASM thread output routing
• Added lint coverage for supported and unsupported WASM multiprocessing APIs

Demo

Outputs from this test notebook:

Follow-ups

Having all the above primitives introduced in this PR, we could add support for:

• threading.Timer, cooperative-only: can be implemented as a delayed AsyncioThread scheduled on the Pyodide event loop. It does not need native threads, only delayed cooperative execution plus cancel() before the callback runs.

• multiprocessing.Event, cooperative-only: we can map it to the existing AsyncEvent shape already used for threading.Event and queue waits. This would cover common library code that just needs set, clear, is_set, and wait.

• multiprocessing.Lock / RLock, cooperative-only: we could add same-interpreter locks backed by asyncio state and JSPI waits.

• multiprocessing.Semaphore / BoundedSemaphore, cooperative-only: we can build on the same cooperative wait primitive as queues. Acquisition would only progress when Python yields to Pyodide.

• multiprocessing.Condition, cooperative-only: we could layer over the cooperative lock plus waiter list. This is mechanically straightforward once Lock/RLock exist and would have browser-event-loop scheduling semantics.

• multiprocessing.JoinableQueue, serialized: we could extend the in-memory queue with unfinished-task accounting, task_done(), and join(). Since Queue is already same-interpreter, this is probably the cleanest multiprocessing follow-up.

• multiprocessing.Pipe, serialized: can be implemented as paired in-memory endpoints with send, recv, poll, and close.

• multiprocessing.pool.ThreadPool, serialized: could be just aliased to the existing serialized executor/Pool machinery.

• Process.sentinel, cooperative-only: we can expose a synthetic wait token for adapted processes to support simple completion checks.

• Mixed pending concurrent.futures.wait / as_completed, cooperative-only: we can wrap observable foreign futures with callbacks and keep the current error for futures that cannot be observed without blocking the Pyodide event-loop lane.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
marimo-docs	Ready	Preview, Comment	Jun 14, 2026 9:33am

[cubic-dev-ai]

4 issues found across 11 files

Architecture diagram

sequenceDiagram
    participant Boot as marimo/__init__.py
    participant PyodideBoot as marimo/_pyodide/bootstrap.py
    participant WasmInit as marimo/_runtime/_wasm/__init__.py
    participant Install as marimo/_runtime/_wasm/_concurrency/_install.py
    participant State as marimo/_runtime/_wasm/_concurrency/_state.py
    participant Threading as marimo/_runtime/_wasm/_concurrency/_threading.py
    participant Wait as marimo/_runtime/_wasm/_concurrency/_wait.py
    participant Patches as marimo/_runtime/_wasm/_patches.py
    participant Threads as marimo/_runtime/threads.py
    participant Runtime as Runtime Context Storage

    Note over Boot,Patches: WASM Threading Bootstrap Flow

    alt Platform is emscripten
        Boot->>WasmInit: ensure_wasm_runtime_bootstrapped()
        PyodideBoot->>WasmInit: ensure_wasm_runtime_bootstrapped()
        WasmInit->>Install: install_wasm_threading_shims()
        
        Install->>State: set_patch_state() save originals
        Install->>State: Check if already installed
        
        alt Not yet installed
            Install->>Patches: WasmPatchSet() create
            Install->>Patches: replace(threading.Thread, AsyncioThread)
            Install->>Patches: replace(threading.Event, AsyncEvent)
            Install->>Patches: replace(threading.local, AsyncLocal)
            Install->>Patches: replace(threading.current_thread, state.current_thread)
            Install->>Patches: replace(threading.get_ident, state.current_ident)
            Install->>Patches: replace(threading.enumerate, state.active_threads)
            Install->>Patches: replace(threading.active_count, state.active_count)
            
            Note over Install,Patches: NEW: Runtime context repair
            
            Install->>Runtime: Check if context.types already imported
            alt Already imported
                Install->>Runtime: Replace _THREAD_LOCAL_CONTEXT with AsyncLocal
                Install->>Patches: before_restore callback syncs context back
            end
            
            Patches-->>Install: return unpatch handle
            Install->>State: set_active_unpatch()
        else Already installed
            Install->>Install: return no-op unpatch
        end
        
        State-->>Install: Active unpatch handle
        Install-->>WasmInit: Unpatch handle
    end

    Note over Threads,Wait: Thread Lifecycle (after bootstrap)

    Threads->>Threads: mo.Thread.__init__() inherits from patched threading.Thread
    
    Threads->>Threads: run() method called
    
    alt Managed thread with marimo context
        Threads->>Runtime: initialize_context() copy to thread
        Threads->>Threading: super().run() calls AsyncioThread.run()
        
        Threading->>State: get_event_loop()
        State-->>Threading: asyncio loop
        Threading->>State: new_ident() get synthetic ID
        State-->>Threading: ident
        Threading->>State: live_threads.add(self)
        
        alt Loop is running
            Threading->>State: create_task_in_empty_wasm_context(loop, coroutine)
            State->>State: contextvars.Context() isolate
            State-->>Threading: asyncio.Task
        else Loop not running
            Threading->>State: run_until_complete_in_empty_wasm_context(loop, coroutine)
        end
        
        Threading->>Threading: _run_in_context() set context var
        Threading->>Threads: Call target() or run()
        
        alt Target returns awaitable
            Threads->>Threads: _finish_awaitable_run() cleanup
            Threads-->>Threading: return finished
        else Target returns non-awaitable
            Threads->>Runtime: teardown_context()
            Threads->>Threads: THREADS.discard(thread_id)
        end
        
        alt thread.join() called
            Threads->>Threading: join() via cooperative_wait()
            Threading->>Wait: cooperative_wait(join_future)
            Wait->>Wait: pyodide.ffi.run_sync()
            Wait-->>Threading: result
            Threading-->>Threads: joined complete
        end
    end

    Note over Threading,Wait: AsyncLocal lifecycle

    Threading->>Threading: AsyncLocal.__new__() create storage
    Threading->>Threading: AsyncLocal._namespace() get per-ident dict
    
    alt New thread identity
        Threading->>Threading: Initialize per-thread storage
    end
    
    Threading->>Threading: __getattribute__/__setattr__ on namespace
    Note over Threading: Uses current_ident() for lookup
    
    alt Thread completes
        Threading->>Threading: clear_thread_local_state(ident)
        Threading->>Threading: Remove storage for finished ident
    end

Loading

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[Copilot]

Pull request overview

This PR introduces WebAssembly (Pyodide) concurrency adapters so common Python concurrency APIs remain callable in WASM notebooks by mapping their work onto the browser event loop with synthetic identities (instead of OS threads/processes). It also updates marimo’s runtime/bootstrap, messaging streams, lint rules, and adds broad unit + Pyodide acceptance coverage for the new behavior.

Changes:

• Add Pyodide/WASM shims for threading, concurrent.futures, and process-shaped multiprocessing APIs, including cooperative “blocking” waits bridged via pyodide.ffi.run_sync.
• Ensure WASM runtime patches install early (during bootstrap / marimo import on emscripten) and add runtime shutdown handling for Pyodide sessions.
• Extend stream support via Stream.copy_for_thread() and add comprehensive tests + docs/lint updates to reflect the new WASM support boundaries.

Reviewed changes

Copilot reviewed 51 out of 51 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/_runtime/test_wasm_threading_poc.py	Unit coverage for threading shim lifecycle, pre-import repairs, and bootstrap ordering.
tests/_runtime/test_wasm_process_pool_executor.py	Tests for ProcessPoolExecutor adapter installation/cleanup and behavior.
tests/_runtime/test_wasm_multiprocessing.py	Tests for process-shaped multiprocessing.Process adapter semantics and lifecycle.
tests/_runtime/test_wasm_multiprocessing_queue.py	Tests for WASM multiprocessing.Queue/SimpleQueue adapters, close semantics, and waits.
tests/_runtime/test_wasm_multiprocessing_pool.py	Tests for WASM multiprocessing.Pool adapter API shape, iterators, callbacks, and termination semantics.
tests/_runtime/test_threads.py	Ensures mo.Thread correctly copies script-context Pyodide streams across threads.
tests/_runtime/_helpers/wasm.py	Shared helpers for installing WASM shims in tests and async polling utility.
tests/_pyodide/test_pyodide_streams.py	Adds PyodideStream.copy_for_thread() test coverage.
tests/_pyodide/test_pyodide_session.py	Verifies Pyodide kernel teardown disposes lifecycle items and shuts down WASM runtime work (including logging on timeout).
tests/_pyodide/test_pyodide_acceptance.mjs	Makes Pyodide import more robust in Node acceptance harness (fallback to local pyodide.mjs).
tests/_pyodide/fixtures/wasm_concurrency/wasm_concurrency_matrix_cases/_shared.py	Shared matrix harness utilities for Pyodide concurrency behavior/tier recording.
tests/_pyodide/fixtures/wasm_concurrency/wasm_concurrency_matrix_cases/threading_cases.py	Matrix coverage for threading/event/local/identity/enumeration/excepthook behavior.
tests/_pyodide/fixtures/wasm_concurrency/wasm_concurrency_matrix_cases/futures_cases.py	Matrix coverage for ThreadPoolExecutor/Future/wait/as_completed and asyncio executor integrations.
tests/_pyodide/fixtures/wasm_concurrency/wasm_concurrency_matrix_cases/marimo_thread_cases.py	Matrix coverage for mo.Thread output routing, UI IDs, and child app embedding.
tests/_pyodide/fixtures/wasm_concurrency/wasm_concurrency_matrix_cases/stress_cases.py	Stress coverage across queues/executors/process-shaped primitives in Pyodide.
tests/_pyodide/fixtures/wasm_concurrency/matrix_cell.py	Pyodide notebook-cell harness that runs matrix groups and emits JSON results/failures.
tests/_messaging/test_types.py	Adds coverage for NoopStream.copy_for_thread().
tests/_messaging/test_streams.py	Ensures thread-copied ThreadSafeStream instances serialize writes via a shared transport lock.
marimo/_runtime/threads.py	Updates mo.Thread to use stream.copy_for_thread(), dataclass replace(), and to correctly handle awaitable targets in Pyodide while cleaning up context/THREADS.
marimo/_runtime/_wasm/_patches.py	Extends patching utilities with typed WrapperFactory, before_restore, descriptor replacement, and custom cleanup hooks.
marimo/_runtime/_wasm/_duckdb/init.py	Updates wrapper factory typing to match new generic WrapperFactory.
marimo/_runtime/_wasm/_concurrency/_wait.py	Adds cooperative wait bridge for “blocking” waits via pyodide.ffi.run_sync, plus timeout-safe helpers.
marimo/_runtime/_wasm/_concurrency/_state.py	Introduces interpreter-wide WASM concurrency state: synthetic identities, live-work tracking, context isolation, and shutdown coordination.
marimo/_runtime/_wasm/_concurrency/_threading.py	Implements asyncio-backed threading.Thread/Event/local adapters with synthetic identities and excepthook handling.
marimo/_runtime/_wasm/_concurrency/_mp_context.py	Adds spawn-only start-method validation and helpers for WASM multiprocessing adapter.
marimo/_runtime/_wasm/_concurrency/_mp_process.py	Implements same-interpreter multiprocessing.Process adapter with owned-work tracking and cooperative kill/terminate.
marimo/_runtime/_wasm/_concurrency/_mp_queue.py	Implements in-memory process-shaped Queue/SimpleQueue with cooperative waits and owner-close semantics.
marimo/_runtime/_wasm/_concurrency/_mp_pool.py	Implements same-interpreter multiprocessing.Pool adapter (serialized executor-backed).
marimo/_runtime/_wasm/_concurrency/_process_pool.py	Implements ProcessPoolExecutor adapter over the serialized WASM executor layer.
marimo/_runtime/_wasm/_concurrency/_process_install.py	Installs/uninstalls multiprocessing and ProcessPoolExecutor shims, including lazy submodule creation/cleanup.
marimo/_runtime/_wasm/_concurrency/_install.py	Installs core threading/futures shims, patches loop task creation for process ownership tracking, and repairs pre-imported runtime/stream locals.
marimo/_runtime/_wasm/_concurrency/init.py	Declares WASM concurrency package.
marimo/_runtime/_wasm/init.py	Provides public bootstrap/unpatch/shutdown APIs for WASM runtime concurrency shims.
marimo/_pyodide/streams.py	Adds PyodideStream.copy_for_thread() used by mo.Thread context copying.
marimo/_pyodide/pyodide_session.py	Disposes lifecycle items and shuts down WASM runtime work during Pyodide kernel teardown.
marimo/_pyodide/bootstrap.py	Bootstraps WASM runtime shims early in Pyodide before importing runtime modules that capture thread locals.
marimo/_messaging/types.py	Adds Stream.copy_for_thread() API (defaulting to error) and NoopStream implementation.
marimo/_messaging/streams.py	Implements ThreadSafeStream.copy_for_thread() sharing the transport lock to serialize writes.
marimo/_lint/rules/wasm/unsafe_system_calls.py	Enhances unsafe-call detection (incl. multiprocessing primitives) using shared alias-aware analysis.
marimo/_lint/rules/wasm/incompatible_imports.py	Allows WASM-compatible multiprocessing adapters while flagging unsupported multiprocessing exports/submodules.
marimo/init.py	Ensures WASM shims install before runtime imports when running on emscripten.
frontend/public/files/wasm-intro.py	Updates WASM intro content to describe cooperative concurrency (and keep cell return tuples).
docs/guides/wasm.md	Documents WASM concurrency support levels and the supported/blocked surface area.
docs/guides/lint_rules/rules/unsafe_system_call.md	Updates lint rule docs to include multiprocessing IPC/synchronization examples.
docs/guides/lint_rules/rules/incompatible_import.md	Updates lint rule docs to reflect nuanced multiprocessing support in WASM.

[mscolnick]

this is impressive and awesome. quite a large change, but i dont have any comments. its pretty isolated enough that if we need to revert or extract it to its own package, it could do so easily.

the test coverage seems great and this is already a huge step above the existing behavior. we can test this out with some real cases