local backend

Author: codeSamuraii

README.md

@@ -38,7 +38,7 @@ Only the entry point needs `@trace`. Everything it calls -- `add()`, imports, cl
 Start an isolated worker (temporary venv with `--tmp`, cleaned up on exit):
 
 ```bash
-pyfuse worker --tmp --backend shm://localhost:9847    # Same machine using shared memory
+pyfuse worker --tmp --backend local://localhost:9748  # Same machine (async TCP)
 pyfuse worker --tmp --backend redis://localhost:6379  # Remote using Redis
 ```
 
@@ -84,7 +84,7 @@ Common mappings (`cv2` -> `opencv-python`, `PIL` -> `Pillow`, etc.) are built in
 - **Class methods** -- `self.method()` and `cls.method()` dependencies are detected. Entire class hierarchies (including `super()`), class-level attributes, decorators (`@dataclass`, etc.), and metaclass keywords are reconstructed.
 - **Retry and timeout** -- `@trace(timeout=30, retries=3)` with exponential backoff.
 - **Batch submission** -- `await func.map([(a1, b1), (a2, b2)])` submits and awaits multiple tasks.
-- **Pluggable backends** -- Redis (`redis://`) for multi-machine, shared memory (`shm://`) for same-machine IPC.
+- **Pluggable backends** -- Redis (`redis://`) for multi-machine, local (`local://`) for same-machine IPC.
 - **Content-hash caching** -- Workers cache compiled functions by content hash. Same code from different clients = cache hit.
 
 ## Examples

docs/CONTEXT.md

@@ -33,7 +33,7 @@ pyfuse/
     └── backends/
         ├── base.py          # Backend ABC: async transport interface
         ├── redis.py         # RedisBackend: redis.asyncio with RPUSH/BLPOP pattern
-        └── shm.py           # SharedMemoryBackend: multiprocessing shared memory IPC
+        └── local.py         # LocalBackend: async-native TCP for same-machine IPC
 ```
 
 ## Architecture overview
@@ -97,7 +97,7 @@ await Worker.run(task)
 - **Class-level attributes**: Class body statements (assignments, annotated assignments, docstrings) are extracted from AST and emitted in reconstructed class blocks. Class decorators (`@dataclass`, etc.) and metaclass keywords (`metaclass=ABCMeta`) are captured and emitted.
 - **Closure handling**: Multi-tier capture: repr validation → traced functions → lambdas (source extraction) → non-traced user functions (auto-registration) → constructor expressions (defaultdict/Counter/deque) → pickle fallback → warning. Traced function references become dependency edges.
 - **Decorator stripping**: `@trace` lines are removed from captured source so reconstructed code doesn't depend on pyfuse.
-- **Backend auto-detection**: `connect()` picks Redis or shared memory based on URL scheme. Falls back to `PYFUSE_BACKEND` env var.
+- **Backend auto-detection**: `connect()` picks Redis or local TCP backend based on URL scheme. Falls back to `PYFUSE_BACKEND` env var.
 - **Worker caching**: Keyed by SHA-256 of all reachable content hashes (sorted + joined). Same code from different clients = cache hit.
 - **Async-native I/O**: All backend methods, worker execution, result handling, pip installation, and subprocess management use `asyncio`. Sync user functions run in `loop.run_in_executor()` to avoid blocking the event loop.
 - **Heartbeat**: Workers send heartbeats via `asyncio.create_task`. Client-side stall detection tracks when heartbeat *values* last changed using local monotonic clock (no cross-machine timestamp comparison).
@@ -166,7 +166,7 @@ pytest                    # run all tests
 pytest tests/test_api.py  # specific module
 ```
 
-15 test modules covering: API surface, AST analysis, async features (Result.result, await, .run(), .start(), .map(), gather, heartbeat, stall detection), auto-discovery (including metaclass keywords, class attributes, class decorators, `__init_subclass__`), dependency management, graph operations, integration scenarios, remote execution, runtime tracing (including closure capture of non-traced functions, lambdas, constructor expressions, pickle fallback), shared memory backend, store operations, stress tests (47 functions across 7 files), task serialization, temp venv management, and worker caching/execution.
+15 test modules covering: API surface, AST analysis, async features (Result.result, await, .run(), .start(), .map(), gather, heartbeat, stall detection), auto-discovery (including metaclass keywords, class attributes, class decorators, `__init_subclass__`), dependency management, graph operations, integration scenarios, local backend (async-native TCP), remote execution, runtime tracing (including closure capture of non-traced functions, lambdas, constructor expressions, pickle fallback), store operations, stress tests (47 functions across 7 files), task serialization, temp venv management, and worker caching/execution.
 
 All async tests use `pytest-asyncio` with `asyncio_mode = "auto"`.
 

docs/QUICK_START.md

@@ -232,14 +232,14 @@ pyfuse supports pluggable transport backends:
 | Backend | URL scheme | Use case |
 |---------|-----------|----------|
 | Redis | `redis://` / `rediss://` | Production, multi-machine |
-| Shared memory | `shm://` | Same-machine IPC, zero-copy |
+| Local | `local://` | Same-machine IPC (async-native TCP) |
 
 ```python
 # Redis (requires pip install redis)
 pyfuse.connect("redis://localhost:6379")
 
-# Shared memory (no external services needed)
-pyfuse.connect("shm://localhost:9847")
+# Local (no external services needed)
+pyfuse.connect("local://localhost:9748")
 ```
 
 The backend can also be set via the `PYFUSE_BACKEND` environment variable:
@@ -447,7 +447,7 @@ except RemoteError as e:
 
 ```bash
 # Remote execution
-python -m pyfuse worker --backend shm://localhost:9847 --tmp    # Terminal 1
+python -m pyfuse worker --backend local://localhost:9748 --tmp   # Terminal 1
 python -m pyfuse run examples/remote_execution.py               # Terminal 2
 ```
 

docs/TECHNICAL_OVERVIEW.md

@@ -43,7 +43,7 @@ pyfuse/
         backends/
             base.py          Backend ABC: async pluggable transport interface
             redis.py         RedisBackend: redis.asyncio with RPUSH/BLPOP pattern
-            shm.py           SharedMemoryBackend: same-machine IPC via shared memory
+            local.py         LocalBackend: async-native TCP for same-machine IPC
 ```
 
 ## Remote execution flow
@@ -107,15 +107,13 @@ Result notifications use Redis Pub/Sub (`PUBLISH`/`SUBSCRIBE`). Batch heartbeat
 
 The `redis` package is imported lazily and is an optional dependency.
 
-### SharedMemoryBackend
+### LocalBackend
 
-Uses `multiprocessing.shared_memory` for zero-copy payload transfer and `multiprocessing.managers.BaseManager` for cross-process coordination. No external services required -- suitable for same-machine worker pools.
+An async-native TCP backend for same-machine IPC.  A lightweight broker server built on `asyncio.start_server` handles task dispatch, result routing, and heartbeats -- no threads, no `multiprocessing`, no external services.
 
-URL format: `shm://host:port?authkey=secret` (defaults: `127.0.0.1:9847`, authkey `pyfuse`).
+URL format: `local://host:port` (default: `127.0.0.1:9748`).
 
-The backend auto-detects its role: it tries to connect as a client first; if no server is running, it starts one. Shared memory blocks are tracked and cleaned up on exit via `atexit`.
-
-Since `BaseManager` is inherently synchronous, proxy RPC calls are wrapped in `asyncio.to_thread()`. The overhead is negligible for sub-millisecond IPC calls.
+The broker auto-starts as a subprocess on first connection (or can be started explicitly with `server=True`). All I/O is native `asyncio` -- the backend opens TCP connections to the broker and communicates via a length-prefixed JSON protocol. Streaming operations (`listen()`, `subscribe_results()`) use dedicated connections; RPC operations share a single connection protected by an `asyncio.Lock`.
 
 ### Custom backends
 

examples/async_execution.py

@@ -12,7 +12,7 @@
 import pyfuse
 from pyfuse import trace
 
-pyfuse.connect("redis://localhost:6379")
+pyfuse.connect("local://localhost:9748")
 
 async def add(a: int, b: int) -> int:
     return a + b

pyfuse/worker/backends/__init__.py

@@ -1,3 +1,3 @@
 from pyfuse.worker.backends.base import Backend
+from pyfuse.worker.backends.local import LocalBackend
 from pyfuse.worker.backends.redis import RedisBackend
-from pyfuse.worker.backends.shm import SharedMemoryBackend

pyfuse/worker/backends/base.py

@@ -8,7 +8,7 @@ class Backend(abc.ABC):
     """Abstract transport backend for remote task execution.
 
     Subclass this to implement custom transports (Redis, RabbitMQ,
-    shared memory, etc.).
+    TCP, etc.).
     """
 
     @abc.abstractmethod