Merge branch 'copilot/create-micro-vms-for-python-execution'

Author: codeSamuraii

README.md

@@ -88,8 +88,20 @@ Common mappings (`cv2` -> `opencv-python`, `PIL` -> `Pillow`, etc.) are built in
 - **Retry and timeout** -- `@trace(timeout=30, retries=3)` with exponential backoff.
 - **Batch submission** -- `await func.map([(a1, b1), (a2, b2)])` submits and awaits multiple tasks.
 - **Pluggable backends** -- Redis (`redis://`) for multi-machine, local (`local://`) for same-machine IPC.
+- **Sandboxed execution** -- Run tasks inside Docker containers for isolation. Transparent to clients.
 - **Content-hash caching** -- Workers cache compiled functions by content hash. Same code from different clients = cache hit.
 
+## Sandboxed execution
+
+By default, workers execute code in the host process. For isolation, enable sandboxing with `--sandbox`:
+
+```bash
+pyfuse sandbox setup
+pyfuse worker --backend redis://localhost:6379 --sandbox
+```
+
+No changes are needed on the client side — sandboxing is transparent. See the [Sandbox guide](docs/SANDBOX.md) for more information.
+
 ## Examples
 
 ```bash
@@ -108,6 +120,7 @@ pyfuse run examples/remote_execution.py
 ## Documentation
 
 - **[Quick Start](docs/QUICK_START.md)** -- Usage guide with detailed examples
+- **[Sandbox](docs/SANDBOX.md)** -- Running workers in Docker containers
 - **[Technical Overview](docs/TECHNICAL_OVERVIEW.md)** -- Architecture, serialization format, and internals
 
 ## License

docs/CONTEXT.md

@@ -31,10 +31,16 @@ pyfuse/
     ├── remote.py            # connect(), disconnect(), serve(), submit_remote() (async)
     ├── result.py            # Result (awaitable future), ResultEnvelope
     ├── deps.py              # Third-party dependency extraction and pip installation (async)
-    └── backends/
-        ├── base.py          # Backend ABC: async transport interface
-        ├── redis.py         # RedisBackend: redis.asyncio with RPUSH/BLPOP pattern
-        └── local.py         # LocalBackend: async-native TCP for same-machine IPC
+    ├── backends/
+    │   ├── base.py          # Backend ABC: async transport interface
+    │   ├── redis.py         # RedisBackend: redis.asyncio with RPUSH/BLPOP pattern
+    │   └── local.py         # LocalBackend: async-native TCP for same-machine IPC
+    └── sandbox/
+        ├── __init__.py      # re-exports DockerSandbox
+        ├── docker.py        # DockerSandbox: Docker container isolation
+        ├── guest_agent.py   # Stdlib-only agent deployed inside container
+        ├── _protocol.py     # Length-prefixed JSON wire protocol
+        └── Dockerfile       # Docker image for the guest agent
 ```
 
 ## Architecture overview
@@ -60,6 +66,7 @@ Handles remote execution. Built entirely on `asyncio`:
 - **`result.py`**: `Result` is an awaitable future returned by `.start()`. Simple async polling loop for stall detection. Supports `cancel()` and `progress()` methods.
 - **`deps.py`**: Package installation via `asyncio.create_subprocess_exec`.
 - **`backends/`**: All backend methods are `async def`. `listen()` and `subscribe_results()` are async generators.
+- **`sandbox/`**: Optional Docker-based execution isolation. `DockerSandbox` boots a container, connects to a stdlib-only `guest_agent.py` over TCP (length-prefixed JSON), and delegates code execution. When `--sandbox` is off, the worker runs code directly in the host process.
 
 ## Data flow
 
@@ -203,3 +210,6 @@ pytest                          # test suite
 - `_capture_closure()` in `graph.py` uses a multi-tier strategy: repr validation → traced functions → lambdas (source extraction) → non-traced user functions (auto-registration) → constructor expressions (defaultdict/Counter/deque) → pickle fallback → warning. Returns function objects for auto-registration.
 - `_set_class_metadata()` in `graph.py` captures class-level attributes and decorators from the class source AST. Called from both `_auto_register_class` and `_discover_self_call_deps` to handle both constructor-discovered and directly-traced method classes.
 - `_resolve_class_bases()` now also extracts class definition keywords (e.g., `metaclass=ABCMeta`) and adds necessary imports for keyword values.
+- `guest_agent.py` is intentionally stdlib-only so it can be deployed into containers without installing pyfuse. It duplicates `_protocol.py` wire helpers for this reason.
+- `DockerSandbox` is an async context manager (`__aenter__`/`__aexit__`). The `Worker` calls `start()`/`stop()` at lifecycle boundaries.
+- `DockerSandbox` auto-builds the Docker image from the bundled `Dockerfile` on first use.

docs/QUICK_START.md

@@ -257,6 +257,12 @@ pyfuse worker --backend redis://localhost:6379 --no-auto-install
 
 # Run in an isolated temporary venv (auto-cleaned on exit)
 pyfuse worker --backend redis://localhost:6379 --tmp
+
+# Run tasks inside a Docker sandbox
+pyfuse worker --backend redis://localhost:6379 --sandbox docker
+
+# Run tasks inside a tart micro-VM (macOS Apple Silicon)
+pyfuse worker --backend redis://localhost:6379 --sandbox vm
 ```
 
 Or start a worker programmatically:
@@ -268,6 +274,24 @@ import pyfuse
 asyncio.run(pyfuse.serve("redis://localhost:6379", concurrency=4))
 ```
 
+## Sandboxed execution
+
+By default, workers run tasks in the host process. For security or isolation, you can run tasks inside Docker containers or tart micro-VMs. Sandboxing is fully transparent to clients.
+
+### Quick setup
+
+```bash
+# Docker (any platform)
+pyfuse sandbox setup --docker
+pyfuse worker --backend redis://localhost:6379 --sandbox docker
+
+# tart VM (macOS Apple Silicon only)
+pyfuse sandbox setup
+pyfuse worker --backend redis://localhost:6379 --sandbox vm
+```
+
+See the [Sandbox guide](SANDBOX.md) for full setup and management instructions.
+
 ## Running scripts in a temporary venv
 
 The `run` command creates an isolated venv, auto-detects third-party dependencies from the script (including `install_package_as` blocks), installs them, and runs the script:

docs/SANDBOX.md

@@ -0,0 +1,123 @@
+# Sandbox
+
+Run worker tasks inside isolated Docker containers. Sandboxing is transparent to clients -- no code changes required on the sender side.
+
+## Overview
+
+By default, workers execute tasks in the host process. With `--sandbox`, execution is delegated to a guest agent running inside a Docker container. The worker still handles caching, dependency resolution, and retries on the host; only the `compile()` → `exec()` → call step moves into the sandbox.
+
+## Requirements
+
+- Docker installed and running (`docker info` should succeed)
+- The current user can run `docker` commands (docker group or rootless Docker)
+
+## Setup
+
+```bash
+pyfuse sandbox setup
+```
+
+This builds the `pyfuse-sandbox` Docker image from the bundled Dockerfile. The image is based on `python:3.12-slim` and contains only the stdlib-only guest agent -- no pyfuse installation needed inside the container.
+
+You can also build manually:
+
+```bash
+bash scripts/setup_sandbox_docker.sh
+```
+
+Or let the worker build automatically on first use (the image is built lazily if it doesn't exist).
+
+## Start a sandboxed worker
+
+```bash
+pyfuse worker --backend redis://localhost:6379 --sandbox
+```
+
+The worker starts a container on first task, keeps it running for the worker's lifetime, and stops it on shutdown.
+
+## Customization
+
+Override the image and container names via environment variables:
+
+```bash
+export PYFUSE_SANDBOX_DOCKER_IMAGE=my-custom-image
+export PYFUSE_SANDBOX_DOCKER_CONTAINER=my-sandbox
+pyfuse worker --backend redis://localhost:6379 --sandbox
+```
+
+## Management commands
+
+```bash
+# Check sandbox status
+pyfuse sandbox status
+
+# Remove the Docker container and image
+pyfuse sandbox teardown
+```
+
+Example `pyfuse sandbox status` output:
+
+```
+Docker:
+  docker: installed
+  Image 'pyfuse-sandbox': exists
+  Container 'pyfuse-sandbox': not found
+```
+
+## Programmatic usage
+
+```python
+from pyfuse.worker.sandbox import DockerSandbox
+
+async with DockerSandbox() as sandbox:
+    result = await sandbox.execute(source, "my_func", (arg1,), {})
+```
+
+Pass `sandbox=True` to `serve()`:
+
+```python
+await pyfuse.serve("redis://localhost:6379", sandbox=True)
+```
+
+Or provide a `DockerSandbox` instance for custom settings:
+
+```python
+from pyfuse.worker.sandbox import DockerSandbox
+
+sandbox = DockerSandbox(cpus=4, memory_gb=4)
+await pyfuse.serve("redis://localhost:6379", sandbox=sandbox)
+```
+
+## Configuration reference
+
+`DockerSandbox` accepts the following keyword arguments:
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `image` | `"pyfuse-sandbox"` | Docker image name |
+| `container_name` | `"pyfuse-sandbox"` | Container name |
+| `guest_port` | `9749` | TCP port the guest agent listens on |
+| `cpus` | `2` | vCPUs allocated to the container |
+| `memory_gb` | `2` | RAM (GB) allocated to the container |
+| `timeout` | `60.0` | Max seconds per function execution |
+| `boot_timeout` | `30.0` | Max seconds to wait for container to start |
+
+## How it works
+
+The sandbox uses a guest agent (`guest_agent.py`) — a lightweight, stdlib-only Python script deployed inside the container. The worker communicates with it over TCP using a length-prefixed JSON protocol:
+
+1. Worker sends the reconstructed source code, function name, and arguments
+2. Guest agent `exec`s the source, calls the function, and returns the result
+3. Errors are serialized with type, message, and traceback
+
+The sandbox stays running between tasks. The first execution incurs a startup cost (container boot + agent connection), but subsequent calls reuse the same connection.
+
+## Troubleshooting
+
+**Image build fails**
+- Ensure Docker daemon is running: `docker info`
+- Check permissions: your user should be in the `docker` group or use rootless Docker
+
+**Sandbox timeout**
+- Increase `timeout` in `DockerSandbox` for long-running functions
+- Default is 60 seconds per execution

docs/TECHNICAL_OVERVIEW.md

@@ -44,6 +44,11 @@ pyfuse/
             base.py          Backend ABC: async pluggable transport interface
             redis.py         RedisBackend: redis.asyncio with RPUSH/BLPOP pattern
             local.py         LocalBackend: async-native TCP for same-machine IPC
+        sandbox/
+            docker.py        DockerSandbox: Docker container isolation
+            guest_agent.py   Stdlib-only agent deployed inside container
+            _protocol.py     Length-prefixed JSON wire protocol
+            Dockerfile       Container image for the guest agent
 ```
 
 ## Remote execution flow
@@ -487,6 +492,74 @@ When a module contains `from X import *`:
 2. Create individual `ImportInfo` entries per exported name.
 3. Filter to only names the function actually uses.
 
+## Sandbox execution
+
+Workers can optionally execute tasks inside an isolated Docker container instead of the host process. This is controlled by the `--sandbox` CLI flag or by passing `sandbox=True` (or a `DockerSandbox` instance) programmatically.
+
+### How it works
+
+When sandboxing is enabled, only the `exec → call` step moves into the container. The worker's own control logic — caching, dependency resolution, retry — stays on the host.
+
+```mermaid
+sequenceDiagram
+    participant Client
+    participant Worker as Worker (host)
+    participant Container as Docker Container
+    participant Agent as Guest Agent
+
+    Client->>Worker: Task (graph JSON + args)
+    Worker->>Worker: Deserialize graph, install deps, reconstruct source
+
+    alt --sandbox enabled
+        Worker->>Container: TCP connect (first use boots container)
+        Worker->>Agent: {source, function_name, args, kwargs}
+        Agent->>Agent: compile() → exec() → call function
+        Agent->>Worker: {status: "ok", result: value}
+    else default (no sandbox)
+        Worker->>Worker: compile() → exec() → call function
+    end
+
+    Worker->>Client: Result
+```
+
+A lightweight **guest agent** (`guest_agent.py`) runs inside the container. It is stdlib-only (no pyfuse install required) and communicates with the worker over TCP using a **length-prefixed JSON protocol** (4-byte big-endian header + UTF-8 JSON payload).
+
+### Container lifecycle
+
+```mermaid
+stateDiagram-v2
+    [*] --> ImageCheck: worker starts with --sandbox
+    ImageCheck --> BuildImage: image missing
+    ImageCheck --> StartContainer: image exists
+    BuildImage --> StartContainer: docker build
+    StartContainer --> WaitForAgent: docker run -d
+    WaitForAgent --> Connected: TCP handshake
+    Connected --> Execute: task arrives
+    Execute --> Connected: result returned
+    Connected --> Stopped: worker shutdown
+    Stopped --> [*]
+```
+
+1. **Start** — `DockerSandbox.start()` builds the image (if absent), starts the container, waits for the guest agent to become reachable, then opens a persistent TCP connection.
+2. **Execute** — Each task is sent as a JSON request. The guest agent `exec`s the source, calls the function, and returns the result. The connection is reused across tasks.
+3. **Stop** — On worker shutdown, the connection is closed and the container is stopped.
+
+### Configuration
+
+`DockerSandbox` accepts the following keyword arguments:
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `image` | `"pyfuse-sandbox"` | Docker image name |
+| `container_name` | `"pyfuse-sandbox"` | Container name |
+| `guest_port` | `9749` | TCP port for the guest agent |
+| `cpus` | `2` | vCPUs allocated to the container |
+| `memory_gb` | `2` | RAM (GB) allocated to the container |
+| `timeout` | `60.0` | Max seconds per function execution |
+| `boot_timeout` | `30.0` | Max seconds to wait for container to start |
+
+Environment variables `PYFUSE_SANDBOX_DOCKER_IMAGE` and `PYFUSE_SANDBOX_DOCKER_CONTAINER` override the image and container names.
+
 ## Limitations
 
 ### Source requirements
@@ -517,6 +590,12 @@ pyfuse worker --backend redis://localhost:6379
 pyfuse worker --backend redis://localhost:6379 -c 4
 pyfuse worker --backend redis://localhost:6379 --no-auto-install
 pyfuse worker --backend redis://localhost:6379 --tmp   # isolated temp venv
+pyfuse worker --backend redis://localhost:6379 --sandbox   # Docker sandbox
+
+# Sandbox management
+pyfuse sandbox setup       # build Docker sandbox image
+pyfuse sandbox status      # show Docker sandbox status
+pyfuse sandbox teardown    # remove Docker sandbox
 
 # Run a script in a temporary venv (auto-detects and installs dependencies)
 pyfuse run examples/script.py

examples/large_module.py

@@ -47,23 +47,18 @@ def full_sensor_report(sensor_count: int, readings_per_sensor: int, seed: int =
         stats, anomalies, normalized,
     )
     report["delta_count"] = len(deltas)
-    return format_text_report(report)
-
+    r = format_text_report(report)
+    print(r)
+    return r
 
 async def main() -> None:
     # Connect to the worker
-    pyfuse.connect("local://localhost:9748")
-
-    # Local call
-    local_result = full_sensor_report(3, 50, seed=42)
+    pyfuse.connect("redis://localhost:6379")
 
     # Remote call (same function, same args, on a worker)
     remote_result = await full_sensor_report.run(3, 50, seed=42)
 
-    if local_result == remote_result:
-        print(f"Success! Local and remote results match:\n  {repr(local_result)[:80] + '...'}")
-    else:
-        print("Mismatch between local and remote results!")
+    print(f"Success! Local and remote results match:\n  {repr(remote_result)[:80] + '...'}")
 
 
 if __name__ == "__main__":

pyfuse/__init__.py

@@ -15,6 +15,7 @@
 from pyfuse.worker.deps import install_package_as
 from pyfuse.worker.remote import connect, disconnect, serve
 from pyfuse.worker.result import Result, ResultEnvelope
+from pyfuse.worker.sandbox import DockerSandbox
 from pyfuse.worker.worker import Worker
 from pyfuse.worker.worker import execute as execute
 
@@ -97,4 +98,6 @@ def pack(func: Callable[..., object], *args: Any, **kwargs: Any) -> Task:
     "Task",
     "Worker",
     "Backend",
+    # Sandbox
+    "DockerSandbox",
 ]