Skip to content

Commit 87f2ab4

Browse files
codeSamuraiicodeSamuraii
committed
Update documentation
1 parent 28238ff commit 87f2ab4

3 files changed

Lines changed: 80 additions & 16 deletions

File tree

‎docs/AGENTS.md‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ Add `@offwork.task` to one entry-point function. Call `await func.run(...)`. Tha
2626
| Auto-capture (source, imports, closures, classes, module vars) | `Graph.serialize` | [offwork/graph/analyzer.py](../offwork/graph/analyzer.py), [offwork/graph/graph.py](../offwork/graph/graph.py) |
2727
| Reconstruction → self-contained source | `Graph.reconstruct` | [offwork/graph/store.py](../offwork/graph/store.py) |
2828
| Runtime call-stack tracing | `contextvars` | [offwork/graph/tracing.py](../offwork/graph/tracing.py) |
29-
| Remote submit / await | `func.run`, `func.start`, `func.map` | [offwork/worker/remote.py](../offwork/worker/remote.py), [offwork/graph/decorator.py](../offwork/graph/decorator.py) |
29+
| Remote submit / await | `func.run`, `func.submit`, `func.map` | [offwork/worker/remote.py](../offwork/worker/remote.py), [offwork/graph/decorator.py](../offwork/graph/decorator.py) |
3030
| Worker loop (signing, scheduling, throttle, heartbeat) | `serve` | [offwork/worker/remote.py](../offwork/worker/remote.py) |
3131
| Subgraph caching, reconstruct, retry, timeout | `Worker.run_with_policy` | [offwork/worker/worker.py](../offwork/worker/worker.py) |
3232
| Auto-install of third-party packages | `install_package_as`, `ensure_dependencies` | [offwork/worker/deps.py](../offwork/worker/deps.py) |
@@ -39,6 +39,7 @@ Add `@offwork.task` to one entry-point function. Call `await func.run(...)`. Tha
3939
| Local TCP backend | `local://` | [offwork/worker/backends/local.py](../offwork/worker/backends/local.py) |
4040
| Redis backend | `redis://` | [offwork/worker/backends/redis.py](../offwork/worker/backends/redis.py) |
4141
| RabbitMQ backend | `amqp://` | [offwork/worker/backends/rabbitmq.py](../offwork/worker/backends/rabbitmq.py) |
42+
| WebSocket backend (hosted broker) | `ws://` / `wss://` | [offwork/worker/backends/ws.py](../offwork/worker/backends/ws.py) |
4243
| Docker sandbox isolation | `--sandbox`, `DockerSandbox` | [offwork/worker/sandbox/](../offwork/worker/sandbox/) |
4344
| Signed envelopes (per-client HMAC + Ed25519 + TOFU + replay protection) | `--require-signing`, token, pairing, `offwork clients` | [offwork/core/envelope.py](../offwork/core/envelope.py), [offwork/core/ed25519.py](../offwork/core/ed25519.py), [offwork/core/identity.py](../offwork/core/identity.py), [offwork/core/clients.py](../offwork/core/clients.py), [offwork/core/signing.py](../offwork/core/signing.py), [offwork/core/token.py](../offwork/core/token.py), [offwork/core/pairing.py](../offwork/core/pairing.py) |
4445
| Temp venv (for `--tmp` and `offwork run`) | `temp_venv` | [offwork/_venv.py](../offwork/_venv.py) |
@@ -88,6 +89,7 @@ offwork/
8889
local.py Async TCP broker (auto-spawned subprocess).
8990
redis.py redis.asyncio (RPUSH/BLPOP, Pub/Sub, MGET).
9091
rabbitmq.py aio-pika (durable queue, fanout exchange, TTL queues).
92+
ws.py WebSocketBackend — one persistent WS, multiplexed by request id.
9193
sandbox/
9294
docker.py DockerSandbox: build image, start container, TCP exec.
9395
guest_agent.py Stdlib-only agent running inside the container.
@@ -128,14 +130,21 @@ The `__all__` in [offwork/__init__.py](../offwork/__init__.py) is the public sur
128130

129131
- Decorator: `task`.
130132
- Lifecycle: `connect(url)`, `disconnect()`, `serve(url, concurrency=, sandbox=, ...)`.
133+
`connect()` accepts `local://`, `redis://`, `rediss://`, `amqp://`, `amqps://`,
134+
`ws://`, `wss://`. The `ws://` / `wss://` schemes instantiate `WebSocketBackend`
135+
(requires `pip install offwork[ws]`).
131136
- Power-user: `Task`, `Worker`, `Backend`, `serialize`, `reconstruct`, `pack`, `execute`, `get_graph`, `Graph`.
132137
- Result: `Result`, `ResultEnvelope`, `ProgressInfo`, `progress`.
133138
- Scheduling: `ScheduleHandle`.
134139
- Errors: `Error` (base), `WorkerError`, `RemoteError`, `DependencyError`, `TaskStalled`, `TaskCancelled`, `ThrottleError`, `SignatureError` and its subclasses `ReplayError`, `StaleTaskError`, `ClientRevokedError`, `IdentityMismatchError`, `PairingError`, `WorkerOnlyError`.
135140
- Auth: `generate_token`, `save_token`, `load_token`, `clear_token`, `resolve_root_token`, `compute_signature`, `verify_signature`, `derive_key`, `NonceLRU`, `build_signed_envelope`, `verify_task_envelope`, `KnownClients`, `ClientEntry`, `get_client_id`, `get_identity_seed`, `get_public_key`, `get_identity_fingerprint`, `clear_identity`, plus pairing helpers.
136141
- Sandbox: `DockerSandbox`.
137142

138-
`func.run`, `func.start`, `func.map`, `func.run_in`, `func.run_at`, `func.run_every` are attributes attached by `@offwork.task` ([graph/decorator.py](../offwork/graph/decorator.py)).
143+
`func.run`, `func.submit`, `func.map`, `func.run_in`, `func.run_at`,
144+
`func.run_every` are attributes attached by `@offwork.task`
145+
([graph/decorator.py](../offwork/graph/decorator.py)).
146+
`func.submit` is the non-blocking form of `func.run` — it returns a
147+
`Result` handle without awaiting the result.
139148

140149
## Conventions and invariants
141150

‎docs/FEATURES.md‎

Lines changed: 34 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ Full feature guide and API walkthrough for offwork. For architecture internals s
88
pip install offwork
99
pip install offwork[redis] # Redis backend (multi-machine)
1010
pip install offwork[rabbitmq] # RabbitMQ backend (multi-machine, AMQP)
11+
pip install offwork[ws] # WebSocket backend (hosted cloud broker)
1112
```
1213

1314
offwork has zero required runtime dependencies. Backend extras are only needed when you use the corresponding URL scheme.
@@ -44,9 +45,9 @@ python my_script.py # Terminal 2 → 5.0
4445
## Async API
4546

4647
```python
47-
result = await func.run(3.0, 4.0) # submit + await
48+
result = await func.run(3.0, 4.0) # submit + await result
4849

49-
future = await func.submit(3.0, 4.0) # submit, get handle
50+
future = await func.submit(3.0, 4.0) # submit, get Result handle
5051
result = await future # await later
5152

5253
results = await func.map([(3, 4), (5, 12)]) # batch
@@ -56,6 +57,10 @@ r1, r2 = await asyncio.gather(func.run(3, 4), func.run(5, 12)) # concurrent
5657

5758
`async def` functions are awaited directly on the worker.
5859

60+
`func.submit()` is the non-blocking form of `func.run()`. It enqueues the task
61+
and returns a `Result` handle immediately, letting you do other work while the
62+
task executes and `await` the result later.
63+
5964
## Retry and timeout
6065

6166
```python
@@ -229,15 +234,42 @@ After setup, tasks are signed automatically. No client-side code changes. See [S
229234
| Local | `local://host:port` | (built-in) | Same-machine IPC (async TCP, no deps) |
230235
| Redis | `redis://host:port` | `pip install offwork[redis]` | Multi-machine production |
231236
| RabbitMQ | `amqp://host:port` | `pip install offwork[rabbitmq]` | Multi-machine production with AMQP |
237+
| WebSocket | `ws://host/path` or `wss://` | `pip install offwork[ws]` | Hosted cloud broker (one persistent socket) |
232238

233239
```python
234240
offwork.connect("local://localhost:9748")
235241
offwork.connect("redis://localhost:6379")
236242
offwork.connect("amqp://guest:guest@localhost/")
243+
offwork.connect("ws://localhost:8000/api/v1/broker/ws?api_key=<key>") # hosted
237244
```
238245

239246
Or: `export OFFWORK_BACKEND=redis://localhost:6379`
240247

248+
### Hosted broker (WebSocket)
249+
250+
When connecting to a hosted broker such as cloud_poc, use a `ws://` or `wss://`
251+
URL. `WebSocketBackend` opens one persistent socket per process and multiplexes
252+
all broker operations over it by request id. Authentication is via
253+
`?api_key=<key>` in the URL (stripped from the URL and sent in the handshake).
254+
255+
```python
256+
import offwork
257+
258+
# URL returned by /api/v1/users/register or /api/v1/users/me
259+
offwork.connect("wss://example.com/api/v1/broker/ws?api_key=<your key>")
260+
261+
@offwork.task
262+
def hello(name: str) -> str:
263+
return f"hello {name}"
264+
265+
async def main():
266+
print(await hello.run("world")) # executes on a cloud worker pod
267+
```
268+
269+
Reconnect is automatic with bounded backoff. Mutating ops that were in-flight
270+
when the socket dropped surface as `ConnectionError` — the caller decides
271+
whether to retry.
272+
241273
## Worker options
242274

243275
```bash

‎docs/TECHNICAL_OVERVIEW.md‎

Lines changed: 35 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,7 @@ offwork/
5656
redis.py RedisBackend: redis.asyncio (RPUSH/BLPOP, Pub/Sub, MGET)
5757
local.py LocalBackend: async TCP broker for same-machine IPC
5858
rabbitmq.py RabbitMQBackend: aio-pika (durable queue, fanout exchange)
59+
ws.py WebSocketBackend: one persistent WS, request-id multiplexing
5960
sandbox/
6061
docker.py DockerSandbox: build image, start container, TCP exec
6162
guest_agent.py Stdlib-only agent running inside the container
@@ -74,15 +75,15 @@ offwork/
7475
# Remote execution (all async)
7576
offwork.connect("redis://localhost:6379") # configure backend (sync)
7677
await offwork.serve("redis://...", concurrency=4) # start worker loop
77-
await func.run(*args) # submit + await result
78-
future = await func.start(*args) # submit, returns Result handle
79-
results = await func.map([(a1, b1), ...]) # batch submit + await all
78+
await func.run(*args) # submit + await result
79+
future = await func.submit(*args) # submit, returns Result handle
80+
results = await func.map([(a1, b1), ...]) # batch submit + await all
8081

8182
# Scheduling
82-
await func.run_in(timedelta(minutes=5), *args) # execute after delay
83-
await func.run_at(datetime(2026, 1, 1), *args) # execute at specific time
84-
schedule = await func.run_every(timedelta(hours=1), *args) # recurring
85-
await schedule.cancel() # stop recurring
83+
await func.run(*args, run_in=timedelta(minutes=5)) # execute after delay
84+
await func.run(*args, run_at=datetime(2026, 1, 1)) # execute at specific time
85+
schedule = await func.submit(*args, run_every=timedelta(hours=1)) # recurring
86+
await schedule.cancel() # stop recurring
8687

8788
# Result handling
8889
result = await future # await result value
@@ -108,12 +109,12 @@ offwork.get_graph().to_mermaid(func) # -> Mermaid diagram string
108109

109110
## Remote execution flow
110111

111-
### Client side: `await func.run(*args)` / `await func.start(*args)`
112+
### Client side: `await func.run(*args)` / `await func.submit(*args)`
112113

113114
1. **Serialize** -- `Graph.serialize(func)` captures the function's subgraph (source, imports, dependencies) as JSON.
114115
2. **Pack** -- A `Task` envelope bundles the serialized graph, function name, arguments, and execution options (timeout, retries).
115116
3. **Submit** -- `await backend.submit(task_json)` sends the task to the transport layer.
116-
4. **Return** -- `.run()` awaits the result and returns the value directly. `.start()` returns a `Result` handle immediately.
117+
4. **Return** -- `.run()` awaits the result and returns the value directly. `.submit()` returns a `Result` handle immediately.
117118

118119
### Worker side: `await serve()` / `offwork worker`
119120

@@ -199,13 +200,35 @@ Uses `aio-pika` (async AMQP 0-9-1). Tasks go through a single durable queue (`of
199200

200201
URL scheme: `amqp://` or `amqps://` (e.g. `amqp://guest:guest@localhost/`). The `aio-pika` package is an optional dependency installed via `pip install offwork[rabbitmq]`.
201202

203+
### WebSocketBackend
204+
205+
Single persistent WebSocket to a hosted broker (e.g. cloud_poc's
206+
`/api/v1/broker/ws`). All ops are multiplexed over one socket by request id;
207+
no TCP handshake per call.
208+
209+
- URL schemes: `ws://` and `wss://`
210+
- Install: `pip install offwork[ws]` (uses `websockets >= 15.0`)
211+
- Authentication: `?api_key=<key>` in the URL, stripped and sent in the
212+
handshake `hello` frame
213+
- Reconnects automatically with bounded backoff (0.5 s → 30 s). Mutating ops
214+
in-flight when the socket drops surface as `ConnectionError`; the backend
215+
never silently replays them
216+
- Workers opened via `serve()` or `offwork worker --backend ws://...`
217+
connect with `role="worker"` in the handshake
218+
219+
```python
220+
offwork.connect("wss://example.com/api/v1/broker/ws?api_key=<key>")
221+
```
222+
223+
The `websockets` package is imported lazily and is an optional dependency.
224+
202225
### Custom backends
203226

204-
Subclass `Backend` to implement any transport (HTTP, NATS, gRPC, etc.):
227+
Subclass `Backend` to implement any transport (NATS, gRPC, etc.):
205228

206229
```python
207230
offwork.connect("redis://...") # built-in
208-
await func.start(*args, backend=my_custom_backend) # per-call override
231+
await func.submit(*args, _backend=my_custom_backend) # per-call override
209232
```
210233

211234
## How `@offwork.task` works
@@ -269,7 +292,7 @@ For generators and async generators, a proxy pattern intercepts each iteration s
269292

270293
The returned wrapper gains:
271294
- `.run(*args)` -- submit to remote worker and await result (coroutine)
272-
- `.start(*args)` -- submit to remote worker, returns `Result` handle (coroutine)
295+
- `.submit(*args)` -- submit to remote worker, returns `Result` handle (coroutine)
273296
- `.map(args_list)` -- batch submit and await all results (coroutine)
274297
- `__offwork_traced__ = True` -- marker attribute
275298

0 commit comments

Comments
 (0)