|
| 1 | +# Fleet Delivery — Agents, Nodes, and Reliable Messaging |
| 2 | + |
| 3 | +**Status**: Draft |
| 4 | +**Date**: 2026-06-06 |
| 5 | +**Author**: Design session (Will + Claude) |
| 6 | + |
| 7 | +--- |
| 8 | + |
| 9 | +## 1. Vision |
| 10 | + |
| 11 | +Run agents across many machines. Each machine (a **node**) can run a specific set of compute — some can spawn Claude agents, some Codex, some both. Relaycast is the control plane: agents are equal peers in a flat messaging fabric, nodes advertise what they can spawn, and Relaycast routes messages to agents wherever they live and places new agents onto nodes that can run them. |
| 12 | + |
| 13 | +The goal is the simplest deployable unit that maximizes the environments a node can run in, with messaging that survives flaky networks and agent restarts. |
| 14 | + |
| 15 | +## 2. The frame: two planes |
| 16 | + |
| 17 | +Everything below lives on one of two planes. Keeping them separate is what keeps the model simple. |
| 18 | + |
| 19 | +- **Messaging fabric — flat, all equal `agents`.** Every agent is a peer: a stable identity that sends, receives, and may expose actions. No agent is above or routed "through" another. |
| 20 | +- **Compute layer — `nodes`.** A node is a machine where some agents run. It has a **broker** runtime and a set of **capabilities** (what it can spawn). Agents sharing a node is a deployment fact, not a relationship in the fabric. |
| 21 | + |
| 22 | +There is **no "participant" umbrella and no agent subtype.** "Orchestrate vs communicate" is not a type distinction — it's just whether an agent is colocated on a node-with-broker or self-connected (see *location*, §5). |
| 23 | + |
| 24 | +## 3. Core concepts |
| 25 | + |
| 26 | +| Concept | What it is | Plane | |
| 27 | +|---|---|---| |
| 28 | +| **Agent** | a peer in the fabric: identity, send/receive, exposed actions | messaging | |
| 29 | +| **Node** | a named machine that runs agents and advertises capabilities | compute | |
| 30 | +| **Broker** | a node's runtime/delivery engine — **infra, not a peer** (not in the agent roster) | compute | |
| 31 | +| **Location** | where Relaycast routes an agent's **inbound** | routing detail | |
| 32 | +| **Capability** | what a node can spawn (e.g. `spawn:codex`) | compute | |
| 33 | +| **Action** | something an agent exposes/invokes in the fabric | messaging | |
| 34 | + |
| 35 | +The broker is node infrastructure with a control connection to Relaycast. A colocated PTY agent receiving via its broker is the same kind of plumbing as a NIC delivering to a process — a location, not a hierarchy. |
| 36 | + |
| 37 | +## 4. Identities & naming |
| 38 | + |
| 39 | +- **Agent**: stable `agent_id` + workspace-unique **name** (the addressable handle). One **active location** per agent name (a second live claimant is rejected; migration is explicit). |
| 40 | +- **Node**: workspace-unique **name**, operator-set at startup (default: hostname), optionally backed by a stable internal id so a name can move to a replacement machine. Same uniqueness rule as agents: one live owner. |
| 41 | +- The node's broker is the **token authority** for agents it spawns: it asks Relaycast to mint the agent identity + token on spawn, hands the token to the agent, and binds its location. |
| 42 | + |
| 43 | +## 5. Delivery model — keep + delete |
| 44 | + |
| 45 | +Most of this already exists; the work is mostly removal. |
| 46 | + |
| 47 | +**Outbound (all agents): direct & stateless.** An agent sends with its own token straight to Relaycast (PTY agents via their MCP send tools; SDK agents via their own send). Sends are request/response — **no persistent connection required for sending**, and the broker is never in the send path. |
| 48 | + |
| 49 | +**Inbound: delivered to the agent's location.** Location has exactly two shapes — a field, not a type: |
| 50 | +- **Self-connected** (event-loop programs — SDK agents): the agent's own WS + message handler. This *is* the delivery path. |
| 51 | +- **Via its node** (raw PTY harnesses with no event loop): the node's broker receives and injects into the agent's stdin. |
| 52 | + - `steer` = inject + interrupt to a prompt now. |
| 53 | + - `wait` = write to the buffer; the harness reads at its next prompt (the PTY defers naturally). |
| 54 | + |
| 55 | +**Invariant: an agent has exactly one location.** This is the whole cleanup — the old redundancy was a via-node agent *also* holding its own WS. One location → no double delivery, no special-casing. |
| 56 | + |
| 57 | +**Delete (these only ever applied to PTY agents):** |
| 58 | +- The per-agent Relaycast WS (`RealtimeResourceBridge`). |
| 59 | +- The MCP **resource layer** (`relay://inbox`, `relay://channels/...`, subscribe/notify) — it assumes a reactive client; turn-based harnesses don't subscribe. |
| 60 | +- The **inbox piggyback** (stapling inbox onto every tool result). |
| 61 | + |
| 62 | +**Keep:** on-demand read/query tools (`check_inbox`, `list_messages`, `thread`, `search`, `list_channels`) as **stateless cloud-direct reads** with the agent token. Pulling on your own initiative doesn't need a persistent connection. The MCP server for PTY agents becomes outbound + reads only. |
| 63 | + |
| 64 | +**Consistency bolt:** delivery acknowledgement marks a message delivered/read in Relaycast, so a cloud-backed `check_inbox` never re-surfaces something already delivered. One source of truth for *history* (cloud), one for *delivery* (the location); the ack bridges them. |
| 65 | + |
| 66 | +**Why the deleted layers existed (so we don't rebuild them):** the original design was MCP-idiom-first — inbox/channels as subscribable resources, the textbook way to surface stateful data. Turn-based harnesses didn't react to `resources/updated`, so the piggyback was bolted on, and stdin injection became the reliable push. Nothing was removed, leaving three overlapping inbound paths. **Lesson: design delivery around the agent's execution model (turn-based vs event-loop), not the protocol's idiom.** |
| 67 | + |
| 68 | +## 6. Spawn & placement |
| 69 | + |
| 70 | +**Spawn is not a protocol concept — it's a node capability**, expressed through the existing action mechanism (`actions.register('create', handler → driver.spawn)`). The "how spawning happens here" is a node-side harness definition (`definePtyHarness` / `StaticPtyHarnessDefinition`) — the script you provide when spinning up a node. A node advertises the capabilities it defines (e.g. `spawn:claude`, `spawn:codex`). |
| 71 | + |
| 72 | +**Placement** takes an optional target: |
| 73 | + |
| 74 | +``` |
| 75 | +spawn { capability, node?: <name> | "self", session_ref?, ttl_override? } |
| 76 | +
|
| 77 | +eligible = nodes where |
| 78 | + (node.name == target if target given) |
| 79 | + ∧ capability ∈ node.capabilities |
| 80 | + ∧ node.live ∧ capacity_available |
| 81 | +place: target if given, else least-loaded(eligible) |
| 82 | +``` |
| 83 | + |
| 84 | +- `node: "gpu-box-1"` → must place there. Capability mismatch → **hard fail**. Offline → bounded-queue (or fail-fast per override). |
| 85 | +- `node: "self"` → same node as the requester (the common **colocation** case: shared working dir, local artifacts). An agent needn't know its node's name. |
| 86 | +- `node` omitted → scheduler picks any eligible (least-loaded). |
| 87 | +- None eligible → bounded-queue, then fail. |
| 88 | + |
| 89 | +**Resume is a special case of targeted spawn.** "Resume agent X" = spawn with `node: <origin node>` + its `session_ref`. There is no separate resume concept — it is placement constrained to the origin node plus a session reference (see §8.2). |
| 90 | + |
| 91 | +**Node roster:** because agents/humans/schedulers can target by name, Relaycast exposes a **node discovery query** (name, capabilities, liveness, load) — the compute-layer roster, parallel to the agent roster, and what a UI like Pear renders. |
| 92 | + |
| 93 | +## 7. Reliable action invocation (spawn rides on this) |
| 94 | + |
| 95 | +Spawn inherits the action system's async invocation machinery (`invocationId`, ack, result) rather than a bespoke state machine. Exactly-once placement is impossible (dispatch, node dies before ack — did it start?), so the contract is **idempotency + at-least-once + reconcile**: |
| 96 | + |
| 97 | +- `invocationId` is the idempotency key. A node dedups invocations by it; a requester retrying with the same id never double-spawns. |
| 98 | +- Invocation lifecycle: `pending → dispatched(node) → completed(agent_id)`. |
| 99 | +- **Dispatch timeout / node lost** before completion → **reschedule** to another eligible node with the same `invocationId`. |
| 100 | +- **Reconcile on reconnect:** a node re-announces its live agent inventory (with `agent_id`, name, `invocationId`, `session_ref`) — see §9. If an invocation already completed elsewhere → the duplicate is released. **First to `completed` wins.** A dead broker brings no agents back (its children died with it), so the dead-node case reschedules cleanly; duplicates only arise from a live-broker uplink blip and are reconciled away. |
| 101 | + |
| 102 | +## 8. Durability — bounded-durable mailbox |
| 103 | + |
| 104 | +**Decision: bounded-durable.** A message for an unreachable agent is held for a TTL and delivered when it's reachable again; dead-lettered after. Reliable without infinite state. |
| 105 | + |
| 106 | +### 8.1 Message state machine (held in Relaycast) |
| 107 | + |
| 108 | +``` |
| 109 | +queued ──deliver(seq)──▶ delivered ──ack──▶ acked (≈read) |
| 110 | + │ |
| 111 | + └── TTL expiry ──▶ dead-letter |
| 112 | +``` |
| 113 | + |
| 114 | +- **At-least-once + dedup by `msg_id`.** Per-agent ordering falls out of a monotonic per-location `seq`. |
| 115 | +- Relaycast pushes `queued` messages to the agent's location with a per-session `seq`; the location injects/handles, then **cumulative ack** (`up_to_seq`) advances them to `acked`. |
| 116 | + |
| 117 | +### 8.2 Identity continuity requires session continuity |
| 118 | + |
| 119 | +Reclaiming a mailbox without resuming the actual session would dump a backlog on a context-less process — worse than dead-lettering. So an agent reclaims its identity + held mailbox **only by resuming its session** (= origin-targeted spawn + `session_ref`, §6): |
| 120 | + |
| 121 | +- **`resumable`** is a per-harness capability; on spawn of a resumable harness the broker captures the `session_ref` and reports it to Relaycast. |
| 122 | +- **Resume is node-local (v1):** harness session state lives on the origin node's disk, so resumable agents are **node-sticky**. If the origin node is permanently gone, the session is unrecoverable → identity terminal → mailbox dead-lettered. (Cross-node resume needs cloud-synced session state — deferred, §10.) |
| 123 | +- No resumable capability / no recoverable session → respawn is a **new identity**; the old mailbox is dead-lettered (senders notified). |
| 124 | + |
| 125 | +### 8.3 Mailbox resolution |
| 126 | + |
| 127 | +| Situation | Resolution | |
| 128 | +|---|---| |
| 129 | +| Location temporarily unreachable (uplink/WS blip; process alive) | Hold + deliver on reconnect. Applies to **all** agents. | |
| 130 | +| Process dead, resumable + session recoverable | Hold up to TTL; flush on origin-targeted resume (oldest-first). | |
| 131 | +| Process dead, non-resumable OR session lost | **Dead-letter immediately** (notify senders). | |
| 132 | + |
| 133 | +Consequence: persistence *across process death* is a **resumable-only** property. Non-resumable agents are ephemeral-on-death (still resilient to transient blips while alive). |
| 134 | + |
| 135 | +### 8.4 Where state lives |
| 136 | + |
| 137 | +- **Relaycast** holds all durable state (source of truth): mailboxes, agent records (`resumable`, `session_ref`, origin node), locations, node registry. Must survive Relaycast restarts. |
| 138 | +- **Broker** keeps only in-memory per-session state: `seq` cursor, dedup set, local pending-injection queue. **No disk needed for delivery durability.** |
| 139 | + - Uplink blip, broker alive → cursor/dedup survive → clean replay, no duplicates. |
| 140 | + - Broker process dies → its child agents die too → they respawn and *want* redelivery → redelivery is correct, not duplicate. |
| 141 | + |
| 142 | +### 8.5 One durable store |
| 143 | + |
| 144 | +The per-agent mailbox **subsumes** any per-node replay buffer: node-disconnect replay is just "redeliver this node's agents' unacked mail on reconnect." The per-location `seq` + ack is the at-least-once transport on top of the mailbox. |
| 145 | + |
| 146 | +### 8.6 Policy (defaults, all tunable) |
| 147 | + |
| 148 | +- **TTL:** workspace default (~1h placeholder) + per-message override ("5m or drop" for time-sensitive; longer for durable tasks). |
| 149 | +- **Dead-letter on expiry:** retain briefly + **emit `delivery_failed`/expired to the sender** (reuse existing event). Silent drop is the wrong default. |
| 150 | +- **Mailbox overflow:** **reject-new** with sender feedback (so senders learn the agent is backed up) rather than drop-oldest. |
| 151 | +- **Inbound to a down-but-resumable agent:** **lazy by default** (queue; resume via restart-policy or explicit respawn), **eager opt-in** ("wake on message"). |
| 152 | +- **Restart policy:** a resumable agent with auto-restart → broker auto-resumes the session on its node and flushes the mailbox. |
| 153 | + |
| 154 | +## 9. Node lifecycle & control surface |
| 155 | + |
| 156 | +A node's broker holds one control connection to Relaycast, serving two roles: **compute provider** (advertises capabilities, receives spawn/release action invocations, reports results) and **delivery relay** (receives inbound for the PTY agents located on it, injects, acks). |
| 157 | + |
| 158 | +- **Register** (on connect): node name, capabilities, version, `max_agents`, tags, and a resume cursor for replay. |
| 159 | +- **Heartbeat** (~10–15s): `load`, `active_agents`. Relaycast TTL marks offline → stop placing there; mark its located agents unreachable. |
| 160 | +- **Reconnect inventory sync:** after register, the broker re-announces its full live agent inventory (`agent_id`, name, `invocationId`, `session_ref`). Relaycast reconciles **locations** and open **invocations** (§7) from it. |
| 161 | +- **Deregister:** graceful on shutdown; else liveness TTL. |
| 162 | + |
| 163 | +**Narrow control surface** (the only Relaycast protocol the broker implements — *not* the full `@relaycast/sdk`; channels/threads/reactions/search stay in the agent SDK): |
| 164 | + |
| 165 | +- **Broker → Relaycast:** `node.register`, `node.heartbeat`, `node.deregister`, `agent.register` (bind location), `agent.deregister`, `delivery.ack`, `action.result`, `inventory.sync` |
| 166 | +- **Relaycast → Broker:** `action.invoke` (spawn/release are actions), `deliver`, `ping` |
| 167 | + |
| 168 | +## 10. Deferred / open |
| 169 | + |
| 170 | +- **Cross-node session resume** via cloud-synced session state (lifts node-sticky in §8.2). |
| 171 | +- **Same-node fast-path (perf):** local A→B delivery bypassing the cloud, and whether it's allowed when the uplink is down. |
| 172 | +- **Node tags / fuzzy targeting** (`gpu` instead of an exact name) and **access control** on who can target / spawn on which nodes. |
| 173 | +- **Exact tunables:** TTL, mailbox depth cap, heartbeat interval/TTL, dispatch timeout. |
| 174 | +- **Mark-read mechanism:** broker auto-mark on delivery (preferred) vs explicit `mark_read` tool — keep explicit only if a product reason emerges. |
| 175 | + |
| 176 | +## 11. Pre-implementation verification (for the §5 delete) |
| 177 | + |
| 178 | +1. No agent/harness lacking an injectable stdin that relies on resources for inbound (a purely-programmatic MCP agent would need a self-connected location instead). |
| 179 | +2. No external/third-party MCP client consuming `relay://` resources as an API. |
| 180 | + |
| 181 | +## 12. Decisions log |
| 182 | + |
| 183 | +- **Frame:** flat fabric of equal **agents** (peers) + a compute layer of named **nodes**; **broker is node infra, not a peer**. No "participant" umbrella, no agent subtype. Orchestrate/communicate is just an agent's **location** (via-node vs self-connected). |
| 184 | +- **Delivery:** outbound is direct & stateless; inbound goes to the agent's single **location**. Invariant: one location per agent. Delete the per-agent Relaycast WS, MCP resource layer, and piggyback (PTY-only); keep cloud-direct read tools; delivery-ack marks read. |
| 185 | +- **Spawn = node capability** via the action system; **placement** is targeted (`name`/`self`) or any (least-loaded); **resume = origin-targeted spawn + session_ref**. Node roster for discovery. |
| 186 | +- **Reliable invocation:** idempotency (`invocationId`) + at-least-once + reschedule + reconcile (first-to-`completed` wins); rides the action invocation machinery, no bespoke spawn state machine. |
| 187 | +- **Durability:** bounded-durable mailbox; at-least-once + dedup by `msg_id`; per-location `seq`; cumulative ack. Mailbox subsumes node replay; broker stateless across restarts; Relaycast is source of truth. |
| 188 | +- **Identity continuity requires session continuity:** resume is node-local; resumable agents node-sticky; otherwise fresh identity + dead-letter old mailbox. |
| 189 | +- **Policy:** dead-letter → notify sender; overflow → reject-new; down-but-resumable → lazy-resume by default (eager opt-in); one active location per agent name. |
0 commit comments