Skip to content

Commit 431c227

Browse files
committed
Add agent messaging eval suite and reports
Introduce a new integration eval suite that exercises agent-to-agent messaging via the broker and scores protocol adherence (message-sent rate, phantom messages, ACK/DONE protocol, wrong-channel replies). Adds a full eval runner, scenarios, deterministic scoring, reporters (JSON + self-contained HTML viewer), a matrix roll-up, unit tests for scoring, and CLI helpers under tests/integration/broker/evals. Adds npm scripts (eval:build, eval:unit, eval:selftest, eval:toolcheck, eval:html, eval, eval:claude, eval:matrix) and gitignore entry for evals-reports. Also adds a Fleet Delivery design doc (specs/fleet-delivery.md), updates CHANGELOG.md, and adjusts integration test config/files (tsconfig, vitest, and broker harness utilities) to align the broker-harness with the current SDK/harness-driver APIs so the evals build/run cleanly.
1 parent 5012154 commit 431c227

40 files changed

Lines changed: 2786 additions & 28 deletions

.agentworkforce/trajectories/active/traj_b1jrutolckfb/trajectory.json

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,18 @@
4343
"reasoning": "User requested Relaycast request attribution, install/update events, and MCP action-call telemetry while preserving UA-like harness values."
4444
},
4545
"significance": "high"
46+
},
47+
{
48+
"ts": 1780761935387,
49+
"type": "decision",
50+
"content": "Repaired broker-harness.ts to current SDK/harness-driver API and built eval suite on it: Repaired broker-harness.ts to current SDK/harness-driver API and built eval suite on it",
51+
"raw": {
52+
"question": "Repaired broker-harness.ts to current SDK/harness-driver API and built eval suite on it",
53+
"chosen": "Repaired broker-harness.ts to current SDK/harness-driver API and built eval suite on it",
54+
"alternatives": [],
55+
"reasoning": "Broker integration suite was pre-existingly broken: SDK narrowing moved BrokerEvent/HarnessDriverClient/SendMessageInput to @agent-relay/harness-driver and RelayCast to @relaycast/sdk; AgentRelay facade no longer does broker lifecycle. Fixed imports + removed the unused AgentRelay facade from BrokerHarness; added a dedicated evals/tsconfig.json compiling only evals/ + utils/ so eval:build is green without rewriting the still-broken sibling test files."
56+
},
57+
"significance": "high"
4658
}
4759
]
4860
}
@@ -55,4 +67,4 @@
5567
"startRef": "bd42f4f84f41821e33879618d114941d6eabe835",
5668
"endRef": "bd42f4f84f41821e33879618d114941d6eabe835"
5769
}
58-
}
70+
}

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,3 +93,6 @@ web/.open-next
9393
!/workflows/refactor/
9494
!/workflows/relayauth-integration/
9595
!/workflows/cloud-connect/
96+
97+
# Eval harness JSON reports (generated per run)
98+
tests/integration/broker/evals-reports/

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
99

1010
### Added
1111

12+
- Agent messaging eval suite (`npm run eval`, `eval:matrix`, `eval:unit`, `eval:selftest`, `eval:toolcheck`) spawns real agent CLIs and scores, from broker events, whether agents actually used the MCP/CLI to message — reporting message-sent rate, phantom-message rate (intent stated in prose but no send), ACK/DONE protocol adherence, and wrong-channel replies. Includes a `realistic` tier (natural-language tasks where the protocol must come from the injected onboarding) and a `smoke` tier (leading prompts as a plumbing canary), a negative-control self-test, and a deterministic wrong-tool-name trap that flags onboarding referencing tools the MCP server doesn't register. Each run emits a self-contained HTML viewer (overview, per-scenario prompts, full message transcript, phantom call-outs) alongside JSON reports with baseline regression diffing.
1213
- `@agent-relay/harnesses` adds a `grok` PTY harness for the Grok CLI, including Relaycast MCP support for spawned agents.
1314
- `@agent-relay/harnesses` is now published to npm, so SDK consumers can install the prebuilt PTY harnesses and harness-authoring helpers.
1415
- `agent-relay drive` and `agent-relay passthrough` add adaptive predictive echo so typing stays responsive when driving a high-latency or remote agent, and stays invisible on fast local links.

package.json

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,14 @@
4545
"test:integration:broker": "npx tsc -p tests/integration/broker/tsconfig.json && cd tests/integration/broker && node --test dist/*.test.js",
4646
"test:integration:broker:build": "npx tsc -p tests/integration/broker/tsconfig.json",
4747
"test:integration:broker:run": "cd tests/integration/broker && node --test dist/*.test.js",
48+
"eval:build": "npx tsc -p tests/integration/broker/evals/tsconfig.json",
49+
"eval:unit": "vitest run tests/integration/broker/evals",
50+
"eval:selftest": "npm run eval:build && node tests/integration/broker/dist/evals/selftest.js",
51+
"eval:toolcheck": "npm run eval:build && node tests/integration/broker/dist/evals/toolcheck-cli.js",
52+
"eval:html": "npm run eval:build && node tests/integration/broker/dist/evals/report/render-cli.js",
53+
"eval": "npm run eval:build && cd tests/integration/broker && RELAY_INTEGRATION_REAL_CLI=1 node dist/evals/runner.js",
54+
"eval:claude": "npm run eval:build && cd tests/integration/broker && RELAY_INTEGRATION_REAL_CLI=1 node dist/evals/runner.js --harness=claude",
55+
"eval:matrix": "npm run eval:build && cd tests/integration/broker && RELAY_INTEGRATION_REAL_CLI=1 node dist/evals/runner.js --harness=claude,codex,gemini,grok",
4856
"lint": "npm --prefix packages/cli run lint",
4957
"knip": "knip",
5058
"syncpack": "syncpack lint",

specs/fleet-delivery.md

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
# Fleet Delivery — Agents, Nodes, and Reliable Messaging
2+
3+
**Status**: Draft
4+
**Date**: 2026-06-06
5+
**Author**: Design session (Will + Claude)
6+
7+
---
8+
9+
## 1. Vision
10+
11+
Run agents across many machines. Each machine (a **node**) can run a specific set of compute — some can spawn Claude agents, some Codex, some both. Relaycast is the control plane: agents are equal peers in a flat messaging fabric, nodes advertise what they can spawn, and Relaycast routes messages to agents wherever they live and places new agents onto nodes that can run them.
12+
13+
The goal is the simplest deployable unit that maximizes the environments a node can run in, with messaging that survives flaky networks and agent restarts.
14+
15+
## 2. The frame: two planes
16+
17+
Everything below lives on one of two planes. Keeping them separate is what keeps the model simple.
18+
19+
- **Messaging fabric — flat, all equal `agents`.** Every agent is a peer: a stable identity that sends, receives, and may expose actions. No agent is above or routed "through" another.
20+
- **Compute layer — `nodes`.** A node is a machine where some agents run. It has a **broker** runtime and a set of **capabilities** (what it can spawn). Agents sharing a node is a deployment fact, not a relationship in the fabric.
21+
22+
There is **no "participant" umbrella and no agent subtype.** "Orchestrate vs communicate" is not a type distinction — it's just whether an agent is colocated on a node-with-broker or self-connected (see *location*, §5).
23+
24+
## 3. Core concepts
25+
26+
| Concept | What it is | Plane |
27+
|---|---|---|
28+
| **Agent** | a peer in the fabric: identity, send/receive, exposed actions | messaging |
29+
| **Node** | a named machine that runs agents and advertises capabilities | compute |
30+
| **Broker** | a node's runtime/delivery engine — **infra, not a peer** (not in the agent roster) | compute |
31+
| **Location** | where Relaycast routes an agent's **inbound** | routing detail |
32+
| **Capability** | what a node can spawn (e.g. `spawn:codex`) | compute |
33+
| **Action** | something an agent exposes/invokes in the fabric | messaging |
34+
35+
The broker is node infrastructure with a control connection to Relaycast. A colocated PTY agent receiving via its broker is the same kind of plumbing as a NIC delivering to a process — a location, not a hierarchy.
36+
37+
## 4. Identities & naming
38+
39+
- **Agent**: stable `agent_id` + workspace-unique **name** (the addressable handle). One **active location** per agent name (a second live claimant is rejected; migration is explicit).
40+
- **Node**: workspace-unique **name**, operator-set at startup (default: hostname), optionally backed by a stable internal id so a name can move to a replacement machine. Same uniqueness rule as agents: one live owner.
41+
- The node's broker is the **token authority** for agents it spawns: it asks Relaycast to mint the agent identity + token on spawn, hands the token to the agent, and binds its location.
42+
43+
## 5. Delivery model — keep + delete
44+
45+
Most of this already exists; the work is mostly removal.
46+
47+
**Outbound (all agents): direct & stateless.** An agent sends with its own token straight to Relaycast (PTY agents via their MCP send tools; SDK agents via their own send). Sends are request/response — **no persistent connection required for sending**, and the broker is never in the send path.
48+
49+
**Inbound: delivered to the agent's location.** Location has exactly two shapes — a field, not a type:
50+
- **Self-connected** (event-loop programs — SDK agents): the agent's own WS + message handler. This *is* the delivery path.
51+
- **Via its node** (raw PTY harnesses with no event loop): the node's broker receives and injects into the agent's stdin.
52+
- `steer` = inject + interrupt to a prompt now.
53+
- `wait` = write to the buffer; the harness reads at its next prompt (the PTY defers naturally).
54+
55+
**Invariant: an agent has exactly one location.** This is the whole cleanup — the old redundancy was a via-node agent *also* holding its own WS. One location → no double delivery, no special-casing.
56+
57+
**Delete (these only ever applied to PTY agents):**
58+
- The per-agent Relaycast WS (`RealtimeResourceBridge`).
59+
- The MCP **resource layer** (`relay://inbox`, `relay://channels/...`, subscribe/notify) — it assumes a reactive client; turn-based harnesses don't subscribe.
60+
- The **inbox piggyback** (stapling inbox onto every tool result).
61+
62+
**Keep:** on-demand read/query tools (`check_inbox`, `list_messages`, `thread`, `search`, `list_channels`) as **stateless cloud-direct reads** with the agent token. Pulling on your own initiative doesn't need a persistent connection. The MCP server for PTY agents becomes outbound + reads only.
63+
64+
**Consistency bolt:** delivery acknowledgement marks a message delivered/read in Relaycast, so a cloud-backed `check_inbox` never re-surfaces something already delivered. One source of truth for *history* (cloud), one for *delivery* (the location); the ack bridges them.
65+
66+
**Why the deleted layers existed (so we don't rebuild them):** the original design was MCP-idiom-first — inbox/channels as subscribable resources, the textbook way to surface stateful data. Turn-based harnesses didn't react to `resources/updated`, so the piggyback was bolted on, and stdin injection became the reliable push. Nothing was removed, leaving three overlapping inbound paths. **Lesson: design delivery around the agent's execution model (turn-based vs event-loop), not the protocol's idiom.**
67+
68+
## 6. Spawn & placement
69+
70+
**Spawn is not a protocol concept — it's a node capability**, expressed through the existing action mechanism (`actions.register('create', handler → driver.spawn)`). The "how spawning happens here" is a node-side harness definition (`definePtyHarness` / `StaticPtyHarnessDefinition`) — the script you provide when spinning up a node. A node advertises the capabilities it defines (e.g. `spawn:claude`, `spawn:codex`).
71+
72+
**Placement** takes an optional target:
73+
74+
```
75+
spawn { capability, node?: <name> | "self", session_ref?, ttl_override? }
76+
77+
eligible = nodes where
78+
(node.name == target if target given)
79+
∧ capability ∈ node.capabilities
80+
∧ node.live ∧ capacity_available
81+
place: target if given, else least-loaded(eligible)
82+
```
83+
84+
- `node: "gpu-box-1"` → must place there. Capability mismatch → **hard fail**. Offline → bounded-queue (or fail-fast per override).
85+
- `node: "self"` → same node as the requester (the common **colocation** case: shared working dir, local artifacts). An agent needn't know its node's name.
86+
- `node` omitted → scheduler picks any eligible (least-loaded).
87+
- None eligible → bounded-queue, then fail.
88+
89+
**Resume is a special case of targeted spawn.** "Resume agent X" = spawn with `node: <origin node>` + its `session_ref`. There is no separate resume concept — it is placement constrained to the origin node plus a session reference (see §8.2).
90+
91+
**Node roster:** because agents/humans/schedulers can target by name, Relaycast exposes a **node discovery query** (name, capabilities, liveness, load) — the compute-layer roster, parallel to the agent roster, and what a UI like Pear renders.
92+
93+
## 7. Reliable action invocation (spawn rides on this)
94+
95+
Spawn inherits the action system's async invocation machinery (`invocationId`, ack, result) rather than a bespoke state machine. Exactly-once placement is impossible (dispatch, node dies before ack — did it start?), so the contract is **idempotency + at-least-once + reconcile**:
96+
97+
- `invocationId` is the idempotency key. A node dedups invocations by it; a requester retrying with the same id never double-spawns.
98+
- Invocation lifecycle: `pending → dispatched(node) → completed(agent_id)`.
99+
- **Dispatch timeout / node lost** before completion → **reschedule** to another eligible node with the same `invocationId`.
100+
- **Reconcile on reconnect:** a node re-announces its live agent inventory (with `agent_id`, name, `invocationId`, `session_ref`) — see §9. If an invocation already completed elsewhere → the duplicate is released. **First to `completed` wins.** A dead broker brings no agents back (its children died with it), so the dead-node case reschedules cleanly; duplicates only arise from a live-broker uplink blip and are reconciled away.
101+
102+
## 8. Durability — bounded-durable mailbox
103+
104+
**Decision: bounded-durable.** A message for an unreachable agent is held for a TTL and delivered when it's reachable again; dead-lettered after. Reliable without infinite state.
105+
106+
### 8.1 Message state machine (held in Relaycast)
107+
108+
```
109+
queued ──deliver(seq)──▶ delivered ──ack──▶ acked (≈read)
110+
111+
└── TTL expiry ──▶ dead-letter
112+
```
113+
114+
- **At-least-once + dedup by `msg_id`.** Per-agent ordering falls out of a monotonic per-location `seq`.
115+
- Relaycast pushes `queued` messages to the agent's location with a per-session `seq`; the location injects/handles, then **cumulative ack** (`up_to_seq`) advances them to `acked`.
116+
117+
### 8.2 Identity continuity requires session continuity
118+
119+
Reclaiming a mailbox without resuming the actual session would dump a backlog on a context-less process — worse than dead-lettering. So an agent reclaims its identity + held mailbox **only by resuming its session** (= origin-targeted spawn + `session_ref`, §6):
120+
121+
- **`resumable`** is a per-harness capability; on spawn of a resumable harness the broker captures the `session_ref` and reports it to Relaycast.
122+
- **Resume is node-local (v1):** harness session state lives on the origin node's disk, so resumable agents are **node-sticky**. If the origin node is permanently gone, the session is unrecoverable → identity terminal → mailbox dead-lettered. (Cross-node resume needs cloud-synced session state — deferred, §10.)
123+
- No resumable capability / no recoverable session → respawn is a **new identity**; the old mailbox is dead-lettered (senders notified).
124+
125+
### 8.3 Mailbox resolution
126+
127+
| Situation | Resolution |
128+
|---|---|
129+
| Location temporarily unreachable (uplink/WS blip; process alive) | Hold + deliver on reconnect. Applies to **all** agents. |
130+
| Process dead, resumable + session recoverable | Hold up to TTL; flush on origin-targeted resume (oldest-first). |
131+
| Process dead, non-resumable OR session lost | **Dead-letter immediately** (notify senders). |
132+
133+
Consequence: persistence *across process death* is a **resumable-only** property. Non-resumable agents are ephemeral-on-death (still resilient to transient blips while alive).
134+
135+
### 8.4 Where state lives
136+
137+
- **Relaycast** holds all durable state (source of truth): mailboxes, agent records (`resumable`, `session_ref`, origin node), locations, node registry. Must survive Relaycast restarts.
138+
- **Broker** keeps only in-memory per-session state: `seq` cursor, dedup set, local pending-injection queue. **No disk needed for delivery durability.**
139+
- Uplink blip, broker alive → cursor/dedup survive → clean replay, no duplicates.
140+
- Broker process dies → its child agents die too → they respawn and *want* redelivery → redelivery is correct, not duplicate.
141+
142+
### 8.5 One durable store
143+
144+
The per-agent mailbox **subsumes** any per-node replay buffer: node-disconnect replay is just "redeliver this node's agents' unacked mail on reconnect." The per-location `seq` + ack is the at-least-once transport on top of the mailbox.
145+
146+
### 8.6 Policy (defaults, all tunable)
147+
148+
- **TTL:** workspace default (~1h placeholder) + per-message override ("5m or drop" for time-sensitive; longer for durable tasks).
149+
- **Dead-letter on expiry:** retain briefly + **emit `delivery_failed`/expired to the sender** (reuse existing event). Silent drop is the wrong default.
150+
- **Mailbox overflow:** **reject-new** with sender feedback (so senders learn the agent is backed up) rather than drop-oldest.
151+
- **Inbound to a down-but-resumable agent:** **lazy by default** (queue; resume via restart-policy or explicit respawn), **eager opt-in** ("wake on message").
152+
- **Restart policy:** a resumable agent with auto-restart → broker auto-resumes the session on its node and flushes the mailbox.
153+
154+
## 9. Node lifecycle & control surface
155+
156+
A node's broker holds one control connection to Relaycast, serving two roles: **compute provider** (advertises capabilities, receives spawn/release action invocations, reports results) and **delivery relay** (receives inbound for the PTY agents located on it, injects, acks).
157+
158+
- **Register** (on connect): node name, capabilities, version, `max_agents`, tags, and a resume cursor for replay.
159+
- **Heartbeat** (~10–15s): `load`, `active_agents`. Relaycast TTL marks offline → stop placing there; mark its located agents unreachable.
160+
- **Reconnect inventory sync:** after register, the broker re-announces its full live agent inventory (`agent_id`, name, `invocationId`, `session_ref`). Relaycast reconciles **locations** and open **invocations** (§7) from it.
161+
- **Deregister:** graceful on shutdown; else liveness TTL.
162+
163+
**Narrow control surface** (the only Relaycast protocol the broker implements — *not* the full `@relaycast/sdk`; channels/threads/reactions/search stay in the agent SDK):
164+
165+
- **Broker → Relaycast:** `node.register`, `node.heartbeat`, `node.deregister`, `agent.register` (bind location), `agent.deregister`, `delivery.ack`, `action.result`, `inventory.sync`
166+
- **Relaycast → Broker:** `action.invoke` (spawn/release are actions), `deliver`, `ping`
167+
168+
## 10. Deferred / open
169+
170+
- **Cross-node session resume** via cloud-synced session state (lifts node-sticky in §8.2).
171+
- **Same-node fast-path (perf):** local A→B delivery bypassing the cloud, and whether it's allowed when the uplink is down.
172+
- **Node tags / fuzzy targeting** (`gpu` instead of an exact name) and **access control** on who can target / spawn on which nodes.
173+
- **Exact tunables:** TTL, mailbox depth cap, heartbeat interval/TTL, dispatch timeout.
174+
- **Mark-read mechanism:** broker auto-mark on delivery (preferred) vs explicit `mark_read` tool — keep explicit only if a product reason emerges.
175+
176+
## 11. Pre-implementation verification (for the §5 delete)
177+
178+
1. No agent/harness lacking an injectable stdin that relies on resources for inbound (a purely-programmatic MCP agent would need a self-connected location instead).
179+
2. No external/third-party MCP client consuming `relay://` resources as an API.
180+
181+
## 12. Decisions log
182+
183+
- **Frame:** flat fabric of equal **agents** (peers) + a compute layer of named **nodes**; **broker is node infra, not a peer**. No "participant" umbrella, no agent subtype. Orchestrate/communicate is just an agent's **location** (via-node vs self-connected).
184+
- **Delivery:** outbound is direct & stateless; inbound goes to the agent's single **location**. Invariant: one location per agent. Delete the per-agent Relaycast WS, MCP resource layer, and piggyback (PTY-only); keep cloud-direct read tools; delivery-ack marks read.
185+
- **Spawn = node capability** via the action system; **placement** is targeted (`name`/`self`) or any (least-loaded); **resume = origin-targeted spawn + session_ref**. Node roster for discovery.
186+
- **Reliable invocation:** idempotency (`invocationId`) + at-least-once + reschedule + reconcile (first-to-`completed` wins); rides the action invocation machinery, no bespoke spawn state machine.
187+
- **Durability:** bounded-durable mailbox; at-least-once + dedup by `msg_id`; per-location `seq`; cumulative ack. Mailbox subsumes node replay; broker stateless across restarts; Relaycast is source of truth.
188+
- **Identity continuity requires session continuity:** resume is node-local; resumable agents node-sticky; otherwise fresh identity + dead-letter old mailbox.
189+
- **Policy:** dead-letter → notify sender; overflow → reject-new; down-but-resumable → lazy-resume by default (eager opt-in); one active location per agent name.

0 commit comments

Comments
 (0)