Tracking: cross-agent batch coordination layer (heartbeats, leases, status) for multi-machine batch engineering

[justin808]

Umbrella for the cross-agent coordination layer ("batch engineering v2"). Today's operating reality: 3 concurrent batches (1× Claude Fable on hardest items, 2× Codex on simpler), 2 machines (M5 mobile laptop, M1 wired desktop), 3 launch surfaces (Claude Desktop, Codex Desktop, conductor.build), 2+ repos (react_on_rails, react_on_rails_rsc). The two acute pains: (1) manual copy-paste of status between agents with cross-batch dependencies, (2) no guard against two agents working the same issue/PR.

Design decision (made 2026-06-12)

Coordination state lives in a private GitHub repo — not public scratch-pad issues (visibility), not GitHub Projects (no CAS, awkward API), not Linear/Jira (new auth surface on every agent × machine, new subscription). Rationale and schemas: #3969. Core principles:

• Liveness is derived from heartbeat timestamps, never asserted by labels — a closed laptop or dead internet self-expires, no cleanup.
• Claims are leases with atomic acquire (GitHub Contents API sha precondition = compare-and-swap), expiring when the holder's heartbeat dies.
• Dependencies are declared in batch plans and polled via one command, not relayed by a human between desktop apps.

Child issues (dependency order)

• Batch coordination backend: private agent-coordination repo with CAS-safe state files #3969 — private coordination repo + agent-coord CLI (backend)
• Agent heartbeat protocol: timestamp-derived liveness for multi-machine batches (no stale labels) #3970 — heartbeat protocol (in-band skill hooks + out-of-band launchd renewal)
• Lease-based claims: make it near-impossible for two agents to work the same issue/PR #3971 — lease-based claims; preflight gates in pr-batch / plan-pr-batch / address-review / conductor sessions
• batch-status command + declared lane dependencies: end manual copy-paste between concurrent agents #3972 — batch-status command + depends_on lanes (kills the copy-paste)
• Document the multi-batch operating model (machines, launchers, conductor.build handoff, cross-repo) #3973 — document the multi-batch operating model (incl. conductor.build handoff rule)
• Attention contract: auto-fix nits, one decision point per lane, no babysitting CI waits (orchestration-tax reduction) #3975 — attention contract: auto-fix nits, one decision point per lane, no CI babysitting (the attention-side complement to the state-side issues above; see Osmani's orchestration tax)

Suggested rollout

1. Batch coordination backend: private agent-coordination repo with CAS-safe state files #3969 + Agent heartbeat protocol: timestamp-derived liveness for multi-machine batches (no stale labels) #3970 first (pure additions, no skill changes) — run one normal 3-batch day with heartbeats only, confirm the status table reflects reality.
2. Lease-based claims: make it near-impossible for two agents to work the same issue/PR #3971 gates next — start advisory (warn on conflict), flip to hard-stop after one clean week.
3. batch-status command + declared lane dependencies: end manual copy-paste between concurrent agents #3972 + Document the multi-batch operating model (machines, launchers, conductor.build handoff, cross-repo) #3973 last — retire manual status relaying.

Related

• Tracking: skills & PR-batch process improvements from the 17.0.0.rc.2 audit and Claude Workflow-tool comparison #3906 (tracking: skills & PR-batch process improvements) — this umbrella is the coordination workstream of that effort
• Deterministic orchestration for pr-batch: structured worker outputs, resume journal, verify phase (Workflow-tool lessons) #3912 (deterministic orchestration) — per-batch resume journal; composes with the cross-batch layer here
• PR-batch closeout: machine-checkable per-PR merge ledger (extends #3851) #3907 (merge ledger) — lease release events can feed it
• Evidence of the failure modes from the first adoption repo (react_on_rails_rsc): duplicate issues Dry up the manifest of webpack bundles #26/Need to port optimization of pooling renderer from react-rails #28, duplicate PRs Support direct react components and server console messages and linting #20/Switch to therubyracer and general questions about what JS engine #21, 60+ stale batch branches

[justin808]

Codex batch claim for the cross-agent coordination layer. This claim is temporary and expires automatically if not refreshed.

[justin808]

Conversation handoff / archive note (2026-06-13)

Archiving the Claude session that designed this coordination layer. Fable was disabled mid-session, work moved to Codex, and the backend has since shipped. Recording state so nothing is lost.

Shipped since the original design (Codex took these over)

• ✅ Batch coordination backend: private agent-coordination repo with CAS-safe state files #3969 backend → private repo shakacode/agent-coordination + agent-coord CLI (PR Add cross-agent coordination backend workflow #3977)
• ✅ Agent heartbeat protocol: timestamp-derived liveness for multi-machine batches (no stale labels) #3970 heartbeats · ✅ Lease-based claims: make it near-impossible for two agents to work the same issue/PR #3971 leases · ✅ batch-status command + declared lane dependencies: end manual copy-paste between concurrent agents #3972 batch-status · ✅ Document the multi-batch operating model (machines, launchers, conductor.build handoff, cross-repo) #3973 operating-model doc
• Open follow-up: Follow-up: Harden agent-coord contract after PR #3977 #3984 (harden agent-coord contract after Add cross-agent coordination backend workflow #3977)

Still open in this cluster

• Tracking: cross-agent batch coordination layer (heartbeats, leases, status) for multi-machine batch engineering #3974 (this umbrella) · Attention contract: auto-fix nits, one decision point per lane, no babysitting CI waits (orchestration-tax reduction) #3975 (attention contract — incl. rule 6: documented-confidence merge)

New workstream filed this session — "batch lifecycle" (intake → release → evaluation)

A second cluster, distinct from the coordination state layer above:

• Release-train branching + phase-tiered merge gating (beta/RC/final) for the batch era #4000 — Release-train branching + phase-tiered gating (beta/RC/final). Solves: continuous merges to main destabilizing an RC. Model: main never freezes; cut release/X.Y.Z; stabilize there + forward-merge; promote last good RC to final by dropping -rc; gate strictness rises by phase.
• New 'triage' skill: whole-surface issue/PR dependency inventory → split into 3 capacity-aware batch groups #4002 — triage skill: whole-surface issue/PR dependency inventory → split into 3 capacity-aware batch groups (generalizes the manual React on Rails 17.0.0 RC issue triage and machine queue #3788 snapshot; consumes heartbeats to show in-flight vs stalled).
• Capacity-aware batch assignment + per-inbox queue (M5 runs 2 batches, M1 runs 1) + cross-repo session convention #4003 — Capacity-aware assignment + per-inbox queue: M5 (128GB) runs 2 batches (Codex + Fable); M1 (64GB) runs 1. Plus the cross-repo "session wants one repo" convention.
• Continuous evaluation loop: independently verify merged agentic runs achieved issue intent (+ surface stalled runs) #4004 — Continuous evaluation loop: independently verify merged agentic runs achieved issue intent; surface stalled (lost-heartbeat) runs for a resume/reassign/drop decision.

Article / writing

• Draft package in shakacode/sc-articles PR Support direct react components and server console messages and linting #20 (ai/batch-engineering/): article + X/LinkedIn posts + research-notes. Frames batch engineering as the response to Osmani's orchestration tax, built with loop-engineering primitives. Pending: voice pass; incorporate machine-capacity detail (Capacity-aware batch assignment + per-inbox queue (M5 runs 2 batches, M1 runs 1) + cross-repo session convention #4003) and the "flow was designed via grill-me/AI discussions on what agents may merge to main" origin story; forward-ref the release tiers (Release-train branching + phase-tiered merge gating (beta/RC/final) for the batch era #4000).

Origin-story note to capture (raised but not yet written anywhere)

The batch flow itself was designed through AI discussions, notably the grill-me skill, specifically to decide what should be enabled for agents to merge to main. Worth a dedicated section in the article and a line in AGENTS.md history.