Status: current-state description of main after the July 2026 consolidation.
Technology choices live in stack.md; security decisions and narrow
design rationale live in ADR and topic documents. GitHub issues, not a
hand-maintained status page, track unfinished work.
Agent Orchestrator (AO) is a local control plane for coding-agent processes. A single Go daemon owns durable state and process supervision. Clients ask the daemon to act or read projections; they do not own lifecycle policy.
external facts durable facts read models / actions
GitHub, runtimes -> SQLite + lifecycle -> API, CLI, UI, notifications
agent activity reducers + CDC supervisors, Slack transport
Three rules explain most of the design:
- Persist facts, derive presentation state. A session stores activity,
termination, identity, runtime/workspace metadata, and PR facts. Values such
as
working,needs_input, andmergeableare read models. - Observation is not authority. Pollers report GitHub, runtime, resource, and health facts. Lifecycle and service owners decide what those facts mean.
- One daemon owns commands. CLI, Electron, browser, mobile, and ops sidecars call public HTTP/CLI surfaces instead of editing the database or terminal processes directly.
flowchart LR
CLI[ao CLI] -->|HTTP| D
Electron[Electron app] -->|HTTP / SSE / WS| D
Browser[Browser UI] -->|ao-web proxy| D
Mobile[Paired mobile client] -->|authenticated LAN listener| D
Slack[Slack notifier] -->|notifications + session facts| D
Reply[Slack reply listener] -->|ao send / decide| CLI
subgraph Host
D[Go daemon]
DB[(SQLite WAL)]
Runtime[tmux / ConPTY]
Git[Git worktrees]
Web[ao-web.service]
end
D --> DB
D --> Runtime
D --> Git
Web --> Browser
| Process or surface | Owner | Responsibility |
|---|---|---|
ao.service / ao daemon |
backend/internal/daemon |
Storage, lifecycle, sessions, observers, API, terminal mux, mobile listener |
| Electron main + renderer | frontend/ |
Desktop supervision and the React client |
| Browser production server | ops/ao-web-server.mjs |
Static bundle and same-origin proxy to the daemon |
| Slack notifier | ops/ao-slack-notifier.mjs |
Deliver durable notifications and current urgency to Slack |
| Slack reply listener | ops/attention-reply-listener.mjs |
Verify Slack events and route explicit replies through ao |
| Project-config drift timer | ops/project-config* |
Compare/apply committed project config through the CLI |
| Deploy driver | ops/deploy.sh |
Build, restart, and verify the local services |
The daemon's primary listener is loopback-only. Connect Mobile can explicitly enable a second authenticated LAN listener over the same router; see ADR 0001. The browser service is the remote web boundary and relies on its configured origin/tailnet controls.
backend/internal/daemon/daemon.go is wiring, not a policy layer. Startup builds
the graph in this order:
- load process configuration and reject a genuinely live prior daemon;
- open SQLite and start change-data capture (CDC);
- select the runtime adapter and build terminal streaming;
- build notification read/write surfaces and lifecycle/reaper;
- build SCM observation, session manager/service, review, and tracker intake;
- start pause drain, preview, health, model, metrics, and role supervisors;
- mount the HTTP router and optional mobile LAN listener;
- reconcile crash-surviving sessions before serving;
- serve until shutdown, then cancel and join background loops in dependency order.
The composition root may adapt interfaces and establish shutdown ordering. New business decisions belong in the owner packages below.
| Concern | Canonical owner | Boundary |
|---|---|---|
| Shared vocabulary and records | internal/domain |
Types and pure rules; no I/O |
| External contracts | internal/ports |
Narrow interfaces consumed by core code |
| Session commands | internal/session_manager |
Spawn, restore, kill, switch, send, workspace/runtime orchestration |
| Durable lifecycle facts | internal/lifecycle |
Reduce activity, runtime, and SCM observations; emit reactions |
| Controller-facing read models | internal/service/* |
Validate use cases and assemble domain projections |
| External observation | internal/observe/* |
Poll and report facts; do not become another lifecycle owner |
| Persistence | internal/storage/sqlite |
Migrations, sqlc queries, stores, and write serialization |
| HTTP transport | internal/httpd |
Decode, call a service, encode; terminal and SSE transports |
| CLI transport | internal/cli |
Thin HTTP client plus daemon/process bootstrap commands |
| External implementations | internal/adapters/* |
Agent, runtime, workspace, SCM, tracker, telemetry implementations |
| Production wiring | internal/daemon |
Construct the graph; own goroutine lifetime only |
| Browser experience | frontend/src/renderer |
Query API projections and render; no duplicated daemon policy |
| Host-specific integration | ops/ |
Deploy, web proxy, Slack, units, and config drift |
Adapters may depend on domain and ports, not on services or daemon wiring. Services may depend on domain/ports and narrower collaborators. HTTP and CLI must not reach through services to SQLite.
SQLite under ~/.ao is the durable source of truth. Its major fact families
are:
- projects and typed project configuration, plus pause state;
- sessions, activity, runtime/workspace identity, and pending decisions;
- session worktrees and launched harness identity;
- pull requests, checks, reviews, comments, and threads;
- durable notifications and their user acknowledgement state;
- telemetry/metric samples and daemon settings;
change_log, populated by database triggers.
Every migration is immutable after merge. Add a new numbered migration; never edit an old one. The dedicated migration CI job verifies unique versions and that a migration-bearing PR is based on the current target branch.
The CDC poller tails change_log and publishes events to in-process subscribers
and GET /api/v1/events. CDC invalidates clients and transports facts; it does
not replace service reads. Terminal bytes never enter SQLite or CDC.
sequenceDiagram
participant C as CLI / UI / intake
participant S as session service
participant M as session manager
participant W as workspace adapter
participant R as runtime adapter
participant A as agent adapter
participant L as lifecycle
C->>S: Spawn(config)
S->>M: Spawn(config)
M->>M: resolve project, cap, harness/model, prompt
M->>W: create worktree or in-place workspace
M->>A: prepare hooks + launch command
M->>R: create runtime and launch
M->>R: verify launch process alive
M->>R: deliver title/prompt if required
M->>R: verify launch process still alive
M->>L: MarkSpawned(metadata)
L-->>S: durable session
The two liveness observations are intentional. The first prevents a title or prompt from being pasted into a keep-alive shell after an immediate agent exit; the second catches an exit during input delivery. A failed probe is not proof of process death unless the adapter can make a definitive observation.
sessionguard.Guard is the one pre-write safety primitive for live panes. It
re-reads activity immediately before delivery and refuses terminated sessions,
permission dialogs, and unsafe unsolicited nudges.
Worktree creation is the default. In-place mode is explicit and has different branch/restore constraints. Cleanup never force-deletes unknown dirty work.
Tracker intake is an opt-in background observer:
- read enabled, unpaused projects and their open tracker issues;
- exclude configuration that makes an issue ineligible (assignment is the admission signal);
- derive whether the issue has a live worker, a landed PR, or a dead worker with unfinished work;
- spawn through the session service for issues with no session history;
- create or advance a durable recovery incident when a worker died with unfinished work, then emit one terminal escalation for the current incident attempt. Intake never launches blind replacement workers (#313 removed the automatic respawn subsystem); a respawn is allowed only after a new fix/remediation reference is recorded and is then treated as verification.
An open PR is not by itself a live driver. A terminated worker with an open PR must be escalated with the orphaned PR named; silently treating the PR as handled strands work. Conversely, an issue with a live worker or merged PR must not be dispatched again. Repeated same-fingerprint deaths increment the recovery attempt and escalate the investigation rung from worker to Orc to prime; the same dead session seen on later polls does not advance the incident.
Unresolved recovery incidents preserve the dead worker's workspace during session cleanup. Cleanup fails toward preservation if recovery state cannot be read, so diagnosis does not lose evidence.
The SCM observer independently detects two simultaneously open PRs for the same issue. Lifecycle owns the resulting comment and operator notification. This is a safety net, not the primary intake dedupe mechanism.
| Loop | Input | Output | Failure posture |
|---|---|---|---|
| Runtime reaper | runtime/process liveness | lifecycle runtime observation | Probe error is unknown, not dead |
| SCM observer | GitHub PR/check/review state | PR facts and lifecycle reactions | Lazy auth; semantic diff; retry |
| Tracker intake | project config, issues, sessions, PRs | worker spawn or escalation | Project backoff; no duplicate on uncertain reads |
| Pause drain | fleet/project pause and sessions | clean worker termination as work idles | Never starts work while paused |
| Preview poller | sessions and preview metadata | preview URL/revision facts | Best effort |
| Agent-health monitor | configured harnesses | install/auth snapshot | Async; inconclusive is unknown |
| Model-health monitor | configured exact model pins | reachability transitions/notifications | Probe infrastructure failure retains prior verdict |
| Metrics observer | host and session resources | metric samples/alerts | Configurable and disableable |
| Orchestrator/Prime supervisors | project/role sessions | ensure, wake, replace, or cap notification | Bounded replacement; respects pause |
All loops start asynchronously and stop by context cancellation. A slow external probe must not hold daemon readiness or shutdown indefinitely.
AO currently has three deliberately different health mechanisms:
service/agenthealthperiodically reports whether a harness binary exists and its authentication is usable;service/modelhealthperiodically revalidates configured exact model pins;candidatehealthreacts when an exact selected candidate fails during a real operation and can debit that candidate from a selection surface.
These mechanisms answer different questions: advisory readiness, configured pin validity, and authoritative execution failure. They should share model resolution and one exact-candidate projection, but not become a single mutable mega-monitor. Actual spawn remains authoritative.
Notifications are durable events. Producers submit a NotificationIntent to
notify.Manager, which validates, enriches, deduplicates through storage, and
publishes newly inserted rows to SSE subscribers. The desktop notification
center lists unread rows and lets the user acknowledge them.
Operator attention is a current-state projection. GET /api/v1/attention/operator combines live decisions, dead daemon roles,
merge-ready PRs, and selected durable escalations. The Waiting page and ao waiting consume this projection.
Slack is a delivery transport, not a lifecycle owner. Today it consumes durable notifications but also polls sessions for urgency and main-CI/agent-health facts. This duplicates part of the attention classifier. Consolidation must first extend the canonical projection to parity, then delete the duplicate JS classification. Slack delivery acknowledgement must remain distinct from the user's dashboard read state.
The former standalone outbound attention engine is deleted. Deploy/install code still disables its historical systemd unit and removes its legacy state so upgraded hosts cannot double-page. The inbound reply listener remains separate.
Fleet pause and project pause are durable gates, not configuration rewrites. While paused:
- tracker intake dispatches nothing;
- role supervisors do not create work and tell an existing orchestrator to idle once per instance;
- the drain observer lets active work reach a safe point and terminates workers through the normal session service;
- config drift ignores pause state.
Resume wakes the existing orchestrator or allows a replacement and restarts normal intake on the next observation cycle.
- The CLI maps commands to HTTP routes. Local daemon bootstrap, diagnostics, and terminal attachment are the justified process-aware exceptions.
- Electron owns desktop daemon discovery/launch and native integration. The renderer uses the same HTTP/SSE/WS contracts as browser mode.
- Browser mode is a first-class surface owned in this repository.
ao-webserves the production bundle and proxies the daemon; it must not fork product semantics from Electron. - Connect Mobile is an opt-in authenticated listener over the shared router. Loopback-only control routes remain excluded from the LAN surface.
- Ops programs may read APIs and invoke the public CLI. They do not patch daemon storage or tmux state.
The upstream-shaped backend and Electron core follow the vanilla rule: changes
are issue-first and are proposed upstream even when this repository carries the
delta. Browser-mode UX and ops/ are owned here. oldao/ is reference-only and
must not be resurrected.
This boundary affects refactors: a local ops simplification may land directly; a backend policy change needs its own upstream-shaped issue, tests, and PR.
- Do not persist display status when it can be derived from durable facts.
- Do not treat a failed liveness probe as proof of death.
- Do not write into a pane without the shared just-in-time session guard.
- Do not force-delete dirty or unknown user worktrees.
- Do not mutate old migrations or generated storage/API output by hand.
- Do not let a client, observer, or ops sidecar become a second lifecycle authority.
- Do not put terminal byte streams in SQLite/CDC.
- Do not block daemon readiness on optional credentials or external probes.
- Do not expose loopback control routes through the optional LAN listener.
- Do not patch backend policy locally without the upstream-shaped workflow.
The July 2026 audit deliberately left three broader decisions for focused work:
- Make daemon-derived operator attention canonical for web, CLI, and Slack; separate per-consumer delivery cursors from user read acknowledgement and replace synthetic notification targets with typed subjects.
- Make one worker-dispatch owner select exact harness/model candidates, reuse one model-precedence policy, and project harness, model, and reactive launch health consistently. Tracker intake must not bypass that owner.
- Make self-deploy build an immutable requested commit and make every service execute artifacts from the same release, independent of the caller's dirty or linked checkout.
Until those land, new work must not add another attention classifier, worker-mix selector, health truth, notification dedupe layer, or checkout-dependent deploy path.
- New durable noun: domain type, migration/query/store, service, then transport.
- New external system: port in core, implementation under
adapters. - New observation:
observereports a fact to an existing owner. - New session command: session service/manager, exposed by thin HTTP and CLI.
- New read projection: a service package consumed by every client.
- New host integration:
ops/, calling the public API/CLI. - New production wiring only:
daemon; move policy out if a wiring helper must be unit-tested as a decision.
Prefer deleting a duplicate path over introducing a shared abstraction whose only purpose is to preserve both copies.