feat: zero-downtime auto-upgrade for the daemon

The daemon can hot-swap its own binary to a newer release WITHOUT
interrupting running agents. It re-execs in place (same PID) via
unix.Exec, inheriting the held repo flock and the API listener sockets
through CLOEXEC-cleared fds, and re-adopts the still-running agent child
processes — so only the control-plane binary changes underneath them,
with no lock gap and no dropped connection.

New internal/upgrade package:
- source: GitHub releases by default, or a mirror via AGENTENV_UPDATE_URL
  (latest.txt + agentenv_<ver>_<os>_<arch>.tar.gz + checksums.txt)
- verify: SHA256 against checksums.txt, then a `<new-binary> version`
  smoke test — either failing keeps the old daemon serving
- schedule: poll every AGENTENV_UPDATE_INTERVAL (default 1h), apply
  inside AGENTENV_UPDATE_WINDOW (e.g. 02:00-04:00), or on demand
- handoff: CLOEXEC clearing, proc manifest, atomic binary rename with
  rollback, and the execve

Triggers: automatic in the maintenance window, or `agentenv upgrade
[--version X]` / `agentenv ctl upgrade` on demand (bypasses the window).
Dev/commit builds don't auto-upgrade (no comparable version); a manual
--version still works. supervise is out of scope (foreground PTY agent).

Supporting changes:
- repo: AdoptProc (per-PID Wait4 reaper — never a global wait4(-1) that
  would steal os/exec's cmd.Wait exit status), LockFromFD, Kill/killAll
  handle adopted procs (unix.Kill by pid)
- api: Gate (shared writer lock so a hot upgrade can't fire mid-mutation),
  ServeListener / ServeHTTPListener (daemon owns the listeners for fd
  handoff), and the out-of-band "upgrade" op via an UpgradeHandler
  interface (breaks the api→upgrade import cycle)
- cli: cmdDaemon gains a resume path (rebuild lock/listeners from inherited
  fds, re-adopt procs); Run skips AcquireLock on the re-exec path; new
  `upgrade` command with transparent ctl routing
- protocol: Version/Force request fields, Message response field

Verified end-to-end on Linux (scripts/e2e_upgrade.sh): daemon re-execs
v0.3.0 -> v0.4.0 keeping the same PID, the socket stays served, the flock
is never released, and a spawned agent is re-adopted and keeps running
uninterrupted across the swap.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Author: css521

CHANGELOG.md

@@ -4,6 +4,27 @@ All notable changes to this project are documented here. The format is based on
 [Keep a Changelog](https://keepachangelog.com/), and the project aims to follow
 [Semantic Versioning](https://semver.org/).
 
+## [0.3.1]
+
+### Added
+- **Zero-downtime auto-upgrade** (`agentenv daemon`): the daemon can hot-swap
+  its own binary to a newer release without interrupting running agents. It
+  re-execs in place (same PID), inheriting the held repo flock and the API
+  listener sockets via CLOEXEC-cleared fds, and re-adopts the still-running
+  agent child processes — so the control-plane binary changes underneath them
+  with no lock gap and no dropped connection.
+  - New `internal/upgrade` package: release source (GitHub by default, or a
+    mirror via `AGENTENV_UPDATE_URL` serving `latest.txt` +
+    `agentenv_<ver>_<os>_<arch>.tar.gz` + `checksums.txt`), SHA256 verification,
+    `<new-binary> version` smoke test, maintenance-window scheduler, and the
+    execve handoff.
+  - Triggers: automatic within `AGENTENV_UPDATE_WINDOW` (e.g. `02:00-04:00`;
+    polled every `AGENTENV_UPDATE_INTERVAL`, default 1h), or on demand via
+    `agentenv upgrade [--version X]` / `agentenv ctl upgrade` .
+  - Safety: checksum + smoke test gate the swap; on any failure the old daemon
+    keeps serving. Dev/commit builds don't auto-upgrade (manual `--version`
+    still works). `supervise` is out of scope (foreground PTY agent).
+
 ## [0.3.0] - 2026-06-17
 
 ### Added

README.md

@@ -90,6 +90,7 @@ agentenv exec -- <cmd...>       run one command in the env (scripting/CI)
 
 # out-of-band client (talks to a running daemon/supervise, no repo lock needed)
 agentenv ctl <op> [args]        log | head | branches | exec | checkout | delete | diff | ...
+agentenv upgrade [--version X]  zero-downtime hot-upgrade of a running daemon (re-exec in place)
 
 # setup / history / inspection
 agentenv init --from <dir|/> | --tarball <path|URL>
@@ -108,6 +109,39 @@ agentenv gc                     reclaim disk from sparsified snapshots
 Env: AGENTENV_ROOT (default /agentfs), AGENTENV_SOCKET, AGENTENV_BACKEND,
      AGENTENV_HTTP (=":PORT" / "127.0.0.1:PORT"), AGENTENV_HTTP_TOKEN (required for non-loopback HTTP),
      AGENTENV_KEEP_RECENT, AGENTENV_IGNORE.
+Auto-upgrade (daemon): AGENTENV_UPDATE_URL (mirror base; default GitHub releases),
+     AGENTENV_UPDATE_REPO (owner/name for GitHub), AGENTENV_UPDATE_INTERVAL (poll cadence,
+     default 1h; "off" disables), AGENTENV_UPDATE_WINDOW ("HH:MM-HH:MM" local maintenance window).
+```
+
+## Zero-downtime auto-upgrade
+
+`agentenv daemon` can hot-swap its own binary to a newer release **without
+interrupting running agents**. It re-execs in place (same PID), so the held repo
+lock, the API sockets, and every spawned agent process survive the swap — only
+the control-plane binary changes underneath them.
+
+- **Discovery**: the daemon polls a release source every `AGENTENV_UPDATE_INTERVAL`
+  (GitHub releases by default, or a mirror via `AGENTENV_UPDATE_URL`). A mirror
+  serves `latest.txt`, `agentenv_<ver>_<os>_<arch>.tar.gz`, and `checksums.txt`.
+- **Safety**: a candidate is SHA256-verified against `checksums.txt` and
+  smoke-tested (`<new-binary> version`) before anything is swapped. If either
+  fails, the daemon keeps serving the old version.
+- **When**: applied automatically inside `AGENTENV_UPDATE_WINDOW` (e.g.
+  `02:00-04:00`; omit for "any time"), or on demand — bypassing the window —
+  via `agentenv upgrade [--version vX.Y.Z]`, `agentenv ctl upgrade`, or the
+  `agentenv__upgrade` MCP tool.
+- **Scope**: auto-upgrade is off for dev/commit builds (no comparable version);
+  a manual `--version` still works. `supervise` is not hot-upgraded (its
+  interactive PTY agent is foreground); use `daemon` for zero-downtime control.
+
+```bash
+# poll GitHub hourly, apply only between 02:00 and 04:00 local
+AGENTENV_UPDATE_WINDOW=02:00-04:00 agentenv daemon
+
+# or upgrade a running daemon right now
+agentenv upgrade                 # to the latest release
+agentenv upgrade --version v0.4.0
 ```
 
 ## Running your agent transparently (zero changes)

internal/api/http.go

@@ -42,11 +42,23 @@ const HTTPVersion = "0.3.0"
 // `Authorization: Bearer <token>` — non-loopback addresses without a token are
 // rejected at startup as a safety guard. Compatible with the socket dispatch:
 // both paths call the same r.* and the same repo handles the locking.
-func ServeHTTP(ctx context.Context, r *repo.Repo, _ *repo.Capturer, addr, token string) error {
+func ServeHTTP(ctx context.Context, r *repo.Repo, c *repo.Capturer, addr, token string) error {
 	if token == "" && !isLoopback(addr) {
 		return fmt.Errorf("refusing to bind HTTP on non-loopback %s without AGENTENV_HTTP_TOKEN — set a token or bind 127.0.0.1", addr)
 	}
+	ln, err := net.Listen("tcp", addr)
+	if err != nil {
+		return err
+	}
+	return ServeHTTPListener(ctx, r, c, ln, token)
+}
 
+// ServeHTTPListener runs the HTTP API on an already-created listener. The
+// hot-upgrade path needs this seam: the daemon owns the tcp listener so it can
+// clear FD_CLOEXEC on it and hand the fd to the new image across execve, which
+// rebuilds the listener with net.FileListener and calls this again — so an
+// in-flight bind is never re-attempted and the port never flaps.
+func ServeHTTPListener(ctx context.Context, r *repo.Repo, _ *repo.Capturer, ln net.Listener, token string) error {
 	mux := http.NewServeMux()
 	cfg := huma.DefaultConfig("agentenv", HTTPVersion)
 	cfg.Info.Description = "REST surface for the agentenv daemon. " +
@@ -68,11 +80,7 @@ func ServeHTTP(ctx context.Context, r *repo.Repo, _ *repo.Capturer, addr, token
 		handler = bearerAuth(mux, token)
 	}
 
-	srv := &http.Server{Addr: addr, Handler: handler}
-	ln, err := net.Listen("tcp", addr)
-	if err != nil {
-		return err
-	}
+	srv := &http.Server{Handler: handler}
 	fmt.Fprintf(os.Stderr, "agentenv HTTP listening on http://%s (docs http://%s/docs)\n", ln.Addr(), ln.Addr())
 	go func() {
 		<-ctx.Done()
@@ -84,6 +92,17 @@ func ServeHTTP(ctx context.Context, r *repo.Repo, _ *repo.Capturer, addr, token
 	return nil
 }
 
+// GuardHTTP returns an error if binding addr without a token would expose the
+// API beyond loopback. Exported so the daemon, which now owns the TCP listener
+// itself (for fd handoff across a hot upgrade), can apply the same guard
+// ServeHTTP applies before it binds.
+func GuardHTTP(addr, token string) error {
+	if token == "" && !isLoopback(addr) {
+		return fmt.Errorf("refusing to bind HTTP on non-loopback %s without AGENTENV_HTTP_TOKEN — set a token or bind 127.0.0.1", addr)
+	}
+	return nil
+}
+
 // isLoopback reports whether addr binds only to a loopback interface. A bare
 // ":PORT" (host empty) in Go's net.Listen binds ALL interfaces, so we treat
 // that as non-loopback — the safer default for the no-token guard.

internal/api/server.go

@@ -29,6 +29,24 @@ type (
 	response = protocol.Response
 )
 
+// Gate serializes mutating operations. The daemon shares ONE Gate between the
+// socket accept loop (writer lock per mutating op) and the upgrade coordinator
+// (which takes the writer lock right before it re-execs), so a hot upgrade can
+// never fire mid-checkout. It embeds sync.RWMutex so read-only ops still run
+// concurrently under RLock.
+type Gate struct{ sync.RWMutex }
+
+// UpgradeHandler processes the out-of-band "upgrade" op. It's an interface (not
+// a concrete *upgrade.Coordinator) purely to break the api→upgrade→api import
+// cycle: internal/upgrade imports this package for Gate, so api must not import
+// it back.
+type UpgradeHandler interface {
+	// HandleUpgrade handles one "upgrade" request. It writes any result
+	// frame(s) via send. On a successful hot-upgrade it re-execs and NEVER
+	// returns; otherwise it returns after sending exactly one result frame.
+	HandleUpgrade(req protocol.Request, send func(protocol.Response) error)
+}
+
 // Serve listens on sockPath and serves requests until ctx is cancelled.
 //
 // Security: the socket is created with default fs perms (~0755 after umask),
@@ -47,6 +65,23 @@ func Serve(ctx context.Context, r *repo.Repo, c *repo.Capturer, sockPath string)
 		ln.Close()
 		return err
 	}
+	// supervise and plain daemon-without-upgrade get a fresh Gate and no
+	// upgrade handler. The daemon that wants hot-upgrade builds its own
+	// listener (so it can hand off the fd across execve) and calls
+	// ServeListener directly with a shared Gate + coordinator.
+	return ServeListener(ctx, r, c, ln, &Gate{}, nil)
+}
+
+// ServeListener serves the JSON protocol on an already-created listener. It
+// does NOT remove/chmod/bind — the caller owns the listener's lifecycle. This
+// is the seam the hot-upgrade path needs: the daemon builds the unix listener
+// itself, hands ServeListener a shared Gate (so the upgrade coordinator can
+// serialize against mutating ops) and an UpgradeHandler, and on re-exec the new
+// image rebuilds the listener from the inherited fd and calls this again.
+func ServeListener(ctx context.Context, r *repo.Repo, c *repo.Capturer, ln net.Listener, gate *Gate, up UpgradeHandler) error {
+	if gate == nil {
+		gate = &Gate{}
+	}
 	// The listening/uid banner is left to the caller: `daemon` and headless
 	// `supervise` print it (useful operational logging + the cross-uid trap
 	// hint), while interactive `supervise` stays silent so it doesn't pollute
@@ -55,17 +90,16 @@ func Serve(ctx context.Context, r *repo.Repo, c *repo.Capturer, sockPath string)
 		<-ctx.Done()
 		ln.Close()
 	}()
-	// RWMutex (was a plain Mutex) so read-only ops (log, head, branches, show,
+	// The Gate is an RWMutex so read-only ops (log, head, branches, show,
 	// diff, ps, status…) can run concurrently with each other. Without this, a
 	// long-running exec held the global lock and starved every other client —
 	// e.g. an agent's `make` blocked an operator's `agentenv ctl log`.
-	var mu sync.RWMutex
 	for {
 		conn, err := ln.Accept()
 		if err != nil {
 			return nil // listener closed via stop
 		}
-		go handle(r, c, conn, &mu)
+		go handle(r, c, conn, gate, up)
 	}
 }
 
@@ -76,7 +110,7 @@ var readOnlyOps = map[string]bool{
 	"show": true, "diff": true, "ps": true,
 }
 
-func handle(r *repo.Repo, c *repo.Capturer, conn net.Conn, mu *sync.RWMutex) {
+func handle(r *repo.Repo, c *repo.Capturer, conn net.Conn, gate *Gate, up UpgradeHandler) {
 	defer conn.Close()
 	// json.Decoder streams over the connection: no max-line ceiling (bufio.Scanner
 	// had a 16MB cap, after which it silently failed), and EOF / parse errors
@@ -89,26 +123,40 @@ func handle(r *repo.Repo, c *repo.Capturer, conn net.Conn, mu *sync.RWMutex) {
 		if err := dec.Decode(&req); err != nil {
 			return // EOF or malformed: drop the connection
 		}
+		// upgrade is handled entirely by the coordinator: it takes the Gate's
+		// writer lock ITSELF (so we must NOT hold it here — that would
+		// deadlock) and, on success, re-execs and never returns. On the happy
+		// path it writes an ack frame just before exec, so the client that
+		// triggered the upgrade reads that frame and then sees EOF as the
+		// connection dies with the old image.
+		if req.Op == "upgrade" {
+			if up == nil {
+				_ = enc.Encode(response{Error: "upgrade not supported in this mode (only `agentenv daemon` hot-upgrades)"})
+				continue
+			}
+			up.HandleUpgrade(req, func(fr response) error { return enc.Encode(fr) })
+			continue
+		}
 		// Read-only ops use the read lock so concurrent log/diff/show calls
 		// don't serialize behind a long-running exec. Mutating ops (and exec,
 		// which mutates via auto-snapshot) take the writer lock.
 		if readOnlyOps[req.Op] {
-			mu.RLock()
+			gate.RLock()
 			err := enc.Encode(dispatch(r, c, req))
-			mu.RUnlock()
+			gate.RUnlock()
 			if err != nil {
 				return
 			}
 			continue
 		}
-		mu.Lock()
+		gate.Lock()
 		if req.Op == "exec" {
 			dispatchExecStream(r, c, enc, req)
 		} else if err := enc.Encode(dispatch(r, c, req)); err != nil {
-			mu.Unlock()
+			gate.Unlock()
 			return
 		}
-		mu.Unlock()
+		gate.Unlock()
 	}
 }
 

internal/cli/cli.go

@@ -69,6 +69,7 @@ func usage() {
 		fmt.Fprintf(os.Stderr, "  %-26s %s\n", c.name+" "+c.args, c.summary)
 	}
 	fmt.Fprintln(os.Stderr, "  ctl <op> [--socket p]      drive a running daemon/supervise out-of-band (e.g. ctl checkout <id>)")
+	fmt.Fprintln(os.Stderr, "  upgrade [--version X]      hot-upgrade a running daemon to a new release (zero-downtime re-exec)")
 	fmt.Fprintln(os.Stderr, "  mcp [--socket p]           MCP server over stdio (for Claude Code / other MCP hosts)")
 	fmt.Fprintln(os.Stderr, "  version                    print version")
 	fmt.Fprintln(os.Stderr, `

internal/cli/ctl.go

@@ -44,6 +44,13 @@ func cmdCtl(args []string) error {
 		req["cmd"] = strings.Join(a, " ")
 	case "commit":
 		req["message"] = strings.Join(a, " ")
+	case "upgrade":
+		// Manual upgrade always bypasses the maintenance window (force=true).
+		// --version pins an explicit target; omit for "latest".
+		if v := flagValue(a, "--version"); v != "" {
+			req["version"] = v
+		}
+		req["force"] = true
 	case "kill":
 		if len(a) < 1 {
 			return fmt.Errorf("usage: agentenv ctl kill <pid>")
@@ -161,6 +168,12 @@ func ctlPrint(op string, r *protocol.Response) error {
 		for _, p := range r.Procs {
 			fmt.Printf("%d\t%s\n", p.PID, p.Args)
 		}
+	case "upgrade":
+		if r.Message != "" {
+			fmt.Println(r.Message)
+		} else {
+			fmt.Println("ok")
+		}
 	case "tag":
 		switch {
 		case len(r.Tags) > 0:

internal/cli/run.go

@@ -13,15 +13,21 @@ import (
 	"path/filepath"
 
 	"github.com/css521/agentenv/internal/repo"
+	"github.com/css521/agentenv/internal/upgrade"
 )
 
+// binVersion is the running binary's version string (from main.resolveVersion),
+// stashed by Run so command handlers that need it — cmdDaemon, for the upgrade
+// coordinator — can read it without threading it through every handler.
+var binVersion string
+
 // ctlRoutable reports whether a mutating command has an out-of-band (ctl)
 // equivalent it can be transparently routed to when a daemon/supervise holds
 // the repo lock. Session-establishing commands (init/supervise/daemon/shell)
 // are excluded — they must own the lock themselves.
 func ctlRoutable(name string) bool {
 	switch name {
-	case "checkout", "commit", "exec", "tag", "gc", "tournament", "delete":
+	case "checkout", "commit", "exec", "tag", "gc", "tournament", "delete", "upgrade":
 		return true
 	}
 	return false
@@ -53,6 +59,7 @@ func liveControlSocket(root string) string {
 // of the repo lock (and any other defers a handler set up). main() is the only
 // place allowed to terminate the process.
 func Run(version string) int {
+	binVersion = version
 	args := os.Args[1:]
 	if len(args) == 0 {
 		usage()
@@ -86,6 +93,15 @@ func Run(version string) int {
 			return 1
 		}
 		return 0
+	case "upgrade":
+		// Out-of-band: ask a running daemon to hot-upgrade its own binary
+		// (zero-downtime re-exec). Pure socket client — no repo/lock — so it
+		// works while the daemon holds the lock. `agentenv upgrade [--version X]`.
+		if err := cmdCtl(append([]string{"upgrade"}, rest...)); err != nil {
+			fmt.Fprintln(os.Stderr, "error:", err)
+			return 1
+		}
+		return 0
 	}
 
 	cmd := commands[name]
@@ -109,8 +125,14 @@ func Run(version string) int {
 	// session can be writing it, producing a torn read or an inconsistent
 	// in-memory DAG. Read-only commands skip the lock (they only read, they
 	// don't write back, so a slightly stale view is acceptable).
+	// Hot-upgrade re-exec: the daemon inherits the flock via a CLOEXEC-cleared
+	// fd, so it must NOT AcquireLock (a second open file description would
+	// EWOULDBLOCK against the one still held). cmdDaemon rebuilds the *Lock from
+	// the inherited fd. All other commands take the lock normally.
+	reexecDaemon := name == "daemon" && upgrade.IsReexec()
+
 	var lock *repo.Lock
-	if cmd.mutates {
+	if cmd.mutates && !reexecDaemon {
 		var err error
 		lock, err = repo.AcquireLock(root)
 		if err != nil {
@@ -146,6 +168,9 @@ func Run(version string) int {
 		fmt.Fprintln(os.Stderr, "error:", err)
 		return 1
 	}
+	if lock != nil {
+		r.SetLock(lock) // let long-running modes reach the flock fd (hot upgrade)
+	}
 
 	if err := cmd.run(r, rest); err != nil {
 		// ExitError = "the inner command produced a non-zero exit; propagate it