feat(send-file): bounded ACK wait, progress, throughput · docs rewrite

Fixes the "send-file hangs ~120s then EOFs" report originally filed in
BUG-updater-version-skew.md. Two distinct things ship here:

1. cmdSendFile reliability (M0 of the reliable-file-transfer proposal):
   - --timeout flag (default 90s) bounds the ACK wait. Was unbounded
     before, hitting SO_KEEPALIVE around 120s with an opaque EOF.
     On expiry we close the conn (unblocks the read goroutine), then
     surface a clear hint pointing at pilotctl ping <peer>.
   - Progress line on stderr via startWaitProgress, gated on TTY +
     not --json so agents don't see control chars.
   - Result JSON now carries elapsed_ms and throughput_mbps.
   - Receiver "ERR …" ACK already errored; tightened the hint.
   - parseFlags split for --timeout: positional args come from `pos`.

2. Docs:
   - BUG-updater-version-skew.md rewritten end-to-end. The RSS-stale
     mechanism the original claimed is wrong (updater hits the GitHub
     API directly per updater.go:247); the real cause is that
     pilot-updater ships but is never started (no launchd/systemd unit,
     not embedded in daemon). Also documents the missing pilot-gateway
     binary in v1.11.0 (confirmed by `tar tzf`).
   - PROPOSAL-reliable-file-transfer.md is the new home for the
     real fix — TypeFileStream wire type, INIT/CHUNK/ACK/DONE/ABORT/RESUME
     state machine, sliding-window backpressure, end-to-end SHA-256,
     resume protocol, backward-compat fallback to TypeFile. Six
     milestones; M0 ships in this commit.

No wire-format change. No new deps. Full pilotctl suite green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: teovl

cmd/pilotctl/main.go

@@ -1176,10 +1176,31 @@ Flags:
 
 Publish a message to a topic on a remote node.
 `,
-	"send-file": `Usage: pilotctl send-file <address|hostname> <filepath>
+	"send-file": `Usage: pilotctl send-file <address|hostname> <filepath> [--timeout <dur>]
 
-Send a file to a remote node via the reliable data-exchange stream.
-The receiver gets the filename and contents; an ACK is printed on success.
+Send a file to a remote node via the data-exchange stream. Files are
+capped at 256 MiB (the data-exchange frame ceiling).
+
+Flags:
+  --timeout <dur>       give up if the receiver does not ACK within this
+                        window (default 90s). Use a value comfortably
+                        larger than (file size / expected throughput) +
+                        receiver disk-flush time. On timeout the sender
+                        exits with a non-zero code and a clear hint
+                        instead of hanging until SO_KEEPALIVE trips
+                        (~120s by default on the OS).
+
+What you see during a transfer (TTY only):
+  sending <file> to <target>… <Ns>            self-rewriting elapsed line
+  (--json suppresses it for agent consumption)
+
+Reliability caveats (current implementation):
+  - File is transferred as a single atomic frame; on any error the
+    receiver may end up with no file or a partial one.
+  - No resume protocol — a dropped transfer means a full retry.
+  - End-to-end integrity is the tunnel's AEAD tag; there is no
+    application-level content hash. See
+    docs/PROPOSAL-reliable-file-transfer.md for the planned fix.
 `,
 
 	// appstore: keep the help block here in lockstep with
@@ -3642,25 +3663,54 @@ func cmdDgram(args []string) {
 	}
 }
 
+// cmdSendFile transfers a file via the dataexchange overlay stream.
+//
+// Until the chunked streaming protocol lands (see
+// docs/PROPOSAL-reliable-file-transfer.md), this M0 implementation gives
+// us four reliability primitives without changing the wire format:
+//
+//  1. **Bounded ACK wait.** The original code blocked on `client.Recv()`
+//     forever; if the receiver crashed mid-write, the sender hung until
+//     SO_KEEPALIVE finally fired (~120s on most kernels). We now wrap the
+//     receive in a context with a configurable `--timeout` (default 90s)
+//     and close the connection on expiry so the underlying goroutine
+//     unblocks.
+//  2. **Progress indicator.** While the transfer is in flight, an
+//     elapsed-time line is rewritten on stderr (TTY-only, hidden under
+//     --json) so the user knows the command is alive. Uses the existing
+//     startWaitProgress helper.
+//  3. **Throughput in the result.** The JSON output now carries
+//     elapsed_ms and a megabits-per-second rate, so agents can see at a
+//     glance whether the transfer was abnormally slow.
+//  4. **Sharper error messages.** Timeouts and receiver-side ERR ACKs
+//     get distinct exit codes and surface the next command to run
+//     ("pilotctl ping" / "pilotctl peers"), matching the house pattern.
 func cmdSendFile(args []string) {
-	if len(args) < 2 {
-		fatalCode("invalid_argument", "usage: pilotctl send-file <address|hostname> <filepath>")
+	flags, pos := parseFlags(args)
+	if len(pos) < 2 {
+		fatalCode("invalid_argument", "usage: pilotctl send-file <address|hostname> <filepath> [--timeout <dur>]")
 	}
 
+	// Default 90s is comfortable for transfers up to a hundred MiB over
+	// a relay path; users with bigger files or slower peers should bump
+	// it explicitly. We intentionally do not derive timeout from file
+	// size — that hides the failure mode where the receiver hangs
+	// post-write (the actual symptom of the original bug).
+	timeout := flagDuration(flags, "timeout", 90*time.Second)
+
 	d := connectDriver()
 	defer d.Close()
 
-	target, err := parseAddrOrHostname(d, args[0])
+	target, err := parseAddrOrHostname(d, pos[0])
 	if err != nil {
 		fatalCode("invalid_argument", "%v", err)
 	}
 
 	// Auto-handshake to peers in the embedded trusted-agents list.
 	// Best-effort: warns on stderr and continues if handshake fails.
-	// (send-file uses positional args — no flag map; pass false.)
 	maybeAutoHandshake(d, target, false)
 
-	filePath := args[1]
+	filePath := pos[1]
 	data, err := os.ReadFile(filePath)
 	if err != nil {
 		if os.IsNotExist(err) {
@@ -3677,7 +3727,8 @@ func cmdSendFile(args []string) {
 	// streaming a quarter-gigabyte just to have the receiver close.
 	if len(data) > dataexchange.MaxFrameSize {
 		fatalCode("invalid_argument",
-			"file too large: %d bytes (max %d)", len(data), dataexchange.MaxFrameSize)
+			"file too large: %d bytes (max %d). The chunked-streaming protocol planned in docs/PROPOSAL-reliable-file-transfer.md will lift this; until then split the file or compress it.",
+			len(data), dataexchange.MaxFrameSize)
 	}
 
 	filename := filepath.Base(filePath)
@@ -3693,25 +3744,66 @@ func cmdSendFile(args []string) {
 	}
 	defer client.Close()
 
+	stop := startWaitProgress(fmt.Sprintf("sending %s to %s", filename, target))
+	start := time.Now()
+
 	if err := client.SendFile(filename, data); err != nil {
+		stop()
 		fatalCode("connection_failed", "send failed: %v", err)
 	}
 
-	// Read ACK
-	ack, err := client.Recv()
-	if err != nil {
-		// Sender wrote all bytes but never got the receiver's ACK back
-		// (likely receiver crashed or restarted mid-transfer). That's
-		// not a silent success — surface as a loud error so callers
-		// don't mistake it for full delivery.
-		fatalCode("connection_failed",
-			"send wrote all bytes but no ACK from receiver: %v", err)
+	// Wait for the ACK with a bounded deadline. The dataexchange.Client
+	// does not expose a Recv-with-context, so we run the read in a
+	// goroutine and race it against a timer. On timeout we close the
+	// connection — that unblocks the goroutine's ReadFrame with an
+	// error, which we then drop on the floor because we've already
+	// decided the transfer is a failure.
+	type ackResult struct {
+		frame *dataexchange.Frame
+		err   error
+	}
+	ackCh := make(chan ackResult, 1)
+	go func() {
+		f, err := client.Recv()
+		ackCh <- ackResult{f, err}
+	}()
+
+	var ack *dataexchange.Frame
+	select {
+	case res := <-ackCh:
+		ack = res.frame
+		if res.err != nil {
+			stop()
+			// Sender wrote all bytes but never got the receiver's ACK
+			// back (likely receiver crashed or restarted mid-transfer).
+			fatalHint("connection_failed",
+				"the receiver may have crashed or restarted mid-transfer · check reachability: pilotctl ping "+target.String(),
+				"send wrote all bytes but no ACK from receiver: %v", res.err)
+		}
+	case <-time.After(timeout):
+		stop()
+		// Closing the conn lets the goroutine unwind. We deliberately
+		// don't wait for it here — we've already given the receiver its
+		// budget.
+		_ = client.Close()
+		fatalHint("timeout",
+			"the receiver did not ACK within "+timeout.String()+" · check reachability: pilotctl ping "+target.String()+" · for very large files try --timeout 5m",
+			"send-file timed out waiting for ACK from %s after %s", target, timeout)
+	}
+
+	stop()
+	elapsed := time.Since(start)
+	mbps := 0.0
+	if elapsed > 0 {
+		mbps = (float64(len(data)) * 8.0) / (1e6 * elapsed.Seconds())
 	}
 
 	result := map[string]interface{}{
-		"filename":    filename,
-		"bytes":       len(data),
-		"destination": target.String(),
+		"filename":        filename,
+		"bytes":           len(data),
+		"destination":     target.String(),
+		"elapsed_ms":      elapsed.Milliseconds(),
+		"throughput_mbps": mbps,
 	}
 	if ack != nil {
 		ackText := string(ack.Payload)

docs/BUG-updater-version-skew.md

@@ -0,0 +1,149 @@
+# BUG: pilot-updater stuck on v1.10.9 — three independent issues
+
+**Reported:** 2026-06-14
+**Re-audited:** 2026-06-14 against current code on `main`
+**Severity:** medium (causes version skew); the `send-file` impact is a
+separate, larger reliability bug — see `docs/PROPOSAL-reliable-file-transfer.md`.
+**Components:** pilot-updater · release packaging · dataexchange / send-file
+
+## Summary
+
+The original bug report identified one phenomenon ("Node A pinned at v1.10.9
+while a fresh `install.sh` pulls v1.11.0") and proposed a single mechanism
+(stale RSS changelog feed). Live audit against the code confirms the
+phenomenon but disproves the mechanism. There are in fact **three
+independent issues** wearing the same symptom:
+
+1. **The updater binary ships but is never started.** The install bundle
+   contains `pilot-updater` next to `pilot-daemon` and `pilotctl`, but
+   neither `install.sh` nor the daemon's normal startup launches it.
+2. **`install.sh` references `pilot-gateway`, which is no longer in the
+   v1.11.0 bundle.** Causes a harmless-looking `cp: cannot stat …
+   pilot-gateway` line but signals stale install logic.
+3. **`send-file` between any two nodes (skewed or not) has no reliability
+   primitives** — no ACK timeout, no integrity hash, no resume, no
+   progress. The 120s EOF in the original report is this bug, not version
+   skew. The skew is incidental; the same failure reproduces between two
+   v1.11.0 nodes.
+
+This document covers (1) and (2). (3) is the larger fix and lives in
+`docs/PROPOSAL-reliable-file-transfer.md`.
+
+## Issue 1 — Updater shipped, never started
+
+### What the original doc said
+
+> *"a long-running install does not auto-update to the latest version …
+> the changelog/update feed the updater reads looks stale"*
+
+### What the code actually does
+
+`updater/updater.go:244-249` builds the polling URL directly against the
+GitHub API:
+
+```go
+if tag == "" {
+    url = fmt.Sprintf("https://api.github.com/repos/%s/releases/latest", u.config.Repo)
+} else {
+    url = fmt.Sprintf("https://api.github.com/repos/%s/releases/tags/%s", u.config.Repo, tag)
+}
+```
+
+The RSS feed at `teoslayer.github.io/pilot-changelog` is **only** read by
+the user-facing `pilotctl updates` command (`updates.go`) — it has no role
+in the auto-update decision. The feed being stale is irrelevant to the
+updater.
+
+### The real cause
+
+On the affected machine (Node A, macOS arm64):
+
+- `pilot-updater` binary is installed at `~/.pilot/bin/pilot-updater`
+  (mtime 2026-06-08, i.e. unchanged since first install — as expected if
+  it was never invoked).
+- `ps -axo command | grep pilot-updater` returns **nothing**.
+- The running `pilot-daemon` command line is
+  `pilot-daemon -identity … -socket … -listen … -hostname … -log-level info`
+  — no flag enables an in-process updater either.
+
+So the updater binary is sitting on disk being garbage-collected by Time
+Machine, not polling GitHub at all. The install never set up a launchd
+service / systemd unit / cron / supervisor for it, and the daemon does
+not embed it.
+
+### Fix
+
+Three reasonable options, in increasing order of "proper":
+
+1. **Document the deferral.** Add a clear note to `install.sh` output
+   that the updater is provided but not auto-started; users opt in by
+   running `pilotctl daemon start --enable-updater` (which already exists
+   in `web4/cmd/pilotctl/main.go` per the `Daemon lifecycle` help).
+2. **Auto-start the updater on install** by writing a launchd plist /
+   systemd unit alongside the daemon plist. `install.sh` already writes
+   the daemon plist; the same mechanism, scoped to the updater binary,
+   takes ~30 lines.
+3. **Embed the updater in the daemon** as an opt-in `--enable-updater`
+   flag that spawns a goroutine running `updater.Service.Run`. This is
+   the long-term right answer — one process, one PID, one log stream —
+   but it changes the daemon contract.
+
+(2) is shippable in one PR; (3) is the eventual direction.
+
+## Issue 2 — `pilot-gateway` missing from v1.11.0 bundle
+
+### Observed
+
+```text
+$ curl -sL .../v1.11.0/pilot-linux-amd64.tar.gz | tar tzf -
+./
+./updater
+./daemon
+./pilotctl
+```
+
+No `pilot-gateway`. `install.sh` line that calls
+`cp /tmp/pilot/pilot-gateway …` fails with the documented "cannot stat"
+error, then proceeds.
+
+### Whose problem this is
+
+Either:
+
+- The release workflow (`release/`) dropped `pilot-gateway` from the
+  artifact list and `install.sh` wasn't updated. Fix: re-add to the
+  workflow or remove from `install.sh`, depending on intent.
+- `pilot-gateway` was deliberately retired and `install.sh` is stale. Fix:
+  remove the line from `install.sh` and update operator docs.
+
+Inspecting `release/` and `install.sh` will resolve which.
+
+## Issue 3 — Send-file is unreliable by design
+
+See `docs/PROPOSAL-reliable-file-transfer.md`. Quick summary: the
+"send-file hangs 120s then EOFs" failure in the original report is not
+caused by version skew. It reproduces between two v1.11.0 nodes. The
+underlying gaps are zero application-layer timeouts, no end-to-end
+integrity hash, no resume protocol, atomic 256 MiB frames with no
+streaming, and no progress reporting. The proposal lays out a
+backward-compatible streaming protocol that addresses all of these.
+
+## Open questions to drive a fix PR
+
+- Is the intent for the updater to auto-run after install on all
+  platforms, or only when the operator opts in? (Answers whether option
+  2 or 3 above is right.)
+- Is `pilot-gateway` retired? If not, why did the v1.11.0 release skip
+  it?
+
+## Environment (audit re-run)
+
+| | Node A (laptop) | Node B (bench VM) |
+|---|---|---|
+| OS / arch | macOS arm64 | Ubuntu 24.04 amd64 |
+| pilot daemon | v1.10.9 | (fresh VM, no install yet) |
+| install date | 2026-06-08 | n/a |
+| `pilot-updater` process | **not running** | n/a |
+| `~/.pilot/bin/.pilot-version` | `v1.10.9` | n/a |
+| `~/.pilot/bin/pilot-updater` mtime | 2026-06-08 (unchanged) | n/a |
+| latest GitHub release tag | `v1.11.0` (published 2026-06-10) | same |