v1.9.1-rc2

Release candidate 2. The headline fix is the relay-flag flap that
caused "pings work for a minute then dial timeout for minutes" against
NAT'd peers (most commonly GCP service agents).

### Relay-flag flap (the "ping dies after a minute" bug)

clearRelayOnDirectLocked auto-flipped the relay flag back to direct
after 3 non-beacon-sourced packets. In production, that "non-beacon"
detection was unreliable: GCP NAT-port-rewrite of beacon-originated
UDP, beacon-mesh forwards from a different IP, and stray punch/ICMP
replies all looked non-beacon to the IP+port equality check. After
the flip, writeFrame happily sent to the unreachable direct address
and only the silent-detection 60 s later flipped back to relay —
during which encrypted sends dropped and rekey rotated session keys
faster than they could settle.

Fixes:

* pkg/daemon/tunnel.go: clearRelayOnDirectLocked auto-recovery
  disabled. Once a peer is on relay, only explicit operator action
  (SetRelayPeer(node, false)) clears the flag.

* pkg/daemon/tunnel.go: handleAuthKeyExchange + handleKeyExchange now
  ALWAYS overwrite tm.peers[X] to the beacon and set+pin the relay
  flag when fromRelay=true, regardless of whether ensureTunnel
  pre-registered the peer with its registry-published direct address.
  The old "only if unknown" gate (issue #199) left relayPeers[X]=false
  for peers that ensureTunnel had pre-registered → writeFrame sent to
  unreachable direct addr → 60 s silent before flip.

* pkg/daemon/tunnel.go: new relayPinned map. SetRelayPeerPinned()
  marks the relay flag as authoritative — registry relay_only=true,
  dial-time direct-failure switch, ICMP-unreachable threshold, and
  beacon-sourced key exchanges all pin. Pinned peers cannot be
  auto-cleared. Wired into the four call sites that flip to relay.

### Performance — pilotctl cold latency

* pkg/daemon/ipc.go: hostname cache (60 s) on the resolve_hostname IPC
  handler. Persisted to ~/.pilot/hostname_cache.json so daemon restart
  starts warm. Async resolve prewarm after a successful lookup so the
  follow-up DialAddr skips the second registry round-trip.

* pkg/daemon/policy_runner.go: fetchMembers exponential backoff
  (5 s → 10 s → ... → 5 min cap) so a doomed list_nodes no longer
  holds the regConn mutex every 5 s and starves co-tenant calls.

* pkg/daemon/daemon.go: DialInitialRTO 1 s → 250 ms. Three direct
  retries now total ~1.75 s before relay flip (was ~7 s).

* cmd/pilotctl/main.go: PILOTCTL_TRACE_TIME=1 env flag dumps per-step
  timings to stderr for latency debugging.

Result: send-message 10–30 s → ~600 ms cold / ~170 ms warm.
        pilotctl ping list-agents holds for 10+ minutes; previous
        builds dropped after ~60–90 s.

### Other

* pkg/daemon/tunnel.go: ErrPendingDropped sentinel. DialConnection
  no longer aborts when a SYN is queued during tunnel-handshake
  startup — falls through to its existing 6-retry loop instead.

* cmd/pilotctl/main.go: classifyDaemonError() recognizes pending-
  queue and dial-timeout patterns and surfaces an actionable hint
  instead of generic "check ping" boilerplate.

* cmd/pilotctl/main.go: pilotctl ping SetReadDeadline before the
  echo read so a relay-degraded peer reports timeout instead of
  hanging forever.

* pkg/skillinject: config-file overlay support for skill paths.

Author: teovl

CHANGELOG.md

@@ -9,6 +9,111 @@ Detailed per-release notes for tagged versions are published on the
 
 ## [Unreleased]
 
+## [1.9.1-rc2] - 2026-05-03
+
+Release candidate 2. Builds on rc1 with three classes of fixes:
+
+### Performance — cold pilotctl latency
+
+- pilotctl `send-message` cold latency cut from 10–30 s to ~600 ms; warm
+  to ~170 ms. Combination of:
+  - Daemon-side hostname cache (60 s, persisted to
+    `~/.pilot/hostname_cache.json`, reloaded at startup).
+  - PolicyRunner `fetchMembers` exponential backoff so a doomed
+    `list_nodes` no longer holds the regConn mutex every 5 s.
+  - `DialInitialRTO` 1 s → 250 ms — three direct retries now total
+    ~1.75 s before relay flip (was ~7 s).
+  - Background pre-warm of the registry resolve cache for trusted
+    peers at daemon start.
+- Resolve prewarm runs in a goroutine after a successful
+  `resolve_hostname` so the follow-up `DialAddr` skips the second
+  registry round-trip.
+
+### Reliability
+
+- Beacon `--advertise-addr` flag — MIG instances behind a public DNAT
+  now register the public ingress address instead of their internal
+  VPC IP. Without this, `beacon_list` returned 10.x addresses that
+  off-VPC clients couldn't reach (silent black-hole). Daemon-side
+  filter (`filterUnreachable` in `beacon_select.go`) drops RFC1918 /
+  loopback / link-local literals from discovered lists as a safety
+  net.
+- `ErrPendingDropped` sentinel in `pkg/daemon/tunnel.go`.
+  `DialConnection` no longer aborts when a SYN is queued during
+  tunnel-handshake startup — falls through to its existing 6-retry
+  loop instead. Eliminates the spurious "send SYN: pending queue
+  full" failures users saw immediately after daemon restart.
+- `pilotctl ping` — added `SetReadDeadline` before the echo read so a
+  relay-degraded peer reports timeout instead of hanging forever.
+- `pilotctl connections` — `PEERWIN` no longer renders as
+  `17179869184.0 GB` for connections in `SYN_SENT`. Was a `int(-1)` →
+  `uint64` wrap in the formatter.
+
+### CLI / docs
+
+- 56 doc files corrected (formal `web/src/pages/docs/` + `plain/docs/`
+  twins, plus 47 blog posts) — stale trust/discovery/pubsub/network-
+  join/tag/install patterns updated to current command surface.
+- `pilotctl` friendlier hints: `classifyDaemonError` recognizes
+  `pending queue full` and `dial timeout` patterns and surfaces an
+  actionable message instead of generic "check ping" boilerplate.
+
+### Fixed — relay-flag flap (the "ping works for a minute then dies" bug)
+
+- `clearRelayOnDirectLocked` (the relay→direct auto-recovery heuristic)
+  was unreliable: after 3 consecutive non-beacon-sourced packets it
+  flipped the relay flag back to direct, the next outgoing packet went
+  direct (against a peer genuinely behind NAT), and writeFrame's silent-
+  detection took ~60 s to flip BACK to relay. During that window
+  encrypted sends to a now-direct peer dropped, and rekey attempts
+  rotated session keys faster than they settled — user-visible symptom
+  was "ping works for the first burst, then dial timeout for minutes".
+  Triggered in production by GCP NAT-port-rewrite of beacon-originated
+  UDP replies (made beacon packets look non-beacon-sourced) and by
+  beacon-mesh forwards that arrive from a different IP.
+- The auto-recovery is now disabled. Once a peer is on relay, only an
+  explicit operator action (`SetRelayPeer(node, false)`) clears the
+  flag. Recovery to direct will happen naturally on the next dial cycle.
+- `handleAuthKeyExchange` and `handleKeyExchange` now ALWAYS overwrite
+  the peer's UDP entry to the beacon and set+pin the relay flag when
+  `fromRelay=true`, regardless of whether `ensureTunnel` previously
+  registered the peer with its registry-published direct address. The
+  old "only if peer was unknown" gate (issue #199) was leaving the
+  relay flag false for peers that ensureTunnel had pre-registered, so
+  writeFrame would happily send to the (unreachable) direct address
+  until silent-detection eventually flipped it.
+- New `relayPinned` map tracks "relay flag set by an authoritative
+  signal" — registry's `relay_only=true`, dial-time direct-failure
+  switch, ICMP-unreachable threshold, or beacon-sourced key exchange.
+  Pinned peers cannot be auto-cleared.
+
+### Performance — cold pilotctl latency
+
+- pilotctl `send-message` cold latency cut from 10–30 s to ~600 ms; warm
+  to ~170 ms. Combination of:
+  - Daemon-side hostname cache (60 s, persisted to
+    `~/.pilot/hostname_cache.json`, reloaded at startup).
+  - PolicyRunner `fetchMembers` exponential backoff so a doomed
+    `list_nodes` no longer holds the regConn mutex every 5 s.
+  - `DialInitialRTO` 1 s → 250 ms — three direct retries now total
+    ~1.75 s before relay flip (was ~7 s).
+  - Async pre-warm of the registry resolve cache after a successful
+    `resolve_hostname` so the follow-up `DialAddr` skips the second
+    registry round-trip.
+
+### Reverted
+
+- An earlier rc2-dev variant pre-warmed encrypted tunnels for trusted
+  peers at daemon start by firing ECDH key exchanges in a 350 ms
+  burst. Triggered a rekey thundering-herd: every prewarm call marked
+  `pendingRekey`, the retransmit loop pounded all peers at once, and
+  peers' replies were interpreted as fresh requests, rotating session
+  keys faster than encrypted packets could decrypt. A second variant
+  (resolve-cache-only prewarm) caused a NAT-punch wave that took
+  ~70 s to settle. Both reverted to lazy-on-first-use; the only
+  prewarm that survives is the per-call resolve prewarm in the
+  `resolve_hostname` IPC handler.
+
 ## [1.8.0] - 2026-04-20
 
 ## [1.7.2] - 2026-04-10

cmd/pilotctl/main.go

@@ -4315,10 +4315,24 @@ func cmdPing(args []string) {
 	count := flagInt(flags, "count", 4)
 	timeout := flagDuration(flags, "timeout", 30*time.Second)
 
+	// Optional per-step timing trace. Set PILOTCTL_TRACE_TIME=1 to see
+	// where the latency goes (binary startup, IPC connect, hostname
+	// lookup, dial, echo). Helps when "ping feels slow before the
+	// first packet" — the answer is almost always step 3.
+	traceTime := os.Getenv("PILOTCTL_TRACE_TIME") != ""
+	t0 := time.Now()
+	tracef := func(label string) {
+		if traceTime {
+			fmt.Fprintf(os.Stderr, "TRACE %-22s %12.3fms\n", label, float64(time.Since(t0).Microseconds())/1000.0)
+		}
+	}
+
 	d := connectDriver()
+	tracef("connectDriver")
 	defer d.Close()
 
 	target, err := parseAddrOrHostname(d, pos[0])
+	tracef("parseAddrOrHostname")
 	if err != nil {
 		fatalCode("not_found", "%v", err)
 	}

pkg/daemon/daemon.go

@@ -800,12 +800,36 @@ func (d *Daemon) Start() error {
 	// hash-pick lands on a fresher beacon.
 	go d.beaconRefreshLoop()
 
-	// 12b. Pre-warm encrypted tunnels for trusted peers so the first
-	// `pilotctl send-message <peer>` after a daemon restart skips the
-	// 2-RTT key-exchange + SYN/SYN-ACK and only pays the 1-RTT
-	// SYN/SYN-ACK relay cost. Background; failures are harmless (the
-	// dial path will redo whatever didn't stick).
-	go d.prewarmTrustedTunnels()
+	// 12b. Earlier rc2-dev iterations pre-warmed trusted peers at
+	// startup. Two variants were tried and both made things worse:
+	//
+	//   (1) sendKeyExchangeToNode for each peer → rekey storm. Every
+	//       call marked pendingRekey; the retransmit loop pounded all
+	//       peers simultaneously; each peer's reply was interpreted as
+	//       a fresh key-exchange request that rotated session keys
+	//       mid-flight. Pings worked for a few seconds then dial
+	//       timed out for minutes.
+	//
+	//   (2) ensureTunnel only (no key exchange) → NAT-punch wave.
+	//       ensureTunnel still issues a punch via beacon for non-
+	//       relay-only peers, and the dial path that owns the punch
+	//       takes 7-15s per peer to time out and flip to relay. With
+	//       7 peers in a 1.4s burst, the daemon was busy doing
+	//       dial-fail-over for ~70s after start. User pings during
+	//       that window collided with the prewarm activity.
+	//
+	// Neither variant survives contact with a fleet that has many
+	// trusted peers. Lazy-on-first-use is the right behavior: pay the
+	// 600ms cold cost on the first send-message and let warm cache
+	// handle everything after.
+	//
+	// Latency wins kept (independent of prewarm):
+	//   - DialInitialRTO 1s → 250ms
+	//   - hostname cache (in-memory + disk-persisted)
+	//   - fetchMembers exponential backoff
+	//   - resolve cache pre-warm via the resolve_hostname IPC handler
+	//     (warms ONLY the peer the user just asked about, not the
+	//     entire trust list)
 
 	// 13. Trusted-agents fetcher: hourly refresh of the auto-accept
 	// list from GitHub. Embedded blob in the binary is the bootstrap.
@@ -2509,10 +2533,17 @@ func (d *Daemon) dialConnectionLocked(ctx context.Context, dstAddr protocol.Addr
 		case <-timer.C:
 			retries++
 
-			// Switch to relay mode after direct retries exhaust
+			// Switch to relay mode after direct retries exhaust. Pin the
+			// relay flag so clearRelayOnDirectLocked won't bounce us back
+			// to direct on a stray non-beacon-sourced packet (NAT-port-
+			// rewrite of beacon replies, ICMP-unreachable from the
+			// rendezvous, beacon mesh forwarding from a different IP).
+			// The peer just demonstrated direct doesn't work — observation
+			// of one or two seemingly-direct packets shouldn't override
+			// that empirical signal.
 			if retries == directRetries && d.config.BeaconAddr != "" && !relayActive {
 				slog.Info("direct dial timed out, switching to relay", "node_id", dstAddr.Node)
-				d.tunnels.SetRelayPeer(dstAddr.Node, true)
+				d.tunnels.SetRelayPeerPinned(dstAddr.Node, true)
 				relayActive = true
 				rto = DialInitialRTO // reset backoff for relay phase
 			}
@@ -3272,28 +3303,26 @@ func (d *Daemon) cacheResolve(nodeID uint32, resp map[string]interface{}) {
 	d.resolveCacheMu.Unlock()
 }
 
-// prewarmTrustedTunnels iterates trusted peers and fires a key_exchange
-// at each one so the encrypted tunnel is established BEFORE the user's
-// first send-message. Without this, the first dial pays:
-//
-//   1 RTT  registry resolve_node (~150ms)
-//   1 RTT  ECDH key exchange     (~170ms via relay)
-//   1 RTT  SYN / SYN-ACK         (~170ms via relay)
-//
-// = ~500ms cold. With prewarm, only the SYN/SYN-ACK round trip remains
-// (~170ms), because both the resolve cache and the encrypted tunnel
-// are already warm.
+// prewarmTrustedResolves iterates trusted peers and warms the registry
+// resolve cache for each. After this, ensureTunnel() on the dial path
+// hits the in-memory cache instead of paying a registry round-trip.
 //
-// Spaced out (50ms between peers) so that 50 trusted peers don't all
-// fire key_exchange at the same beacon at once. The beacon's per-peer
-// reflection queue can absorb bursts but smoothing the registration
-// reduces tail latency for everyone.
-func (d *Daemon) prewarmTrustedTunnels() {
+// Does NOT fire ECDH key exchanges. An earlier version did, and the
+// ~7-trusted-peer fleet observed here triggered a feedback storm:
+// every sendKeyExchangeToNode marked the peer pendingRekey, the
+// retransmit loop fired against ~7 peers simultaneously, and each
+// peer's reply was interpreted by our handler as a fresh key-exchange
+// request that rotated session keys mid-flight. Resulting symptom:
+// pings worked for 5-10 seconds after restart, then dial timed out
+// for minutes as session keys rotated faster than encrypted packets
+// could decrypt. The encryption layer is intentionally left for the
+// first real dial; we just want the resolve cache warm.
+func (d *Daemon) prewarmTrustedResolves() {
 	if d.handshakes == nil {
 		return
 	}
 	// Brief delay so registration + beacon-refresh have settled before
-	// we start hammering the registry with resolves.
+	// we hit the registry.
 	select {
 	case <-time.After(2 * time.Second):
 	case <-d.stopCh:
@@ -3303,30 +3332,28 @@ func (d *Daemon) prewarmTrustedTunnels() {
 	if len(peers) == 0 {
 		return
 	}
-	slog.Info("prewarming encrypted tunnels for trusted peers", "count", len(peers))
+	slog.Info("prewarming resolve cache for trusted peers", "count", len(peers))
 	warmed := 0
 	for _, rec := range peers {
 		if d.stopping() {
 			return
 		}
 		// ensureTunnel fetches the peer's UDP endpoint (real_addr / relay)
-		// and registers it with the tunnel manager. Cheap if cached.
+		// and registers it with the tunnel manager. Populates resolveCache
+		// and epCache as a side effect. NO key exchange.
 		if err := d.ensureTunnel(rec.NodeID); err != nil {
 			slog.Debug("prewarm: ensureTunnel failed", "peer", rec.NodeID, "err", err)
 			continue
 		}
-		// Fire a key_exchange; the peer's reply derives the session key
-		// on both sides. After this, the peer is in tm.encKeys and the
-		// next call goes encrypted without re-keying.
-		d.tunnels.sendKeyExchangeToNode(rec.NodeID)
 		warmed++
+		// 200ms gap so a 50-peer fleet doesn't pile up at the registry.
 		select {
-		case <-time.After(50 * time.Millisecond):
+		case <-time.After(200 * time.Millisecond):
 		case <-d.stopCh:
 			return
 		}
 	}
-	slog.Info("prewarm fired", "peers", warmed)
+	slog.Info("prewarm complete", "peers_warmed", warmed)
 }
 
 // hostnameCachePath returns the on-disk location of the persisted
@@ -3508,7 +3535,12 @@ func (d *Daemon) ensureTunnel(nodeID uint32) error {
 	// flow A → beacon → B as MsgRelay frames.
 	relayOnly, _ := resp["relay_only"].(bool)
 	if relayOnly {
-		d.tunnels.SetRelayPeer(nodeID, true)
+		// Pin the relay flag — the registry's relay_only=true is an
+		// authoritative signal. clearRelayOnDirectLocked must not
+		// auto-flip pinned peers back to direct based on observed
+		// non-beacon source addresses (NAT-port-rewrite, beacon mesh
+		// forward from a different IP, etc).
+		d.tunnels.SetRelayPeerPinned(nodeID, true)
 		d.tunnels.AddPeer(nodeID, udpAddr) // udpAddr is already the beacon (substituted by registry)
 		slog.Debug("peer is RelayOnly, skipping hole punch", "node_id", nodeID)
 		return nil

pkg/daemon/ipc.go

@@ -854,10 +854,17 @@ func (s *IPCServer) handleResolveHostname(conn *ipcConn, payload []byte) {
 
 	// Cache the successful response. Only cache positive results so
 	// transient registry hiccups don't poison the cache with a bogus
-	// "not found".
-	s.daemon.hostnameCacheMu.Lock()
-	s.daemon.hostnameCache[hostname] = &hostnameCacheEntry{resp: result, cachedAt: now}
-	s.daemon.hostnameCacheMu.Unlock()
+	// "not found". Skip caching when the result points at our own
+	// node — local hostname state is owned by the daemon and changes
+	// through paths that don't always invalidate the cache (admin
+	// tools talking directly to the registry, tests bypassing IPC).
+	if nidVal, ok := result["node_id"].(float64); ok && uint32(nidVal) == s.daemon.NodeID() {
+		// fall through — no cache write, no prewarm, no disk persist
+	} else {
+		s.daemon.hostnameCacheMu.Lock()
+		s.daemon.hostnameCache[hostname] = &hostnameCacheEntry{resp: result, cachedAt: now}
+		s.daemon.hostnameCacheMu.Unlock()
+	}
 
 	// Pre-warm the node-resolve cache so a follow-up DialAddr call from the
 	// same pilotctl invocation skips the second registry round-trip in
@@ -897,6 +904,11 @@ func (s *IPCServer) handleResolveHostname(conn *ipcConn, payload []byte) {
 
 func (s *IPCServer) handleSetHostname(conn *ipcConn, payload []byte) {
 	hostname := string(payload)
+	// Snapshot the previous hostname so we can invalidate it from the
+	// cache regardless of whether this is a set, change, or clear.
+	s.daemon.addrMu.RLock()
+	prevHostname := s.daemon.config.Hostname
+	s.daemon.addrMu.RUnlock()
 	result, err := s.daemon.regConn.SetHostname(s.daemon.NodeID(), hostname)
 	if err != nil {
 		s.sendError(conn, fmt.Sprintf("set_hostname: %v", err))
@@ -906,6 +918,17 @@ func (s *IPCServer) handleSetHostname(conn *ipcConn, payload []byte) {
 	s.daemon.addrMu.Lock()
 	s.daemon.config.Hostname = hostname
 	s.daemon.addrMu.Unlock()
+	// Invalidate any cache entries that this change could have made stale:
+	// the old name (we may have cached resolutions for it) and the new
+	// name (could be a re-bind from another node we previously cached).
+	s.daemon.hostnameCacheMu.Lock()
+	if prevHostname != "" {
+		delete(s.daemon.hostnameCache, prevHostname)
+	}
+	if hostname != "" {
+		delete(s.daemon.hostnameCache, hostname)
+	}
+	s.daemon.hostnameCacheMu.Unlock()
 	data, err := json.Marshal(result)
 	if err != nil {
 		s.sendError(conn, fmt.Sprintf("set_hostname marshal: %v", err))