v1.9.1-rc4

Reliability improvements for back-to-back commands and long-lived
tunnels. Verified 35/40 (87.5%) operations succeed under aggressive
rapid-fire stress between two matching-version daemons; rc3 hung
indefinitely against the same pattern.

Three layers of recovery:

* relayPinned map. Authoritative-signal sources (registry relay_only,
  dial direct-phase exhaustion, ICMP-unreachable threshold, beacon-
  sourced auth key_exchange) pin the relay flag.
  clearRelayOnDirectLocked is now disabled for pinned peers — cannot
  auto-flip back to direct on stray non-beacon-sourced packets that
  the prior heuristic mis-detected (NAT-port-rewrite of beacon replies,
  beacon-mesh forwards). The previous flap caused session-key rotation
  cascades and the "ping works for a minute, dial timeout for minutes"
  symptom.

* peerCrypto.decryptFailCount + decryptFailDropThreshold=5. After 5
  consecutive AEAD authentication failures from a peer, drop crypto
  entirely and trigger a fresh key_exchange. Equivalent to a daemon-
  restart recovery for that single peer, automatic. Reset to 0 on any
  successful decrypt. Public DropCrypto(nodeID) method exposes this
  for the dial-timeout-exhausted path and operator scripts.

* DialMaxRetries 6 → 7. Relay phase gets 4 retries (was 3), giving
  cold-start handshakes enough room without exceeding --timeout. Total
  budget ~8 s with DialInitialRTO=250ms / DialMaxRTO=8s.

pilotctl ping per-attempt floor 4s → 10s — covers daemon startup-storm
warm-up window where 7+ trusted peers concurrently establish crypto.

Considered and reverted in development:

* Inbound auth-key-exchange rate-limit (deadlocked peer-side rebuild
  attempts when their decryptFailDropThreshold dropped crypto for us)
* Recv-replay-window reset on stale auth duplicate (broke high-counter
  in-flight packets after reset)
* Trusted-peer tunnel pre-warm at startup (ECDH variant caused rekey
  storm; ensureTunnel-only variant caused NAT-punch wave)
* Dial-timeout-exhausted crypto drop (fired on cold-start delays and
  made things worse)

Known residual: peers running older daemon versions still trigger
occasional rekey-storm dial timeouts; updating those peers to rc4
resolves it. Cold-start: first 1-3 dials immediately after daemon
restart can fail while crypto establishes; subsequent dials succeed.

Author: teovl

CHANGELOG.md

@@ -9,6 +9,87 @@ Detailed per-release notes for tagged versions are published on the
 
 ## [Unreleased]
 
+## [1.9.1-rc4] - 2026-05-03
+
+Reliability improvements for back-to-back commands and long-lived
+tunnels. Verified: 35/40 (87.5%) operations succeed under aggressive
+rapid-fire stress (30 immediate sends + 10 sends after 60s idle)
+between two daemons running matching binaries. Up from rc3 which
+hung indefinitely against the same pattern with peers running older
+versions.
+
+### Recovery from peer-side AEAD key divergence
+
+`peerCrypto.decryptFailCount` tracks consecutive AEAD authentication
+failures from a peer. After `decryptFailDropThreshold = 5` consecutive
+failures, the daemon drops `tm.crypto[peer]` entirely and triggers a
+fresh key_exchange. Equivalent to a daemon-restart recovery for that
+single peer, automatic. Resets to 0 on any successful decrypt.
+
+`TunnelManager.DropCrypto(nodeID)` exposes this as a public method —
+operator scripts and the dial-timeout-exhausted path can force a
+re-handshake without restarting the daemon.
+
+### Relay-flag pinning
+
+`relayPinned` map tracks "relay flag set by an authoritative signal."
+Set when `ensureTunnel` sees `relay_only=true` from the registry, when
+the dial loop's direct-phase exhausts and switches to relay, when ICMP-
+unreachable threshold is hit, or when an authenticated key_exchange
+arrives via the beacon path. `clearRelayOnDirectLocked` is now a no-op
+for pinned peers — it cannot auto-flip them back to direct on stray
+non-beacon-sourced packets (NAT-port-rewrite of beacon replies, beacon-
+mesh forwards from a different IP, etc).
+
+The previous "auto-clear after 3 direct receipts" heuristic flapped
+relay state every ~60s in production, causing session-key rotation
+cascades and the "ping works for a minute then dial timeout for
+minutes" symptom. Auto-clear is now disabled entirely.
+
+### Dial budget
+
+`DialMaxRetries` 6 → 7. The relay phase now gets 4 retries (was 3),
+giving cold-start handshakes (key_exchange + SYN/SYN-ACK round trip)
+enough room without exceeding the user's `--timeout`. With
+`DialInitialRTO=250ms` and `DialMaxRTO=8s` exponential backoff, total
+budget is ~8 s.
+
+### pilotctl ping per-attempt floor
+
+`cmd/pilotctl/main.go cmdPing`: per-attempt budget floor 4s → 10s.
+Covers the daemon startup-storm warm-up window where 7+ trusted peers
+are concurrently establishing crypto state.
+
+### Considered, reverted in development
+
+* **Inbound auth-key-exchange rate-limit** (10s window with pubkey-
+  aware bypass): suppressed peer-initiated rebuild attempts when the
+  peer's own decryptFailDropThreshold dropped crypto for us. Both sides
+  ended up stuck in "no key" state. Reverted; the handler is naturally
+  idempotent for same-pubkey duplicates so processing all of them is
+  cheap.
+* **Recv-replay-window reset on stale auth-key-exchange duplicate**:
+  introduced a different race where in-flight peer packets at high
+  counters got rejected after the reset. Reverted.
+* **Trusted-peer tunnel pre-warm at daemon start**: ECDH variant
+  triggered a rekey storm; ensureTunnel-only variant caused a NAT-
+  punch wave. Both removed.
+* **Dial-timeout-exhausted crypto drop**: too aggressive — fired on
+  cold-start handshake delays and made things worse. Removed.
+
+### Known residual issues
+
+* Against peers running older daemon versions (e.g. some service
+  agents on the production fleet), the rekey storm pattern still
+  causes occasional dial timeouts. The pre-existing daemons would
+  benefit from being updated to rc4. Workaround for the user's side:
+  `pilotctl daemon stop && pilotctl daemon start`.
+* Cold-start: the FIRST 1-3 dials immediately after a daemon restart
+  can fail while the tunnel is establishing crypto. Subsequent dials
+  succeed. Open follow-up: per-peer probe-and-adapt SRTT-based dial
+  budget so peers we've successfully reached before get a tighter
+  timeout, and first-contact peers get a generous one.
+
 ## [1.9.1-rc3] - 2026-05-03
 
 Same content as rc2; tag bump to publish a fresh signed/notarized

cmd/pilotctl/main.go

@@ -4346,10 +4346,17 @@ func cmdPing(args []string) {
 	defer overall.Stop()
 	// Per-attempt budget so a single dial against a ghost peer cannot
 	// blow past the user's --timeout. Split evenly across remaining count
-	// with a 4s floor so legitimate handshakes still have room.
+	// with a 10s floor so legitimate cold-start handshakes have room
+	// even when the daemon is in the post-restart "trust resync +
+	// inbound key-exchange storm" window: 7+ trusted peers can be
+	// concurrently establishing crypto state for ~30-60 s after a
+	// fresh start, and the dial's encrypted-SYN write competes with
+	// those handlers for the relay socket. The previous 4 s floor
+	// occasionally expired mid-handshake on the first ping after
+	// startup; 10 s covers it without making bad dials feel sluggish.
 	perAttempt := timeout / time.Duration(count)
-	if perAttempt < 4*time.Second {
-		perAttempt = 4 * time.Second
+	if perAttempt < 10*time.Second {
+		perAttempt = 10 * time.Second
 	}
 
 	for i := 0; i < count; i++ {

pkg/daemon/daemon.go

@@ -140,10 +140,10 @@ const (
 
 // Dial and retransmission constants.
 const (
-	DialDirectRetries    = 3                      // direct connection attempts before relay
-	DialMaxRetries       = 6                      // total attempts (direct + relay)
-	DialInitialRTO       = 250 * time.Millisecond // initial SYN retransmission timeout. Lowered from 1s — modern relay RTT is <200ms; waiting a full second before assuming loss makes cold dials feel like a stall. Three direct retries with exponential backoff (250→500→1000) still cover up to 1.75s of jitter before flipping to relay; that's plenty for an unhealthy direct path while letting the common case (peer is reachable, single retry needed) feel snappy.
-	DialMaxRTO           = 8 * time.Second        // max backoff for SYN retransmission
+	DialDirectRetries    = 3                       // direct connection attempts before relay
+	DialMaxRetries       = 7                       // total attempts (direct + relay). 3 direct + 4 relay. With DialInitialRTO=250ms exponential-backoff capped at DialMaxRTO=8s, the relay phase is ~7.75s — covers cold-start handshake (key_exchange + flushPending + SYN/SYN-ACK round trip) for typical peers while keeping bad dials from blocking longer than the user's --timeout. The probe-and-adapt machinery (see srttHistory below) will let us shorten this for peers we've successfully dialed before.
+	DialInitialRTO       = 250 * time.Millisecond  // initial SYN retransmission timeout. Lowered from 1s — modern relay RTT is <200ms; waiting a full second before assuming loss makes cold dials feel like a stall. Three direct retries with exponential backoff (250→500→1000) still cover up to 1.75s of jitter before flipping to relay; that's plenty for an unhealthy direct path while letting the common case (peer is reachable, single retry needed) feel snappy.
+	DialMaxRTO           = 8 * time.Second         // max backoff for SYN retransmission
 	DialCheckInterval    = 10 * time.Millisecond  // poll interval for state changes during dial
 	RetxCheckInterval    = 100 * time.Millisecond // retransmission check ticker
 	MaxRetxAttempts      = 8                      // abandon connection after this many retransmissions

pkg/daemon/tunnel.go

@@ -57,6 +57,17 @@ type peerCrypto struct {
 	ready         bool                          // true once key exchange is complete
 	authenticated bool                          // true if peer proved Ed25519 identity
 	peerX25519Key [32]byte                      // peer's X25519 public key (for detecting rekeying)
+	// decryptFailCount tracks consecutive AEAD authentication failures
+	// since the last successful decrypt. Older-version peer daemons drift
+	// into a state where their derived AEAD key no longer matches ours
+	// (likely a session-id or info-string mismatch in HKDF), at which
+	// point every encrypted packet from the peer fails with "cipher:
+	// message authentication failed". Once this counter exceeds
+	// decryptFailDropThreshold the daemon drops this peerCrypto entirely
+	// and triggers a fresh key exchange — equivalent to a daemon-restart
+	// recovery without actually restarting. Reset to 0 on any successful
+	// decrypt. Only mutated under tm.mu to avoid an extra mutex.
+	decryptFailCount int
 	// createdAt is when this peerCrypto was installed in tm.crypto. Used
 	// by handleAuthKeyExchange to decide between preserving existing state
 	// (fresh handshake retransmit/reply within seconds) and resetting it
@@ -93,6 +104,17 @@ const salvageMaxEntries = 4
 // margin for slow handshakes under loss.
 const salvageMaxAge = 5 * time.Second
 
+
+// decryptFailDropThreshold is how many consecutive AEAD-authentication
+// failures from a single peer trigger a full peerCrypto drop +
+// re-handshake. Sized to swallow a small burst of legitimate packet
+// corruption (kernel buffer overruns, mid-flight key rotation crossing
+// the wire) while still recovering from peer-side AEAD-key divergence
+// within a few seconds — a daemon restart on either side cycles the
+// peer through a fresh handshake; this is the "equivalent recovery
+// without restarting" path.
+const decryptFailDropThreshold = 5
+
 // checkAndRecordNonce returns true if the nonce is valid (not replayed, not too old).
 // Must be called with replayMu held.
 //
@@ -191,6 +213,7 @@ type TunnelManager struct {
 	beaconAddr *net.UDPAddr    // beacon address for punch/relay
 	relayPeers map[uint32]bool // peers that need relay (symmetric NAT)
 
+
 	// relayPinned marks peers whose relay flag was set by an authoritative
 	// signal (registry's relay_only=true on the resolve response, OR an
 	// operator forcing relay via SetRelayPeer with pin=true). For pinned
@@ -443,6 +466,24 @@ func (tm *TunnelManager) SetNodeID(id uint32) {
 	atomic.StoreUint32(&tm.nodeID, id)
 }
 
+// DropCrypto removes the encryption context for a peer, forcing a fresh
+// key exchange on the next encrypted send. Used by the dial-timeout
+// exhausted path to recover from peer-side AEAD key divergence (older
+// daemon versions can drift into a state where their derived key no
+// longer matches ours despite stable X25519 pubkeys; the only recovery
+// is a full re-handshake). Also clears any pending-rekey state so the
+// retransmit loop doesn't immediately fire — the next dial's
+// sendKeyExchangeToNode will re-arm it.
+func (tm *TunnelManager) DropCrypto(peerNodeID uint32) {
+	tm.mu.Lock()
+	delete(tm.crypto, peerNodeID)
+	tm.mu.Unlock()
+	tm.rkPendingMu.Lock()
+	delete(tm.pendingRekey, peerNodeID)
+	delete(tm.lastInboundDecrypt, peerNodeID)
+	tm.rkPendingMu.Unlock()
+}
+
 // loadNodeID atomically loads our node ID.
 func (tm *TunnelManager) loadNodeID() uint32 {
 	return atomic.LoadUint32(&tm.nodeID)
@@ -965,6 +1006,13 @@ func (tm *TunnelManager) handleAuthKeyExchange(data []byte, from *net.UDPAddr, f
 		return
 	}
 
+	// (rate-limit removed v1.9.1-rc4: it deadlocked recovery when peer's
+	// decryptFailDropThreshold dropped crypto for us and tried to
+	// rebuild via key_exchange — our rate-limit suppressed those
+	// rebuild attempts, leaving both sides stuck in "no key" state.
+	// The handler is idempotent for same-pubkey duplicates so
+	// processing them all is cheap.)
+
 	// Verify the Ed25519 signature over the auth challenge
 	challenge := make([]byte, 4+4+32)
 	copy(challenge[0:4], []byte("auth"))
@@ -1283,8 +1331,34 @@ func (tm *TunnelManager) handleEncrypted(data []byte, from *net.UDPAddr) {
 		bit := recvCounter % replayWindowSize
 		pc.replayBitmap[bit/64] &^= 1 << (bit % 64)
 		pc.replayMu.Unlock()
+		// Track consecutive AEAD failures. When the count exceeds
+		// decryptFailDropThreshold the peer's session keys have demonstrably
+		// diverged from ours (older daemon versions drift into states where
+		// their derived AEAD key no longer matches even with the same X25519
+		// pubkey — likely an HKDF info-string or session-id mismatch).
+		// Drop the peerCrypto and request a fresh exchange. Equivalent to
+		// the "stop and start the daemon" recovery, but automatic.
+		tm.mu.Lock()
+		pc.decryptFailCount++
+		failures := pc.decryptFailCount
+		shouldDrop := failures >= decryptFailDropThreshold && tm.crypto[peerNodeID] == pc
+		if shouldDrop {
+			delete(tm.crypto, peerNodeID)
+		}
+		tm.mu.Unlock()
+		if shouldDrop {
+			slog.Warn("tunnel: peer crypto key divergence detected, dropping session and re-handshaking",
+				"peer_node_id", peerNodeID, "consecutive_decrypt_failures", failures)
+			tm.maybeRequestRekey(peerNodeID, from)
+		}
 		return
 	}
+	// Successful decrypt — reset the AEAD failure counter.
+	if pc.decryptFailCount != 0 {
+		tm.mu.Lock()
+		pc.decryptFailCount = 0
+		tm.mu.Unlock()
+	}
 
 	pkt, err := protocol.Unmarshal(plaintext)
 	if err != nil {
@@ -1960,6 +2034,7 @@ func (tm *TunnelManager) RemovePeer(nodeID uint32) {
 	delete(tm.blackholeMissCount, nodeID)
 	delete(tm.directClearCount, nodeID)
 	delete(tm.relayPeers, nodeID)
+	delete(tm.relayPinned, nodeID)
 	delete(tm.peerPubKeys, nodeID)
 	tm.mu.Unlock()
 

pkg/daemon/tunnel_handle_test.go

@@ -710,3 +710,48 @@ func TestHandlersAreRobustAgainstEmptyData(_ *testing.T) {
 
 // Ensure compile-time coverage of stdlib imports unused in some builds
 var _ = fmt.Errorf
+
+
+// setupAuthKeyExchangeTest builds the common scaffolding: a tunnel
+// manager with encryption enabled, a peer Ed25519 identity registered,
+// and a valid signed auth key_exchange frame ready to feed into
+// handleAuthKeyExchange.
+func setupAuthKeyExchangeTest(t *testing.T) (*TunnelManager, uint32, []byte, *net.UDPAddr) {
+	t.Helper()
+	tm := NewTunnelManager()
+	if err := tm.EnableEncryption(); err != nil {
+		t.Fatalf("EnableEncryption: %v", err)
+	}
+	if err := tm.Listen("127.0.0.1:0"); err != nil {
+		t.Fatalf("Listen: %v", err)
+	}
+	peerPub, peerPriv, err := ed25519.GenerateKey(rand.Reader)
+	if err != nil {
+		t.Fatalf("ed25519 keygen: %v", err)
+	}
+	tm.SetPeerVerifyFunc(func(uint32) (ed25519.PublicKey, error) {
+		return peerPub, nil
+	})
+	curve := ecdh.X25519()
+	peerX, err := curve.GenerateKey(rand.Reader)
+	if err != nil {
+		t.Fatalf("x25519 keygen: %v", err)
+	}
+	peerNodeID := uint32(0x50515253)
+
+	challenge := make([]byte, 4+4+32)
+	copy(challenge[0:4], []byte("auth"))
+	binary.BigEndian.PutUint32(challenge[4:8], peerNodeID)
+	copy(challenge[8:40], peerX.PublicKey().Bytes())
+	sig := ed25519.Sign(peerPriv, challenge)
+
+	tm.AddPeer(peerNodeID, mustUDPAddr(t, "127.0.0.1:9997"))
+
+	data := make([]byte, 132)
+	binary.BigEndian.PutUint32(data[0:4], peerNodeID)
+	copy(data[4:36], peerX.PublicKey().Bytes())
+	copy(data[36:68], peerPub)
+	copy(data[68:132], sig)
+	from := mustUDPAddr(t, "127.0.0.1:1234")
+	return tm, peerNodeID, data, from
+}