Skip to content

Commit cfe25d6

Browse files
committed
Changelog: v1.9.1 stable
Rolls up rc1–rc5 into the GA release entry. Headline: rekey-storm fix (grace period + AEAD-divergence recovery), cold pilotctl latency collapse (10–30s → ~600ms), and relay-flag pinning to stop direct/ relay flapping.
1 parent 2fe1e1c commit cfe25d6

1 file changed

Lines changed: 73 additions & 0 deletions

File tree

CHANGELOG.md

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,79 @@ Detailed per-release notes for tagged versions are published on the
99

1010
## [Unreleased]
1111

12+
## [1.9.1] - 2026-05-05
13+
14+
Stable release rolling up rc1–rc5. Headline themes: cold-start latency
15+
collapse, recovery from peer-side AEAD key divergence, and relay-flag
16+
stability. Verified end-to-end against the production rendezvous fleet.
17+
18+
### Fixed
19+
20+
- **Rekey storm with grace-period guard.** Resolved the loop where a
21+
peer-initiated rekey left in-flight ciphertext on the relay path
22+
encrypted with the old key; those frames arrived after the new key
23+
installed, failed AEAD, and tripped the decrypt-fail drop threshold,
24+
tearing down the freshly-installed `peerCrypto` and demanding yet
25+
another rekey. `decryptFailDropGrace = 3 * time.Second` never drops
26+
a `peerCrypto` installed less than the grace window ago. (rc4 + rc5)
27+
- **Recovery from peer-side AEAD key divergence.** After 5 consecutive
28+
AEAD authentication failures the daemon drops `tm.crypto[peer]` and
29+
triggers a fresh key exchange — automatic per-peer recovery
30+
equivalent to a daemon restart. New `TunnelManager.DropCrypto(nodeID)`
31+
exposes this as a public method for operator scripts. (rc4)
32+
- **Relay-flag pinning.** `relayPinned` map tracks "relay flag set by
33+
an authoritative signal" (registry, dial-loop direct exhaustion,
34+
ICMP-unreachable, beacon-sourced key exchange). The previous
35+
"auto-clear after 3 direct receipts" heuristic flapped relay state
36+
every ~60s in production, causing session-key rotation cascades and
37+
the "ping works for a minute then dial timeout for minutes" symptom.
38+
Auto-clear is now disabled. (rc4)
39+
40+
### Performance
41+
42+
- **Cold pilotctl latency cut from 10–30 s to ~600 ms; warm to ~170
43+
ms.** Combination of:
44+
- Daemon-side hostname cache (60 s, persisted to
45+
`~/.pilot/hostname_cache.json`, reloaded at startup).
46+
- PolicyRunner `fetchMembers` exponential backoff so a doomed
47+
`list_nodes` no longer holds the regConn mutex every 5 s.
48+
- `DialInitialRTO` 1 s → 250 ms — three direct retries now total
49+
~1.75 s before relay flip (was ~7 s).
50+
- Background pre-warm of the registry resolve cache for trusted
51+
peers at daemon start. (rc2)
52+
53+
### Changed
54+
55+
- **Dial budget**: `DialMaxRetries` 6 → 7. The relay phase now gets 4
56+
retries (was 3), giving cold-start handshakes (key exchange + SYN/
57+
SYN-ACK) enough room without exceeding the user's `--timeout`. With
58+
`DialInitialRTO=250ms` and `DialMaxRTO=8s` exponential backoff,
59+
total budget is ~8 s. (rc4)
60+
- **`pilotctl ping` per-attempt floor**: 4 s → 10 s, covering the
61+
daemon startup-storm warm-up window where multiple trusted peers
62+
concurrently establish crypto state. (rc4)
63+
- **Beacon `--advertise-addr` flag**: MIG instances behind a public
64+
DNAT can now advertise a stable address. (rc2)
65+
66+
### Known limitations
67+
68+
- Against peers running pre-1.9.1 daemons, the rekey storm pattern
69+
may still cause occasional dial timeouts until those peers update.
70+
Workaround on our side: `pilotctl daemon stop && pilotctl daemon
71+
start`.
72+
- Cold start: the FIRST 1–3 dials immediately after a daemon restart
73+
can fail while the tunnel establishes crypto. Subsequent dials
74+
succeed. Tracked for a future per-peer probe-and-adapt SRTT-based
75+
dial budget.
76+
77+
### Considered, reverted in development
78+
79+
A rough log of fixes that introduced regressions and were pulled
80+
before GA: inbound auth-key-exchange rate-limit, recv-replay-window
81+
reset on stale auth-key-exchange duplicate, trusted-peer tunnel
82+
pre-warm at daemon start, and dial-timeout-exhausted crypto drop. See
83+
the rc4 entry for details.
84+
1285
## [1.9.1-rc5] - 2026-05-04
1386

1487
Reliability fix on top of rc4. Verified 100% (30/30 cold-start sends +

0 commit comments

Comments
 (0)