Skip to content

Commit 327c542

Browse files
committed
release(v1.10.4): auto-recover compat-mode WSS connection
In v1.10.2/v1.10.3, when the daemon's WSS connection to the beacon died (network blip, nginx restart, peer hangup), the transport stayed permanently dead — every subsequent Send returned "wss send: failed to write msg: use of closed network connection" and a manual daemon restart was the only recovery. Observed in production twice in one hour today. New design: a supervisor goroutine owns both the read loop and the reconnect lifecycle. 1. drainReads(conn) — block on conn.Read with lifetimeCtx 2. on read error: tear down conn, sleep with exp backoff (250 ms → 30 s cap), poll closed flag at 100 ms 3. dialAndAuth a fresh conn (parented on lifetimeCtx so Close interrupts an in-flight dial) 4. install + resume read loop Send returns the new wss.ErrReconnecting (instead of a raw conn write error) when no live conn is installed — caller's higher-level retransmit (key-exchange retx, dial-retry, writeFrame retry) refires naturally on the next conn. Transient read errors during the reconnect window are NOT surfaced to Recv(), so the daemon's tunnel readLoop doesn't tear itself down over a temporary blip. Tests: - TestReconnect_AfterServerCloseRestoresSendAndRecv: fakeBeacon CloseNow's the conn after the second incoming frame; supervisor reconnects; a probe-loop confirms ErrReconnecting during the gap and a successful echo on the freshly-established conn - TestServer_LastWriterWinsOnReconnect updated to not poll Recv (the test's previous "tr1.Recv returns ErrClosed after kick" assertion is incompatible with auto-reconnect, which is the intended new behavior; the server-side IsConnected check still validates last-writer-wins)
1 parent 822e12d commit 327c542

5 files changed

Lines changed: 396 additions & 54 deletions

File tree

CHANGELOG.md

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,42 @@ project uses [Semantic Versioning](https://semver.org/).
77
Detailed per-release notes are on the
88
[GitHub Releases page](https://github.com/TeoSlayer/pilotprotocol/releases).
99

10+
## [1.10.4] - 2026-05-19
11+
12+
### Fixed
13+
- **Compat-mode WSS connection auto-recovers from drops.** Before this
14+
fix, when the daemon's WSS connection to the beacon died (network
15+
blip, nginx restart, GFW-style spoofed RST, peer-side hangup), the
16+
transport stayed permanently dead — every subsequent `Send` returned
17+
`wss send: failed to write msg: use of closed network connection`
18+
and a manual daemon restart was the only recovery. Observed in
19+
production today (v1.10.3) twice in one hour.
20+
21+
The new design introduces a supervisor goroutine that owns both the
22+
read loop AND the reconnect lifecycle. When the underlying conn
23+
fails, the supervisor:
24+
1. tears down the dead conn
25+
2. waits with exponential backoff (250 ms → 30 s cap)
26+
3. re-dials + re-auths against the same `-compat-beacon` URL
27+
4. installs the fresh conn and resumes reading
28+
All under `lifetimeCtx`, so `Close()` interrupts an in-flight dial
29+
or read within ~100 ms.
30+
31+
`Send` now returns the new `wss.ErrReconnecting` (instead of a raw
32+
conn-write error) when no live conn is installed — caller's
33+
higher-level retransmit (key-exchange retx, dial-retry, write-frame
34+
retry) refires naturally on the next conn. Transient read errors
35+
during the gap are no longer surfaced to `Recv()` so the daemon's
36+
tunnel read loop doesn't tear itself down over a temporary blip.
37+
38+
### Tests
39+
- `pkg/daemon/transport/wss/zz_wss_test.go::TestReconnect_AfterServerCloseRestoresSendAndRecv`
40+
`fakeBeacon` is configured to `CloseNow()` the WS conn after the
41+
second incoming frame. The test then drives a polling probe loop
42+
and asserts the supervisor (a) flags ErrReconnecting during the
43+
reconnect window and (b) successfully echoes a third frame on the
44+
newly-established conn.
45+
1046
## [1.10.3] - 2026-05-19
1147

1248
### Added

pkg/beacon/wss/zz_server_test.go

Lines changed: 8 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,6 @@ import (
99
"crypto/x509"
1010
"encoding/base64"
1111
"encoding/json"
12-
"errors"
1312
"fmt"
1413
"net/http"
1514
"net/http/httptest"
@@ -22,7 +21,6 @@ import (
2221

2322
"github.com/TeoSlayer/pilotprotocol/internal/crypto"
2423
"github.com/TeoSlayer/pilotprotocol/pkg/beacon/wss"
25-
"github.com/TeoSlayer/pilotprotocol/pkg/daemon/transport"
2624
dwss "github.com/TeoSlayer/pilotprotocol/pkg/daemon/transport/wss"
2725
)
2826

@@ -303,22 +301,19 @@ func TestServer_LastWriterWinsOnReconnect(t *testing.T) {
303301
tr1 := mustDialDaemon(t, wsURL, id, nodeID)
304302
waitForCondition(500*time.Millisecond, func() bool { return s.IsConnected(nodeID) })
305303

304+
// Close tr1 BEFORE the second dial so its auto-reconnect
305+
// supervisor can't race tr2 (the daemon transport now
306+
// transparently re-dials any time the server kicks it). The
307+
// invariant we care about for this test is server-side:
308+
// "the latest writer wins, but the nodeID stays connected".
309+
_ = tr1.Close()
310+
306311
// Second connection from the same identity.
307312
tr2 := mustDialDaemon(t, wsURL, id, nodeID)
308313
defer tr2.Close()
309314

310-
// First connection should drop. Recv on tr1 should return
311-
// ErrClosed shortly.
312-
deadline := time.Now().Add(1 * time.Second)
313-
for time.Now().Before(deadline) {
314-
_, _, err := tr1.Recv()
315-
if errors.Is(err, transport.ErrClosed) || err != nil {
316-
break
317-
}
318-
}
319-
_ = tr1.Close()
320-
321315
// Server still reports nodeID as connected (via tr2).
316+
waitForCondition(500*time.Millisecond, func() bool { return s.IsConnected(nodeID) })
322317
if !s.IsConnected(nodeID) {
323318
t.Error("nodeID not connected after reconnect")
324319
}

pkg/daemon/daemon.go

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -82,9 +82,9 @@ type Config struct {
8282
// daemons pointing at registry.pilotprotocol.network:443 with its
8383
// Let's Encrypt certificate. Mirrors -tls-trust on the WSS side.
8484
RegistryTrust string
85-
IdentityPath string // path to persist Ed25519 identity (empty = no persistence)
86-
Email string // email address for account identification and key recovery
87-
Owner string // deprecated: use Email instead
85+
IdentityPath string // path to persist Ed25519 identity (empty = no persistence)
86+
Email string // email address for account identification and key recovery
87+
Owner string // deprecated: use Email instead
8888

8989
Endpoint string // fixed public endpoint (host:port) — skips STUN discovery (for cloud VMs)
9090
Public bool // make this node's endpoint publicly discoverable

0 commit comments

Comments
 (0)