fix(connections): survive transient map-endpoint timeouts (no more permanent connection death) by HandyS11 · Pull Request #63 · HandyS11/RustPlusBot

HandyS11 · 2026-07-23T16:31:17Z

Problem

The bot would connect fine, then ~20s later log Receive loop exited and go permanently silent — no reconnect, no error. Reproduced live and traced to root cause.

It is not auth/credential (the server answered GetInfo with real player counts) and not related to the #player-events work (the Connections feature is identical between develop and that branch).

Root cause: when the server's map endpoint is intermittently slow, the connect-time GetMap downloads time out. A per-request timeout surfaces as OperationCanceledException even though nothing was cancelled — and GetRigPositionsAsync rethrew it unconditionally, where RunAsync mistook it for a shutdown signal and exited the per-server connection loop forever. Compounding it, the RustPlusApi library raises no event on a server-initiated socket close, so drops went unnoticed for up to a minute.

Changes

fix — a transient request timeout (OperationCanceledException with ct not cancelled) now degrades that one window instead of killing the loop. Guarded the catches in GetRigPositionsAsync, PollMarkersAsync, PollReachabilityAsync, and added the missing .WaitAsync timeout guard to GetClanInfoAsync.
feat — a liveness watchdog (IRustServerConnection.IsConnected + WatchLivenessAsync) races the heartbeat and reconnects within LivenessPollInterval (5s) instead of up to HeartbeatInterval (60s), since the library gives no drop event.
perf — moved the two heavy full-map downloads (GetMapDimensions + GetRigPositions) off the critical connect path into the start of PollMarkersAsync. The connection now goes live in ~1s; a slow map only disables rig detection for that window.
style — member-ordering nit from the pre-push formatter.

Verification

Full solution suite: 1242 passed, 1 skipped (19 projects). 3 new/updated connection tests, incl. Connect_MonumentsTimeout_DoesNotKillLoop_AndStillReconnects and ServerClosesSocket_LivenessWatchdog_DrivesReconnect.
Live run against a genuinely degraded map endpoint: clan probe now lands ~1s after connect (was blocked ~15s behind the map fetch); the monuments + marker-poll timeouts degrade gracefully in the background and the connection stays fully alive — the exact condition that previously killed the bot.

Out of scope

Noticed a pre-existing shutdown race in the Workspace feature (WorkspaceReconciler/WorkspaceHostedService startup self-heal throws TaskCanceledException if the process stops mid-heal) — unrelated to these changes.

🤖 Generated with Claude Code

Copilot

Pull request overview

Improves the Connections feature’s resilience by preventing transient Rust+ API timeouts from permanently killing the per-server connection loop, and by detecting server-initiated socket closes promptly via a liveness watchdog (since the upstream library doesn’t emit a close event).

Changes:

Treat per-request OperationCanceledException (when the outer connection token isn’t cancelled) as a degradable timeout in several polling/probe paths, rather than a fatal shutdown signal.
Add IRustServerConnection.IsConnected and a liveness watchdog loop that races the heartbeat to trigger faster reconnects on silent socket drops.
Move heavy full-map fetches off the critical connect path into the marker polling task; expand tests and fakes to cover the new scenarios.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/RustPlusBot.Features.Connections.Tests/Fakes/FakeRustSocketSource.cs	Extends the fake connection/source to simulate per-request monuments timeouts and server-initiated socket drops (`IsConnected`).
tests/RustPlusBot.Features.Connections.Tests/ConnectionSupervisorTests.cs	Adds regression tests for monuments timeout survival and liveness watchdog-driven reconnect.
src/RustPlusBot.Features.Connections/Supervisor/ConnectionSupervisor.cs	Adds liveness watchdog, adjusts cancellation/timeout handling, and moves map/rig fetches into marker polling.
src/RustPlusBot.Features.Connections/Listening/RustPlusSocketSource.cs	Implements `IsConnected` and adds `WaitAsync` timeout guard to `GetClanInfoAsync`.
src/RustPlusBot.Features.Connections/Listening/IRustServerConnection.cs	Introduces `IsConnected` to support watchdog-driven drop detection.
src/RustPlusBot.Features.Connections/ConnectionOptions.cs	Adds configurable `LivenessPollInterval` (default 5s).
docs/superpowers/specs/2026-07-22-player-events-channel-design.md	Adds design documentation for a separate “player-events channel” feature.
docs/superpowers/plans/2026-07-22-player-events-channel.md	Adds a detailed implementation plan for the “player-events channel” feature.

Comments suppressed due to low confidence (1)

tests/RustPlusBot.Features.Connections.Tests/Fakes/FakeRustSocketSource.cs:134

TimeoutOnMonumentsOnce() XML doc still calls this a "connect-time" monuments fetch, but monuments/rig fetching now happens in the marker poll background task. Adjusting the doc avoids confusion when reading the tests alongside the supervisor’s current behavior.

    /// <summary>
    /// Makes the NEXT connection's connect-time <see cref="FakeConnection.GetMonumentsAsync"/> throw
    /// <see cref="OperationCanceledException"/>, simulating a per-request timeout (the outer connection
    /// token is NOT cancelled). Applies to the next connection only. Call before
    /// <see cref="EnsureConnectionAsync"/>.
    /// </summary>
    public void TimeoutOnMonumentsOnce() => _pendingMonumentsTimeout = true;

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        while (!ct.IsCancellationRequested)
+        {
+            await Task.Delay(_options.LivenessPollInterval, ct).ConfigureAwait(false);
+            if (!connection.IsConnected)
+            {
+                LogSocketDropped(logger, key.Server);
+                return ReconnectReason.Unreachable;
+            }
+        }
+
+        return ReconnectReason.Stopped;
+    }


+        // Transfer any pre-staged monuments timeout so the connect-time rig fetch throws for the NEXT
+        // connection only (mimicking a real per-request timeout, which surfaces as OperationCanceledException).
+        connection.MonumentsTimeout = _pendingMonumentsTimeout;
+        _pendingMonumentsTimeout = false;


+# Player Events Channel — Design
+
+**Date:** 2026-07-22
+**Status:** Approved
+
+## Problem
+
+Every live notification lands in the same per-server `#events` channel: map events (cargo
+ship, patrol helicopter, chinook, oil rigs), wipe announcements, and team-player presence
+transitions (login, logout, death, AFK). The player noise drowns the map events.
+
+## Goal
+
+Route team-player presence transitions to a new per-server `#player-events` channel. Map
+events and wipe announcements stay in `#events`. In-game team-chat messaging is unchanged
+for both feeds.


…onnection loop A per-request timeout during a connected window surfaces as an OperationCanceledException even when the connection token is NOT cancelled. GetRigPositionsAsync rethrew it unconditionally and RunAsync treats any OCE as a shutdown signal, so a transient slow map endpoint permanently terminated the per-server connection loop with no reconnect (bot connected, dropped ~20s later, then went silent). Confirmed via a live repro trace. Guard the timeout catches with `when (ct.IsCancellationRequested)` in GetRigPositionsAsync, PollMarkersAsync and PollReachabilityAsync so a request timeout degrades that one window instead of tearing down the connection, and add the missing `.WaitAsync` timeout guard to GetClanInfoAsync (the one connect-time probe that could block indefinitely on a dead socket). Regression test: Connect_MonumentsTimeout_DoesNotKillLoop_AndStillReconnects. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ss watchdog The Rust+ library raises no event when the server closes the socket, so a silent drop previously went unnoticed until the next heartbeat (up to HeartbeatInterval, default 60s) — a long window of a dead-but-"connected" socket. Add IsConnected to IRustServerConnection (backed by RustPlusSocket.IsConnected) and a WatchLivenessAsync watchdog that races the heartbeat loop and returns Unreachable as soon as the socket is no longer open, bounded by the new ConnectionOptions.LivenessPollInterval (default 5s). Test: ServerClosesSocket_LivenessWatchdog_DrivesReconnect (heartbeat keeps answering Ok, so only the watchdog can detect the drop). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…path GetMapDimensions and GetRigPositions (the two full-map downloads) ran on the connect path before the heartbeat/marker loops started, so a slow or degraded map endpoint delayed the whole connection going live — and its stalls were the trigger for the drop that used to kill the loop. Fetch dims + rig positions at the start of PollMarkersAsync instead. The connection now goes live immediately (heartbeat + chat relay), and marker/rig detection activates once the map resolves; a map timeout just disables rig detection for that window (dims -> null, rigs -> empty) without blocking anything. Verified live against a degraded map endpoint: clan probe now lands ~1s after connect (was blocked ~15s behind the map fetch), and the monuments/marker timeouts degrade in the background with the connection staying fully alive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The pre-push formatter relocates the IsConnected member added with the liveness watchdog (property after the constructor / in declared order). No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- WatchLivenessAsync now catches the cancellation OperationCanceledException and returns ReconnectReason.Stopped, so on shutdown the WhenAny winner is a clean reason instead of a faulted task (its doc already advertised Stopped). - Reword the fake's monuments-timeout helpers/test: the rig/monuments fetch runs in the background marker poll now, not on the connect path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

HandyS11 · 2026-07-23T16:49:41Z

Thanks @copilot — addressed all three:

WatchLivenessAsync never reaching Stopped — fixed. It now wraps the poll loop in try { … } catch (OperationCanceledException) { } and returns ReconnectReason.Stopped, so on shutdown the Task.WhenAny winner is a clean reason instead of a faulted task. (9846511)
Stale "connect-time" wording in the fake/test — fixed. TimeoutOnMonumentsOnce, the Create transfer comment, and the regression test now say the monuments/rig fetch runs in the background marker poll, matching where it moved to. (9846511)
Unrelated player-events docs in this PR — fixed. Those two docs/superpowers/ commits were on a local develop ahead of origin/develop; I rebased the connection commits directly onto origin/develop so the PR is now only the 6 connection files. The docs stay with the player-events work.

Full suite still green (1242 passed / 1 skipped).

Copilot · 2026-07-23T16:53:08Z

Thanks @copilot — addressed all three:

WatchLivenessAsync never reaching Stopped — fixed. It now wraps the poll loop in try { … } catch (OperationCanceledException) { } and returns...

Confirmed — all three fixes from 9846511 are in place and look correct:

WatchLivenessAsync wraps the poll loop in try/catch (OperationCanceledException) and returns ReconnectReason.Stopped, so Task.WhenAny always sees a clean result on shutdown.
Fake/test wording updated to "marker poll's rig fetch" in both the Create transfer comment and the TimeoutOnMonumentsOnce doc.
Unrelated docs not present in this branch.

Local run: 114 passed, 0 failed in RustPlusBot.Features.Connections.Tests.

Copilot AI review requested due to automatic review settings July 23, 2026 16:31

Copilot started reviewing on behalf of HandyS11 July 23, 2026 16:31 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

HandyS11 and others added 5 commits July 23, 2026 18:48

HandyS11 force-pushed the fix/connection-transient-timeout-resilience branch from 83c6c14 to 9846511 Compare July 23, 2026 16:49

Copilot started work on behalf of HandyS11 July 23, 2026 16:50 View session

Copilot finished work on behalf of HandyS11 July 23, 2026 16:53

HandyS11 merged commit 2ffb18f into develop Jul 23, 2026
4 checks passed

HandyS11 deleted the fix/connection-transient-timeout-resilience branch July 23, 2026 21:20

HandyS11 mentioned this pull request Jul 24, 2026

fix: stop one transient failure from permanently killing an event consumer #64

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(connections): survive transient map-endpoint timeouts (no more permanent connection death) - #63

fix(connections): survive transient map-endpoint timeouts (no more permanent connection death)#63
HandyS11 merged 5 commits into
developfrom
fix/connection-transient-timeout-resilience

HandyS11 commented Jul 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

HandyS11 commented Jul 23, 2026

Uh oh!

Copilot AI commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

HandyS11 commented Jul 23, 2026

Problem

Changes

Verification

Out of scope

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

HandyS11 commented Jul 23, 2026

Uh oh!

Copilot AI commented Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants