fix(sessions): recover local agent turns interrupted by system sleep#3151
fix(sessions): recover local agent turns interrupted by system sleep#3151robbie-c wants to merge 2 commits into
Conversation
System sleep mid-turn killed the agent's upstream connection and left the session looking connected while making no progress, and restart resume could fail with a generic message. Three fixes: 1. Classify network/socket failures (ECONNRESET, fetch failed, socket hang up, ...) as recoverable via isTransportError, so sendLocalPrompt triggers auto-recovery instead of silently dropping the turn. 2. Detect wedged turns after wake: the agent service watches sessions with a pending prompt after powerMonitor resume and emits session-stalled for any that produce no ACP traffic within a 45s grace period; the session service reconnects them in place. 3. Harden restart resume: rethrow the real reconnect failure instead of returning null (so the UI shows the actual reason), and report when a reconnect fell back to a fresh session so the user is told the agent lost its context instead of it happening silently. Also hardens startAutoRecoverLocalSession against a throwing reconnectInPlace, which previously escaped as an unhandled rejection. Generated-By: PostHog Code Task-Id: 6b1cbbfe-903b-400d-a76f-45e614c6d0f8
|
Merging to
After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here |
|
Reviews (1): Last reviewed commit: "fix(sessions): recover local agent turns..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| isFatalSessionError(errorMessage, errorDetails) || | ||
| isTransportError(errorMessage, errorDetails) | ||
| ) { | ||
| this.d.log.error("Unrecoverable prompt error, attempting recovery", { |
There was a problem hiding this comment.
startAutoRecoverLocalSession, which reconnects the session. Logging them as "Unrecoverable" will mislead anyone reading the logs while diagnosing a sleep-recovery incident. Consider a message that reflects both error classes accurately.
| this.d.log.error("Unrecoverable prompt error, attempting recovery", { | |
| this.d.log.error("Prompt error, attempting session recovery", { |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
| }); | ||
|
|
||
| describe("stalled turn detection after resume", () => { | ||
| const STALLED_TURN_GRACE_MS = 45 * 1000; |
There was a problem hiding this comment.
AgentService.STALLED_TURN_GRACE_MS, this test will silently become wrong. Accessing the private static via a cast is the idiomatic pattern already used elsewhere in the file for other private members.
| const STALLED_TURN_GRACE_MS = 45 * 1000; | |
| const STALLED_TURN_GRACE_MS = ( | |
| AgentService as unknown as { STALLED_TURN_GRACE_MS: number } | |
| ).STALLED_TURN_GRACE_MS; |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
…test
Reword the recovery log line ("Unrecoverable" contradicted the recovery
it triggers) and read the stalled-turn grace period from AgentService in
the test instead of duplicating the literal.
Generated-By: PostHog Code
Task-Id: 6b1cbbfe-903b-400d-a76f-45e614c6d0f8
1 participant
Problem
When the machine sleeps mid-turn (e.g. lid closed on battery), the local agent's upstream connection dies. The user-visible symptom was an agent that stopped making progress after sleep and couldn't resume on startup. Tracing the code (not reproduced step-by-step) points at three gaps:
ECONNRESET/fetch failed/ hung socket) isn't inFATAL_SESSION_ERROR_PATTERNS, sosendLocalPromptsilently drops the turn — the session shows as connected but the agent makes no progress.powerMonitorresume checks for this.Changes
1. Classify transport failures as recoverable (
packages/shared/src/errors.ts)New
isTransportError()covering network/socket failure patterns;sendLocalPromptnow triggers the existing auto-recovery (startAutoRecoverLocalSession→reconnectInPlacewith backoff) for these instead of silently clearing the pending state.2. Detect turns wedged by sleep (
agent.ts,agent.router.ts,sessionService.ts)lastActivityAt.powerMonitorresume, sessions with a pending prompt get a 45s grace period; any that produce no agent traffic in that window emit a newsession-stalledevent.SessionServicesubscribes (agent.onSessionStalled, optional in theSessionTrpcinterface) and reconnects the session in place. History is preserved via the normal JSONL resume.3. Harden restart resume (
agent.ts,sessionService.ts)null, so the error screen shows the actual cause rather than the generic message.SessionResponsegainsresumedExistingSession; when a reconnect that expected to resume falls back to a fresh session, the user gets an info toast that the agent lost its prior context (the transcript is unaffected) instead of it happening silently.Also hardens
startAutoRecoverLocalSessionagainst a throwingreconnectInPlace(previously an unhandled rejection) and fixes a "Connecting to to" typo in the recovery-failed message.Testing
isTransportErrorclassification (shared), stalled-turn detection with fake timers (workspace-server), stalled-event → in-place reconnect routing incl. cloud/superseded-run guards (core).pnpm --filtertypecheck green for shared, core, workspace-server, host-router, ui, code; Biome clean; core/workspace-server/shared test suites pass (pre-existingbetter-sqlite3NODE_MODULE_VERSION failures in db tests are unrelated).Companion PR: #3148 (UX caveat that keep-awake cannot survive a closed laptop lid).
Created with PostHog Code