fix(sdks): raise typed, actionable errors when sandbox dies mid-request (#1419)

## Problem

When a sandbox is killed (or reaches its end of life) while a request is
in flight, both SDKs surfaced unusable errors:

- **JS**: `SandboxError: 2: [unknown] terminated` — typed, but cryptic
and says nothing about the sandbox being killed.
- **Python**: leaked a completely raw `httpcore.RemoteProtocolError:
<StreamReset stream_id:1, error_code:2, remote_reset:True>`.

This affected the whole envd streaming family (`commands.run`, PTY
sessions, `files.watchDir`/`watch_dir`) and the `files.read`/`write`
HTTP transfers.

The stream-reset signature alone can't distinguish the sandbox dying
from an intermediary (load balancer, network) dropping the connection —
so the SDKs now actively check, and only transform the error when the
sandbox is confirmed gone.

## Fix

**Health-check disambiguation.** When the connection-terminated
signature appears (JS: `ConnectError` `Code.Unknown` + `terminated` or
Undici `TypeError: terminated`; Python: `httpcore`/`httpx`
`RemoteProtocolError`), the SDK probes envd's `/health` endpoint:

- **502 (sandbox confirmed gone)** → `TimeoutError` (JS) /
`TimeoutException` (Python): "The sandbox was killed or reached its end
of life while the request was in flight." This matches how requests to
an *already-dead* sandbox surface today (the 502 / `Code.Unavailable`
mappings raise the timeout error type), so the exception type no longer
depends on whether the sandbox died just before or just during the
request.
- **Anything else** (still running, or probe inconclusive) → the
original error propagates unchanged, exactly as before this PR.

The probe (5s timeout) runs only on the termination signature, never on
the happy path or for other errors. A health-check closure is plumbed
into `Commands`/`Pty`/`Filesystem` and the command/watch handles in JS
and sync/async Python; `Commands`/`Pty` now receive the envd API client
in their constructors (internal signature change).

**Cleanup** (`e2b_connect/client.py`): removed the
`@_retry(RemoteProtocolError, 3)` decorators from
`call_server_stream`/`acall_server_stream`. They never executed —
`inspect.iscoroutinefunction` is false for (async) generator functions,
and calling a generator function doesn't run its body, so the wrapper's
`try/except` could never fire. A *working* mid-stream retry would be
wrong anyway (it would replay already-delivered events). Unary retries
are unchanged.

## Before / after

```ts
const sandbox = await Sandbox.create()
const cmd = await sandbox.commands.run('sleep 60', { background: true })
await sandbox.kill() // e.g. from another process
await cmd.wait()
// before: SandboxError: 2: [unknown] terminated
// after:  TimeoutError: [unknown] terminated: The sandbox was killed or reached
//         its end of life while the request was in flight.
```

```python
sandbox = Sandbox.create()
cmd = sandbox.commands.run("sleep 60", background=True)
sandbox.kill()
cmd.wait()
# before: httpcore.RemoteProtocolError: <StreamReset stream_id:1, error_code:2, remote_reset:True>  (not an e2b type!)
# after:  e2b.exceptions.TimeoutException: <StreamReset ...>: The sandbox was killed
#         or reached its end of life while the request was in flight.
```

If the health probe does not confirm the sandbox is gone (e.g. a load
balancer dropped the connection, or local envd in debug mode), the
original error propagates unchanged — the SDK only makes a claim when it
has verified it.

## Notes

- Not covered: errors raised while consuming a `format: 'stream'` body
**after** `files.read` returns (JS `ReadableStream` consumption happens
in user code). Python is fully covered since httpx buffers non-streaming
responses inside the request call.

## Tests

- Unit: confirmed-kill → `TimeoutError`/`TimeoutException`, raw-error
passthrough for running/unknown/probe-failure, health check skipped for
unrelated errors — 28 Python + 25 JS assertions pass.
- Integration (run against live sandboxes, all passing): start `sleep
60`, kill the sandbox, assert `wait()` raises
`TimeoutError`/`TimeoutException` with the *confirmed* kill message —
JS, sync Python, and async Python.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: mishushakov

.changeset/spotty-llamas-shout.md

@@ -0,0 +1,6 @@
+---
+'e2b': patch
+'@e2b/python-sdk': patch
+---
+
+Raise a typed, actionable error when the sandbox dies while a request is in flight. When the connection is dropped mid-request (streaming RPC calls — commands, PTY, directory watch — and filesystem read/write), the SDKs now probe the sandbox health endpoint: if the sandbox is confirmed gone, a `TimeoutError` (JS) / `TimeoutException` (Python) is raised stating the sandbox was killed or reached its end of life — consistent with how requests to an already-dead sandbox surface. In all other cases the original error propagates unchanged.

packages/js-sdk/src/envd/api.ts

@@ -12,10 +12,12 @@ import {
   formatSandboxTimeoutError,
   AuthenticationError,
   RateLimitError,
+  TimeoutError,
 } from '../errors'
 import { StartResponse, ConnectResponse } from './process/process_pb'
 import { Code, ConnectError } from '@connectrpc/connect'
 import { WatchDirResponse } from './filesystem/filesystem_pb'
+import { SandboxHealthCheck } from './rpc'
 
 type ApiError = { message?: string } | string
 
@@ -29,6 +31,64 @@ const DEFAULT_ERROR_MAP: Record<number, (message: string) => Error> = {
   507: (message) => new NotEnoughSpaceError(message),
 }
 
+const HEALTH_CHECK_TIMEOUT_MS = 5_000
+
+/**
+ * Probes the sandbox's envd health endpoint.
+ *
+ * @param envdApi - The envd API client of the sandbox.
+ * @returns `true` if the sandbox is running, `false` if it is not, `undefined` if its state could not be determined.
+ */
+export async function checkSandboxHealth(
+  envdApi: EnvdApiClient
+): Promise<boolean | undefined> {
+  try {
+    const res = await envdApi.api.GET('/health', {
+      signal: AbortSignal.timeout(HEALTH_CHECK_TIMEOUT_MS),
+    })
+
+    if (res.response.status === 502) {
+      return false
+    }
+    if (res.response.ok) {
+      return true
+    }
+
+    return undefined
+  } catch {
+    return undefined
+  }
+}
+
+/**
+ * Handles transport-level fetch failures from envd API calls. When the connection was
+ * dropped mid-request, probes the sandbox health to tell apart the sandbox being killed
+ * from a transient network failure (e.g. a load balancer dropping the connection).
+ *
+ * @param err - The caught error, expected to be a fetch transport failure.
+ * @param checkHealth - Probe returning whether the sandbox is running, or `undefined` when unknown.
+ * @returns A `TimeoutError` when the connection was terminated mid-request and the sandbox is confirmed gone, or the original error otherwise.
+ */
+export async function handleEnvdApiFetchError(
+  err: unknown,
+  checkHealth?: SandboxHealthCheck
+): Promise<Error> {
+  // Undici surfaces a connection dropped mid-body as a TypeError with the message 'terminated'
+  if (err instanceof TypeError && err.message === 'terminated') {
+    const running = checkHealth
+      ? await checkHealth().catch(() => undefined)
+      : undefined
+
+    if (running === false) {
+      return new TimeoutError(
+        `${err.message}: The sandbox was killed or reached its end of life while the request was in flight.`
+      )
+    }
+  }
+
+  return err as Error
+}
+
 /**
  * Handles errors from envd API responses by mapping HTTP status codes to specific error types.
  *

packages/js-sdk/src/envd/rpc.ts

@@ -14,6 +14,25 @@ import {
 } from '../errors'
 import { ENVD_DEFAULT_USER } from './versions'
 
+/**
+ * Result of a sandbox health probe: `true` if the sandbox is running, `false` if it is not,
+ * `undefined` if its state could not be determined.
+ */
+export type SandboxHealthCheck = () => Promise<boolean | undefined>
+
+/**
+ * Checks whether the error is the signature of the connection to the sandbox being
+ * dropped mid-request — an HTTP/2 stream reset surfaced by connect as `Code.Unknown`
+ * with the message 'terminated'.
+ */
+export function isConnectionTerminatedError(err: unknown): boolean {
+  return (
+    err instanceof ConnectError &&
+    err.code === Code.Unknown &&
+    err.rawMessage === 'terminated'
+  )
+}
+
 const DEFAULT_ERROR_MAP: Partial<Record<Code, (message: string) => Error>> = {
   [Code.InvalidArgument]: (message) => new InvalidArgumentError(message),
   [Code.Unauthenticated]: (message) => new AuthenticationError(message),
@@ -62,6 +81,36 @@ export function handleRpcError(
   return err as Error
 }
 
+/**
+ * Like {@link handleRpcError}, but when the connection to the sandbox was dropped
+ * mid-request it probes the sandbox health to tell apart the sandbox being killed
+ * from a transient network failure (e.g. a load balancer dropping the connection).
+ * When the probe confirms the sandbox is gone, a `TimeoutError` is returned —
+ * consistent with how requests to an already-dead sandbox surface.
+ *
+ * @param err - The caught error, expected to be a `ConnectError` from the gRPC transport.
+ * @param checkHealth - Probe returning whether the sandbox is running, or `undefined` when unknown.
+ * @param errorMap - Optional map of gRPC `Code` values to error factory functions that override the defaults.
+ * @returns The corresponding `Error` instance.
+ */
+export async function handleRpcErrorWithHealthCheck(
+  err: unknown,
+  checkHealth?: SandboxHealthCheck,
+  errorMap?: Partial<Record<Code, (message: string) => Error>>
+): Promise<Error> {
+  if (isConnectionTerminatedError(err) && checkHealth) {
+    const running = await checkHealth().catch(() => undefined)
+
+    if (running === false) {
+      return new TimeoutError(
+        `${(err as ConnectError).message}: The sandbox was killed or reached its end of life while the request was in flight.`
+      )
+    }
+  }
+
+  return handleRpcError(err, errorMap)
+}
+
 function encode64(value: string): string {
   switch (runtime) {
     case 'deno':

packages/js-sdk/src/sandbox/commands/commandHandle.ts

@@ -1,4 +1,7 @@
-import { handleRpcError } from '../../envd/rpc'
+import {
+  handleRpcErrorWithHealthCheck,
+  SandboxHealthCheck,
+} from '../../envd/rpc'
 import { SandboxError } from '../../errors'
 import { ConnectResponse, StartResponse } from '../../envd/process/process_pb'
 import type { CommandRequestOpts } from '.'
@@ -112,7 +115,8 @@ export class CommandHandle
     ) => Promise<void>,
     private readonly handleCloseStdin?: (
       opts?: CommandRequestOpts
-    ) => Promise<void>
+    ) => Promise<void>,
+    private readonly checkHealth?: SandboxHealthCheck
   ) {
     this._wait = this.handleEvents()
   }
@@ -281,7 +285,10 @@ export class CommandHandle
         }
       }
     } catch (e) {
-      this.iterationError = handleRpcError(e)
+      this.iterationError = await handleRpcErrorWithHealthCheck(
+        e,
+        this.checkHealth
+      )
     } finally {
       this.handleDisconnect()
     }

packages/js-sdk/src/sandbox/commands/index.ts

@@ -15,12 +15,20 @@ import {
   setupRequestController,
   Username,
 } from '../../connectionConfig'
-import { handleProcessStartEvent } from '../../envd/api'
+import {
+  checkSandboxHealth,
+  EnvdApiClient,
+  handleProcessStartEvent,
+} from '../../envd/api'
 import {
   Process as ProcessService,
   Signal,
 } from '../../envd/process/process_pb'
-import { authenticationHeader, handleRpcError } from '../../envd/rpc'
+import {
+  authenticationHeader,
+  handleRpcErrorWithHealthCheck,
+  SandboxHealthCheck,
+} from '../../envd/rpc'
 import { ENVD_COMMANDS_STDIN, ENVD_ENVD_CLOSE } from '../../envd/versions'
 import { SandboxError } from '../../errors'
 import { CommandHandle, CommandResult } from './commandHandle'
@@ -129,16 +137,16 @@ export class Commands {
 
   private readonly defaultProcessConnectionTimeout = 60_000 // 60 seconds
   private readonly envdVersion: string
+  private readonly checkHealth: SandboxHealthCheck
 
   constructor(
     transport: Transport,
-    private readonly connectionConfig: ConnectionConfig,
-    metadata: {
-      version: string
-    }
+    private readonly envdApi: EnvdApiClient,
+    private readonly connectionConfig: ConnectionConfig
   ) {
     this.rpc = createClient(ProcessService, transport)
-    this.envdVersion = metadata.version
+    this.envdVersion = envdApi.version
+    this.checkHealth = () => checkSandboxHealth(this.envdApi)
   }
 
   /**
@@ -177,7 +185,7 @@ export class Commands {
         ...(p.config!.cwd && { cwd: p.config!.cwd }),
       }))
     } catch (err) {
-      throw handleRpcError(err)
+      throw await handleRpcErrorWithHealthCheck(err, this.checkHealth)
     }
   }
 
@@ -220,7 +228,7 @@ export class Commands {
         }
       )
     } catch (err) {
-      throw handleRpcError(err)
+      throw await handleRpcErrorWithHealthCheck(err, this.checkHealth)
     }
   }
 
@@ -257,7 +265,7 @@ export class Commands {
         }
       )
     } catch (err) {
-      throw handleRpcError(err)
+      throw await handleRpcErrorWithHealthCheck(err, this.checkHealth)
     }
   }
 
@@ -298,7 +306,7 @@ export class Commands {
         }
       }
 
-      throw handleRpcError(err)
+      throw await handleRpcErrorWithHealthCheck(err, this.checkHealth)
     }
   }
 
@@ -354,11 +362,12 @@ export class Commands {
         opts?.onStderr,
         undefined,
         (data, stdinOpts) => this.sendStdin(pid, data, stdinOpts),
-        (stdinOpts) => this.closeStdin(pid, stdinOpts)
+        (stdinOpts) => this.closeStdin(pid, stdinOpts),
+        this.checkHealth
       )
     } catch (err) {
       cleanup()
-      throw handleRpcError(err)
+      throw await handleRpcErrorWithHealthCheck(err, this.checkHealth)
     }
   }
 
@@ -468,11 +477,12 @@ export class Commands {
         opts?.onStderr,
         undefined,
         (data, stdinOpts) => this.sendStdin(pid, data, stdinOpts),
-        (stdinOpts) => this.closeStdin(pid, stdinOpts)
+        (stdinOpts) => this.closeStdin(pid, stdinOpts),
+        this.checkHealth
       )
     } catch (err) {
       cleanup()
-      throw handleRpcError(err)
+      throw await handleRpcErrorWithHealthCheck(err, this.checkHealth)
     }
   }
 }

packages/js-sdk/src/sandbox/commands/pty.ts

@@ -19,8 +19,16 @@ import {
   setupRequestController,
 } from '../../connectionConfig'
 import { CommandHandle } from './commandHandle'
-import { authenticationHeader, handleRpcError } from '../../envd/rpc'
-import { handleProcessStartEvent } from '../../envd/api'
+import {
+  authenticationHeader,
+  handleRpcErrorWithHealthCheck,
+  SandboxHealthCheck,
+} from '../../envd/rpc'
+import {
+  checkSandboxHealth,
+  EnvdApiClient,
+  handleProcessStartEvent,
+} from '../../envd/api'
 
 export interface PtyCreateOpts
   extends Pick<ConnectionOpts, 'requestTimeoutMs' | 'signal'> {
@@ -74,18 +82,18 @@ export type PtyConnectOpts = Pick<PtyCreateOpts, 'onData' | 'timeoutMs'> &
 export class Pty {
   private readonly rpc: Client<typeof ProcessService>
   private readonly envdVersion: string
+  private readonly checkHealth: SandboxHealthCheck
 
   private readonly defaultPtyConnectionTimeout = 60_000 // 60 seconds
 
   constructor(
     private readonly transport: Transport,
-    private readonly connectionConfig: ConnectionConfig,
-    metadata: {
-      version: string
-    }
+    private readonly envdApi: EnvdApiClient,
+    private readonly connectionConfig: ConnectionConfig
   ) {
     this.rpc = createClient(ProcessService, this.transport)
-    this.envdVersion = metadata.version
+    this.envdVersion = envdApi.version
+    this.checkHealth = () => checkSandboxHealth(this.envdApi)
   }
 
   /**
@@ -144,11 +152,14 @@ export class Pty {
         events,
         undefined,
         undefined,
-        opts.onData
+        opts.onData,
+        undefined,
+        undefined,
+        this.checkHealth
       )
     } catch (err) {
       cleanup()
-      throw handleRpcError(err)
+      throw await handleRpcErrorWithHealthCheck(err, this.checkHealth)
     }
   }
 
@@ -198,11 +209,14 @@ export class Pty {
         events,
         undefined,
         undefined,
-        opts?.onData
+        opts?.onData,
+        undefined,
+        undefined,
+        this.checkHealth
       )
     } catch (err) {
       cleanup()
-      throw handleRpcError(err)
+      throw await handleRpcErrorWithHealthCheck(err, this.checkHealth)
     }
   }
 
@@ -242,7 +256,7 @@ export class Pty {
         }
       )
     } catch (err) {
-      throw handleRpcError(err)
+      throw await handleRpcErrorWithHealthCheck(err, this.checkHealth)
     }
   }
 
@@ -283,7 +297,7 @@ export class Pty {
         }
       )
     } catch (err) {
-      throw handleRpcError(err)
+      throw await handleRpcErrorWithHealthCheck(err, this.checkHealth)
     }
   }
 
@@ -327,7 +341,7 @@ export class Pty {
         }
       }
 
-      throw handleRpcError(err)
+      throw await handleRpcErrorWithHealthCheck(err, this.checkHealth)
     }
   }
 }