Enable default model retries (#4620)

## Summary

Enable provider retries for Electric Agents model calls by default.
Transient LLM/provider failures now get two provider-level retry
attempts before the run fails, while callers can still opt out with
`modelMaxRetries: 0`.

## Root Cause

The agent runtime already threaded `modelMaxRetries` through to `pi-ai`
provider stream calls, but its default was `0`. That meant
normal/default agent runs did not retry retryable provider failures such
as transient server errors, rate limits, or network blips unless each
agent explicitly configured retries.

## Approach

- Change the runtime default from no retries to two provider retries:

```ts
const DEFAULT_MODEL_MAX_RETRIES = 2
```

- Keep the existing `modelMaxRetries` option behavior intact. Because
the runtime still uses nullish coalescing, explicit values continue to
override the default, including `modelMaxRetries: 0` for no-retry
behavior.
- Add coverage for the default options passed into provider stream
calls.
- Update existing option-forwarding tests to use `modelMaxRetries: 5`,
so they clearly verify explicit overrides rather than matching the new
default by coincidence.

## Key Invariants

- Default provider stream calls receive `{ timeoutMs: 30_000,
maxRetries: 2 }`.
- Explicit `modelMaxRetries` values override the default.
- Explicit `modelMaxRetries: 0` remains a valid way to disable retries.
- Existing stream options, especially `AbortSignal`, continue to be
preserved when timeout/retry options are injected.

## Non-goals

- This does not add Electric-runtime-level retry orchestration around
failed assistant messages.
- This does not change model provider error classification.
- This does not alter run/event replay semantics or retry after
partially emitted model output.

## Trade-offs

This is the smallest safe mitigation: retries happen in the
provider/pi-ai layer before the Electric runtime observes a failed
stream. That avoids introducing replay or duplicate-event semantics in
this PR.

The trade-off is that transient failures may now take longer to surface,
because the provider call can retry before failing. Users who prefer
fail-fast behavior can set `modelMaxRetries: 0`.

## Verification

```bash
cd packages/agents-runtime
pnpm exec vitest run test/pi-adapter.test.ts test/model-provider-error.test.ts --pool-options.threads.maxThreads=2
```

Result:

```txt
✓ test/pi-adapter.test.ts (30 tests)
✓ test/model-provider-error.test.ts (11 tests)

Test Files  2 passed
Tests       41 passed
```

## Files changed

- `.changeset/enable-agent-model-retries.md` — Adds a patch changeset
for `@electric-ax/agents-runtime` describing the default retry behavior
and opt-out.
- `packages/agents-runtime/src/pi-adapter.ts` — Changes the default
model retry count from `0` to `2`.
- `packages/agents-runtime/test/pi-adapter.test.ts` — Adds default
option coverage and strengthens explicit override assertions.

Author: KyleAMathews

.changeset/enable-agent-model-retries.md

@@ -0,0 +1,5 @@
+---
+'@electric-ax/agents-runtime': patch
+---
+
+Enable provider retries for agent model calls by default so transient LLM errors are retried. Set `modelMaxRetries: 0` to preserve the previous no-retry behavior.

packages/agents-runtime/src/pi-adapter.ts

@@ -86,7 +86,7 @@ export interface PiAdapterOptions {
 }
 
 const DEFAULT_MODEL_TIMEOUT_MS = 30_000
-const DEFAULT_MODEL_MAX_RETRIES = 0
+const DEFAULT_MODEL_MAX_RETRIES = 2
 
 interface PiAgentAdapterConfig {
   entityUrl: string

packages/agents-runtime/test/pi-adapter.test.ts

@@ -127,6 +127,60 @@ describe(`createPiAgentAdapter`, () => {
     expect(handle.isRunning()).toBe(false)
   })
 
+  it(`passes default timeout and retry options to provider stream calls`, async () => {
+    let seenOptions: { timeoutMs?: number; maxRetries?: number } | undefined
+    const completedMessage: AssistantMessage = {
+      role: `assistant`,
+      content: [{ type: `text`, text: `ok` }],
+      api: `anthropic-messages`,
+      provider: `anthropic`,
+      model: `claude-sonnet-4-5-20250929`,
+      usage: {
+        input: 0,
+        output: 0,
+        cacheRead: 0,
+        cacheWrite: 0,
+        totalTokens: 0,
+        cost: {
+          input: 0,
+          output: 0,
+          cacheRead: 0,
+          cacheWrite: 0,
+          total: 0,
+        },
+      },
+      stopReason: `stop`,
+      timestamp: Date.now(),
+    }
+
+    const factory = createPiAgentAdapter({
+      systemPrompt: `Test system prompt`,
+      model: `claude-sonnet-4-5-20250929`,
+      tools: [],
+      streamFn: (_model, _context, options) => {
+        seenOptions = {
+          timeoutMs: options?.timeoutMs,
+          maxRetries: options?.maxRetries,
+        }
+        const stream = createAssistantMessageEventStream()
+        queueMicrotask(() => stream.end(completedMessage))
+        return stream
+      },
+    })
+
+    const handle = factory({
+      entityUrl: `test/entity-1`,
+      epoch: 1,
+      messages: [],
+      outboundIdSeed: { run: 0, step: 0, msg: 0, tc: 0, reasoning: 0 },
+      writeEvent: (_event: ChangeEvent) => {},
+    })
+
+    await handle.run(`hello`)
+
+    expect(seenOptions).toEqual({ timeoutMs: 30_000, maxRetries: 2 })
+  })
+
   it(`passes timeout and retry options to each provider stream call`, async () => {
     const seenOptions: Array<{ timeoutMs?: number; maxRetries?: number }> = []
     const completedMessage: AssistantMessage = {
@@ -158,7 +212,7 @@ describe(`createPiAgentAdapter`, () => {
       model: `claude-sonnet-4-5-20250929`,
       tools: [],
       modelTimeoutMs: 1234,
-      modelMaxRetries: 2,
+      modelMaxRetries: 5,
       streamFn: (_model, _context, options) => {
         seenOptions.push({
           timeoutMs: options?.timeoutMs,
@@ -180,7 +234,7 @@ describe(`createPiAgentAdapter`, () => {
 
     await handle.run(`hello`)
 
-    expect(seenOptions).toEqual([{ timeoutMs: 1234, maxRetries: 2 }])
+    expect(seenOptions).toEqual([{ timeoutMs: 1234, maxRetries: 5 }])
   })
 
   it(`preserves existing stream options while injecting timeout and retry options`, async () => {
@@ -214,11 +268,11 @@ describe(`createPiAgentAdapter`, () => {
       model: `claude-sonnet-4-5-20250929`,
       tools: [],
       modelTimeoutMs: 1234,
-      modelMaxRetries: 2,
+      modelMaxRetries: 5,
       streamFn: (_model, _context, options) => {
         capturedSignal = options?.signal
         expect(options?.timeoutMs).toBe(1234)
-        expect(options?.maxRetries).toBe(2)
+        expect(options?.maxRetries).toBe(5)
         const stream = createAssistantMessageEventStream()
         queueMicrotask(() => stream.end(completedMessage))
         return stream