fix(storage): raise SQLite busy_timeout + retry headroom for fleet load (#577)

The Storage constructor previously set busy_timeout=5000ms and withBusyRetry
defaulted to 5 attempts capped at 250ms — tuned for the worker + MCP + CLI
trio. Under codex-fleet load (~30 codex panes + active Claude sessions +
worker = 30+ concurrent writers on ~/.colony/data.db) the 5s SQLite window
plus ~355ms Node retry tail got exhausted, surfacing SQLITE_BUSY: database
is locked to callers.

Bump busy_timeout to 15000ms and widen withBusyRetry defaults to 8 attempts
with up to 1000ms per-attempt backoff (~3.85s total worst case). Happy-path
callers are unaffected because no busy error means no retry sleep. Update
the corresponding regression test assertion.

Co-authored-by: NagyVikt <nagy.viktordp@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: 3 people

.changeset/storage-busy-timeout-and-retry-headroom.md

@@ -0,0 +1,13 @@
+---
+'colonyq': patch
+---
+
+Raise SQLite contention headroom in `@colony/storage` so the worker daemon,
+MCP server, CLI hooks, and codex-fleet panes can share `~/.colony/data.db`
+without surfacing `SQLITE_BUSY: database is locked` to callers. The
+`Storage` constructor now sets `busy_timeout=15000` (was 5000), and
+`withBusyRetry` defaults bump to 8 attempts with up-to-1s backoff (was 5
+attempts / 250ms cap). Happy-path callers are unaffected because no busy
+error still means no retry sleep; sustained contention from ~30+ concurrent
+writers — the codex-fleet shape that triggered this — now has ~3.85s of
+combined SQLite + Node retry headroom before raising.

openspec/changes/agent-claude-colony-storage-busy-timeout-retry-under-2026-05-16-12-11/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-05-16

openspec/changes/agent-claude-colony-storage-busy-timeout-retry-under-2026-05-16-12-11/notes.md

@@ -0,0 +1,21 @@
+# agent-claude-colony-storage-busy-timeout-retry-under-2026-05-16-12-11 (minimal / T1)
+
+Branch: `agent/claude/colony-storage-busy-timeout-retry-under-2026-05-16-12-11`
+
+Raise SQLite contention headroom in `@colony/storage` so the worker daemon, MCP server, CLI hooks, and codex-fleet panes can share `~/.colony/data.db` without surfacing `SQLITE_BUSY: database is locked` to callers.
+
+- `busy_timeout` 5000 → 15000 ms (one connection-scoped pragma in the `Storage` constructor).
+- `withBusyRetry` defaults: `maxAttempts` 5 → 8, `baseDelayMs` 5 → 10, `maxDelayMs` 250 → 1000. New worst-case wait ~3.85s vs old ~0.355s; happy-path callers stay sub-ms because no busy error means no retry sleep.
+- Updated `keeps busy_timeout set to N` assertion in `test/busy-retry.test.ts` from 5000 to 15000.
+
+Triggered by a real lock storm on 2026-05-16: ~30 codex-fleet panes + 4 live Claude sessions + 1 worker = 34+ concurrent writers, which exhausted the previous 5s window + 5-retry tail. The fleet teardown is the immediate fix; this PR is the durable headroom.
+
+## Handoff
+
+- Handoff: change=`agent-claude-colony-storage-busy-timeout-retry-under-2026-05-16-12-11`; branch=`agent/claude/colony-storage-busy-timeout-retry-under-2026-05-16-12-11`; scope=`packages/storage src + test only`; action=`finish via PR to main`.
+
+## Cleanup
+
+- [ ] Run: `gx branch finish --branch agent/claude/colony-storage-busy-timeout-retry-under-2026-05-16-12-11 --base main --via-pr --wait-for-merge --cleanup`
+- [ ] Record PR URL + `MERGED` state in the completion handoff.
+- [ ] Confirm sandbox worktree is gone (`git worktree list`, `git branch -a`).

packages/storage/src/busy-retry.ts

@@ -1,26 +1,27 @@
 // Synchronous SQLITE_BUSY retry wrapper for better-sqlite3 writes.
 //
-// Background: the Storage constructor sets busy_timeout=5000 and
+// Background: the Storage constructor sets busy_timeout=15000 and
 // journal_mode=WAL, which together absorb the overwhelming majority of
 // contention between the worker daemon, MCP server, and CLI hooks. The
 // remaining tail comes from edge cases — a long checkpoint, a migration
 // that holds a write transaction, or a misbehaving hook that opens its
-// own connection. In those cases SQLite still raises SQLITE_BUSY after
-// the busy_timeout window expires, which the 168h gain telemetry saw
-// surface once on task_claim_quota_release_expired.
+// own connection — plus sustained pressure from the codex-fleet shape
+// (~30+ concurrent writers). In those cases SQLite still raises
+// SQLITE_BUSY after the busy_timeout window expires.
 //
 // This helper gives callers a defensive Node-level retry on top of
-// SQLite's own busy_timeout. Five attempts with backoff 5/20/80/250ms
-// cap total wait at ~355ms — small enough that the caller is not
-// noticeably slower, large enough that a transient checkpoint or
-// short-held write transaction has time to clear.
+// SQLite's own busy_timeout. Eight attempts with backoff
+// 10/40/160/640/1000/1000/1000ms cap total wait at ~3.85s — bounded
+// enough to keep CLI hooks under the 150ms p95 budget on the happy
+// path while giving a transient checkpoint or fleet-burst window time
+// to clear.
 
 export interface BusyRetryOptions {
-  /** Maximum number of attempts (including the first). Defaults to 5. */
+  /** Maximum number of attempts (including the first). Defaults to 8. */
   maxAttempts?: number;
-  /** Base delay in milliseconds; backoff is base * 4^(attempt-1) capped at 250ms. Defaults to 5. */
+  /** Base delay in milliseconds; backoff is base * 4^(attempt-1) capped at maxDelayMs. Defaults to 10. */
   baseDelayMs?: number;
-  /** Maximum per-attempt delay in milliseconds. Defaults to 250. */
+  /** Maximum per-attempt delay in milliseconds. Defaults to 1000. */
   maxDelayMs?: number;
 }
 
@@ -55,9 +56,9 @@ function sleepSync(ms: number): void {
  * `maxAttempts` retries.
  */
 export function withBusyRetry<T>(fn: () => T, opts: BusyRetryOptions = {}): T {
-  const maxAttempts = opts.maxAttempts ?? 5;
-  const baseDelayMs = opts.baseDelayMs ?? 5;
-  const maxDelayMs = opts.maxDelayMs ?? 250;
+  const maxAttempts = opts.maxAttempts ?? 8;
+  const baseDelayMs = opts.baseDelayMs ?? 10;
+  const maxDelayMs = opts.maxDelayMs ?? 1000;
   let attempt = 0;
   // The loop body always either returns or throws, so the linter is
   // happy with the `while (true)` shape.

packages/storage/src/storage.ts

@@ -320,8 +320,11 @@ export class Storage {
     this.db = new Database(dbPath, opts.readonly ? { readonly: true } : {});
     // busy_timeout is connection-scoped. Multiple processes (worker daemon,
     // MCP server, CLI hooks) hit the same WAL file; without this they trip
-    // SQLITE_BUSY immediately on contention. 5s lets the kernel retry.
-    this.db.pragma('busy_timeout = 5000');
+    // SQLITE_BUSY immediately on contention. 15s absorbs sustained
+    // contention from ~30+ concurrent writers (the codex-fleet shape):
+    // 5s was tuned for the worker+MCP+CLI trio and got exhausted under
+    // fleet load on 2026-05-16.
+    this.db.pragma('busy_timeout = 15000');
     // WAL mode lets readers and a single writer coexist without blocking,
     // which matters because the worker daemon, MCP server, and CLI hooks
     // all hit the same DB file concurrently. Without WAL the default

packages/storage/test/busy-retry.test.ts

@@ -25,11 +25,11 @@ describe('Storage WAL mode', () => {
     expect(mode[0]?.toLowerCase()).toBe('wal');
   });
 
-  it('keeps busy_timeout set to 5000', () => {
+  it('keeps busy_timeout set to 15000', () => {
     const timeout = (storage as unknown as { db: { pragma: (sql: string) => unknown[] } }).db
       .pragma('busy_timeout')
       .map((r: unknown) => (r as { timeout: number }).timeout);
-    expect(timeout[0]).toBe(5000);
+    expect(timeout[0]).toBe(15000);
   });
 });