Skip to content

fix(instance): release resources at worktree boundaries#1709

Merged
wqymi merged 6 commits into
mainfrom
codex/fix-mcp-instance-lifecycle
Jul 14, 2026
Merged

fix(instance): release resources at worktree boundaries#1709
wqymi merged 6 commits into
mainfrom
codex/fix-mcp-instance-lifecycle

Conversation

@wqymi

@wqymi wqymi commented Jul 13, 2026

Copy link
Copy Markdown
Collaborator

Problem

Isolated worktrees can finish or be removed while their directory-scoped Instance remains cached. The stale Instance keeps subprocesses and other scoped resources alive, including after the directory itself has disappeared.

Fix

Add bounded, directory-addressable Instance teardown and invoke it when an isolated agent settles and before worktree removal. The captured cache entry is removed before cleanup, replacement instances are protected from late teardown, missing paths are still reclaimed, and session CWD overrides are cleared only for the owning directory.

Verification

  • 50 focused tests passed; 1 platform-only test skipped
  • Package and push-hook TypeScript checks passed
  • Lint completed with 0 errors (3 pre-existing workflow warnings)
  • Added coverage for blocked disposers, replacement-instance races, missing worktree paths, retained worktrees, and cross-directory session CWD isolation

Post-Deploy Monitoring & Validation

  • Watch logs for instance dispose did not complete and unexpected increases in long-lived child processes grouped by removed worktree directories.
  • Healthy signal: completed/removed worktrees no longer retain directory-scoped subprocesses, while resumed worktrees initialize once and unrelated session CWDs remain unchanged.
  • Failure signal: teardown warnings repeat for the same directory, workflow completion gains sustained ~2s latency, or child-process counts continue growing after worktree removal.
  • Validate over the first 24 hours; owner: MiMo Code maintainers.
  • Roll back this commit if teardown warnings become frequent or worktree reuse/session path behavior regresses.

Compound Engineering
🤖 Generated with GPT-5 (context window not disclosed, extended reasoning) via Codex

wqymi added a commit that referenced this pull request Jul 14, 2026
- Add RECLAIM_WORKTREE_TIMEOUT_MS (10s) to worktree.remove during reclaim
- Add RECLAIM_ACTOR_TIMEOUT_MS (5s) to actor.cancel during reclaim
- Add FIBER_INTERRUPT_TIMEOUT_MS (5s) to Fiber.interrupt in cancelEntry

These timeouts prevent cancel from hanging indefinitely when:
1. git worktree remove --force blocks on processes using the worktree
2. A child actor is stuck in an uninterruptible operation
3. Fiber.interrupt stalls on a hung LLM fetch

The specific fix addresses the 30s timeout in:
'WorkflowRuntime worktree isolation > cancel removes worktrees of in-flight isolated agents'

Root cause: PR #1709 added Instance.disposeDirectory calls in worktree.remove
which increased the time for cancel cleanup. Combined with git worktree remove
--force potentially blocking, the cancel operation could exceed the test timeout.
wqymi added a commit that referenced this pull request Jul 14, 2026
- Add RECLAIM_WORKTREE_TIMEOUT_MS (10s) to worktree.remove during reclaim
- Add RECLAIM_ACTOR_TIMEOUT_MS (5s) to actor.cancel during reclaim
- Add FIBER_INTERRUPT_TIMEOUT_MS (5s) to Fiber.interrupt in cancelEntry

These timeouts prevent cancel from hanging indefinitely when:
1. git worktree remove --force blocks on processes using the worktree
2. A child actor is stuck in an uninterruptible operation
3. Fiber.interrupt stalls on a hung LLM fetch

The specific fix addresses the 30s timeout in:
'WorkflowRuntime worktree isolation > cancel removes worktrees of in-flight isolated agents'

Root cause: PR #1709 added Instance.disposeDirectory calls in worktree.remove
which increased the time for cancel cleanup. Combined with git worktree remove
--force potentially blocking, the cancel operation could exceed the test timeout.
@wqymi wqymi force-pushed the codex/fix-mcp-instance-lifecycle branch 2 times, most recently from 4d8c592 to 5c1e1f8 Compare July 14, 2026 14:19
wqymi added a commit that referenced this pull request Jul 14, 2026
- Add RECLAIM_WORKTREE_TIMEOUT_MS (10s) to worktree.remove during reclaim
- Add RECLAIM_ACTOR_TIMEOUT_MS (5s) to actor.cancel during reclaim
- Add FIBER_INTERRUPT_TIMEOUT_MS (5s) to Fiber.interrupt in cancelEntry

These timeouts prevent cancel from hanging indefinitely when:
1. git worktree remove --force blocks on processes using the worktree
2. A child actor is stuck in an uninterruptible operation
3. Fiber.interrupt stalls on a hung LLM fetch

The specific fix addresses the 30s timeout in:
'WorkflowRuntime worktree isolation > cancel removes worktrees of in-flight isolated agents'

Root cause: PR #1709 added Instance.disposeDirectory calls in worktree.remove
which increased the time for cancel cleanup. Combined with git worktree remove
--force potentially blocking, the cancel operation could exceed the test timeout.
wqymi added 5 commits July 14, 2026 22:25
- Add RECLAIM_WORKTREE_TIMEOUT_MS (10s) to worktree.remove during reclaim
- Add RECLAIM_ACTOR_TIMEOUT_MS (5s) to actor.cancel during reclaim
- Add FIBER_INTERRUPT_TIMEOUT_MS (5s) to Fiber.interrupt in cancelEntry

These timeouts prevent cancel from hanging indefinitely when:
1. git worktree remove --force blocks on processes using the worktree
2. A child actor is stuck in an uninterruptible operation
3. Fiber.interrupt stalls on a hung LLM fetch

The specific fix addresses the 30s timeout in:
'WorkflowRuntime worktree isolation > cancel removes worktrees of in-flight isolated agents'

Root cause: PR #1709 added Instance.disposeDirectory calls in worktree.remove
which increased the time for cancel cleanup. Combined with git worktree remove
--force potentially blocking, the cancel operation could exceed the test timeout.
…ngUntil

Two 30s-timeout tests used llm.hang (Stream.never + real TCP) and relied on
Fiber.interrupt to unwind through the HTTP client stack within 30s. Under CI
load the TCP cleanup alone can exceed 10s, pushing the total past deadline.

Design-level fix:
- llm-server: add hangUntil(Deferred) that blocks on Effect-level
  Deferred.await instead of Stream.never. Fiber.interrupt unwinds the
  Deferred at the Effect layer without touching TCP — fast and deterministic.
- runtime-worktree cancel test: replace llm.hang with hangUntil; after cancel
  the Deferred is resolved so the agent unwinds cleanly.
- spawn-notification woken-peer test: increase polling budget (600×50ms = 30s)
  to accommodate unbounded woken-turn LLM latency under CI load.
@wqymi wqymi force-pushed the codex/fix-mcp-instance-lifecycle branch from 5c1e1f8 to 1e10597 Compare July 14, 2026 14:25
… deadlock

In cancelEntry, Fiber.interrupt was called AFTER reclaim (which does
worktree.remove → disposeDirectory → ScopedCache.invalidate). The
scope-close finalizer waits for the agent fiber to finish, but the fiber
was still running in Stream.never → 30s deadlock.

Reorder so Fiber.interrupt runs first, letting the scope-close finalize
without waiting on a dead fiber.

PR #1709 — fixes the 30s timeout in 'cancel removes worktrees' test.
@wqymi wqymi merged commit 8aff671 into main Jul 14, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant