fix(concurrency): centralize write transactions with bounded SQLITE_BUSY retry - #2
Closed
codeo1io wants to merge 1 commit into
Closed
fix(concurrency): centralize write transactions with bounded SQLITE_BUSY retry#2codeo1io wants to merge 1 commit into
codeo1io wants to merge 1 commit into
Conversation
…USY retry Replaces 9 hand-rolled BEGIN IMMEDIATE / COMMIT / finally-ROLLBACK blocks with a single shared runWriteTransaction helper that adds a bounded SQLITE_BUSY retry loop (4 attempts, 100ms backoff). This is a better concurrency solution than the previous per-site pattern because: 1. Centralized retry policy: transient SQLITE_BUSY from cross-process WAL writer lock contention (sibling migration, dreamer run, Channel-2 bulk delivery) now waits-and-retries instead of propagating up and disabling Magic Context for the run. Previously each site either swallowed, propagated, or retried inconsistently. 2. Composition-safe: if called inside an existing db.transaction() or another runWriteTransaction, the body runs inline WITHOUT issuing a nested BEGIN (SQLite doesn't allow nested BEGIN; the outer transaction already holds the writer lock). Detected via the bun:sqlite inTransaction flag or the node:sqlite shim's isTransaction flag. 3. One correct transaction plumbing pattern instead of 9 slightly-different copies. Several sites were missing the ROLLBACK path or had subtle ordering bugs at the edges. Sites converted (all sync write paths with hand-rolled BEGIN IMMEDIATE): - dreamer/lease.ts (acquireLease, renewLease, releaseLease) - git-commits/sweep-coordinator.ts (acquireGitSweepLease, renewGitSweepLease, markGitSweepSuccessAndRelease) - message-index.ts (indexMessagesAfterOrdinal — cross-process FTS dedup) - workspaces.ts (bumpEpochsForWorkspaceMembers, bumpEpochsForWorkspaceMemberSet) - compartment-storage.ts (replaceAllCompartmentStateAndBumpDepth, promoteRecompStaging) - key-files/project-key-files.ts (replaceAllKeyFiles) - key-files/identify-key-files.ts (commitKeyFilesUnderLease) - hooks/compartment-runner-recomp.ts (promoteRecompStagingWithM0Mutation) - hooks/compartment-runner-incremental.ts (historian publish path) inject-compartments.ts is intentionally NOT converted: its two BEGIN IMMEDIATE blocks have complex early-return-with-rollback-and-fallback-read patterns (Phase 3 materialization contention retry, soft-refresh cache replay) that don't map cleanly onto the shared helper without significant restructuring. The helper is still a net win across the 9 simpler sites. Verification: - bun run typecheck (tsc --noEmit + scripts) passes - bun run lint (biome check) passes - 12 new unit tests for runWriteTransaction/runWriteTransactionAsync pass (BEGIN/COMMIT, rollback on throw, transient BUSY retry, max-attempts give-up, nested-transaction composition, non-transient error passthrough) - All existing tests in the converted files' test suites pass: dreamer/lease.test.ts (7), git-commits (19), compartment-storage-atomic (8), compartment-storage-v6, compartment-lease, compartment-runner-recomp-fk (4), compartment-runner-partial-recomp, key-files (14), workspaces, storage-db (11), message-index Net: -199 lines / +40 lines across the 9 converted sites, +201 lines for the shared helper + 156 lines for its tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The plugin issues 100+ write transactions across 40+ files against ONE shared SQLite file (
context.db), opened by multiple processes (OpenCode + Pi, or two OpenCode instances). Each hand-rolledBEGIN IMMEDIATE / COMMIT / finally-ROLLBACKblock contends for the single WAL writer lock. Three problems:Inconsistent SQLITE_BUSY handling: when a sibling process held the writer lock past
busy_timeout, the thrownSQLITE_BUSYpropagated up and surfaced asfailed to load plugin ... database is lockedorHit a transient issue comparting history this turn. Some sites retried, some swallowed, some propagated — unpredictable under real multi-process load.Duplicated transaction plumbing: every
BEGIN IMMEDIATEsite reimplemented the same try/commit/finally-rollback pattern, slightly differently, sometimes without the rollback path. Bugs leaked in at the edges.No retry on transient contention: a long-running sibling transaction (large migration, dreamer run, Channel-2 bulk delivery) made the plugin disable itself for the run instead of waiting.
Fix
A single shared
runWriteTransaction(db, body)helper (packages/plugin/src/shared/write-transaction.ts) that:BEGIN IMMEDIATE / COMMITin a bounded SQLITE_BUSY retry loop (4 attempts, 100ms backoff) so transient cross-process contention makes us wait-and-retry instead of throwing.db.transaction()or anotherrunWriteTransaction, the body runs inline WITHOUT issuing a nestedBEGIN(SQLite doesn't allow nestedBEGIN; the outer transaction already holds the writer lock). Detected via the bun:sqliteinTransactionflag or the node:sqlite shim'sisTransactionflag.Also adds
runWriteTransactionAsyncfor async call sites (same retry + composition semantics, body may be async).Sites converted (9 sync write paths)
dreamer/lease.tsacquireLease,renewLease,releaseLeasegit-commits/sweep-coordinator.tsacquireGitSweepLease,renewGitSweepLease,markGitSweepSuccessAndReleasemessage-index.tsindexMessagesAfterOrdinal(cross-process FTS dedup)workspaces.tsbumpEpochsForWorkspaceMembers,bumpEpochsForWorkspaceMemberSetcompartment-storage.tsreplaceAllCompartmentStateAndBumpDepth,promoteRecompStagingkey-files/project-key-files.tsreplaceAllKeyFileskey-files/identify-key-files.tscommitKeyFilesUnderLeasehooks/compartment-runner-recomp.tspromoteRecompStagingWithM0Mutationhooks/compartment-runner-incremental.tsinject-compartments.tsis intentionally NOT converted: its twoBEGIN IMMEDIATEblocks have complex early-return-with-rollback-and-fallback-read patterns (Phase 3 materialization contention retry, soft-refresh cache replay) that don't map cleanly onto the shared helper without significant restructuring. The helper is still a net win across the 9 simpler sites.Removed
Two duplicate local
runImmediatefunctions (indreamer/lease.tsandgit-commits/sweep-coordinator.ts) and the localisInTransactionhelper inworkspaces.ts— all subsumed by the shared helper.Verification
bun run typecheck(tsc --noEmit+ scripts) ✅bun run lint(biome check) ✅runWriteTransaction/runWriteTransactionAsync✅inTransaction/isTransaction)dreamer/lease.test.ts(7),git-commits(19),compartment-storage-atomic(8),compartment-storage-v6,compartment-lease,compartment-runner-recomp-fk(4),compartment-runner-partial-recomp,key-files(14),workspaces,storage-db(11),message-indexRelationship to #1
This complements #1 (cold-start
database is lockedfix). #1 fixed the cold-open race (busy_timeout installed before the first read). This PR fixes the steady-state write contention (bounded retry on transient SQLITE_BUSY during normal operation). Together they eliminate both the startup and runtime sources of thedatabase is lockedplugin-disable failures.Net: −199 lines / +40 lines across the 9 converted sites; +201 lines for the shared helper; +156 lines for its tests.