Skip to content

Fix read-pool lease leak that deadlocks updateSchema (#356)#357

Merged
simolus3 merged 1 commit into
powersync-ja:mainfrom
marioortizmanero:fix/readpool-lease-leak-356
Jun 8, 2026
Merged

Fix read-pool lease leak that deadlocks updateSchema (#356)#357
simolus3 merged 1 commit into
powersync-ja:mainfrom
marioortizmanero:fix/readpool-lease-leak-356

Conversation

@marioortizmanero

@marioortizmanero marioortizmanero commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

What

ReadPool's connection-lease handoff channel (available) lacked an onUndeliveredElement handler. When a read {} (or the withAllConnections acquisition loop) is cancelled during the channel rendezvous — after a worker sends (connection, done) but before the receiver delivers it into its try/finally — the element is silently dropped. done.complete(Unit) is never called, so that worker parks in done.await() forever and the connection is permanently lost from the pool.

Once any lease leaks, ReadPool.withAllConnections { repeat(size) { available.receive() } } can never acquire all connections and hangs forever. updateSchema() reaches the pool through exactly this path:

PowerSyncDatabaseImpl.updateSchema
  → mutex.withLock
    → InternalDatabaseImpl.updateSchema
      → InternalConnectionPool.withAllConnections
        → ReadPool.withAllConnections → repeat(size){ available.receive() }

so a single leaked read lease wedges updateSchema indefinitely while it holds the database mutex.

Why #356 looked schema-related

It isn't about passing the same Schema instance or about connect(). The reporter's app calls updateSchema on cold start while concurrent watch() queries are being cancelled (auth churn over a dead/slow network) — that cancellation is what leaks the lease. A standalone updateSchema with no concurrent reads (the minimal repro) never leaks, which is why it passed.

Fix

Add onUndeliveredElement to the available channel:

onUndeliveredElement = { (_, done) -> done.complete(Unit) }

onUndeliveredElement fires exactly when an element was handed to the channel but not received (cancelled receiver, or close()/cancel() with an in-flight element). Completing done returns the connection to circulation. CompletableDeferred.complete() is idempotent and non-suspending, so there's no double-completion hazard with the normal finally paths. This covers both leak windows — read() and the withAllConnections add(receive()) gap — and the close path.

How we found this

A thread dump was useless — every thread was idle/parked with no SQLite or native frame (a suspended-coroutine deadlock, not a blocked thread). Coroutine debug probes are inert on Android/ART, so we reflected into the live pool from a debug build at the moment of the hang: all 5 read workers alive, DB mutex held, writeLockMutex still free, but one connection checked out and never returned — i.e. withAllConnections stuck on its final available.receive() after a leaked read lease. That pointed straight at ReadPool and the missing onUndeliveredElement.

Test

ReadPoolLeaseLeakTest hammers the read pool with concurrent reads cancelled at varied points, then asserts updateSchema completes under a timeout. It runs on real dispatchers (the race is hidden by the virtual-time test dispatcher).

  • Before the fix: fails with TimeoutCancellationException from updateSchema.
  • After the fix: passes in a few ms.

Full :common:jvmTest suite: 130 tests, 0 failures.

Fixes #356.

ReadPool's connection-lease handoff channel lacked an onUndeliveredElement
handler. When a read {} (or the withAllConnections acquisition loop) was
cancelled during the channel rendezvous, the leased element was silently
dropped: the receiver never ran done.complete(Unit), so the owning read
worker parked in done.await() forever and that connection was permanently
lost from the pool. Once any lease leaked, updateSchema() (via
withAllConnections -> repeat(size){ available.receive() }) could never
acquire all connections and deadlocked while holding the database mutex.

This is the underlying cause of powersync-ja#356: the reporter's app calls updateSchema
on cold start while concurrent watch() queries are being cancelled (auth
churn over a dead/slow network), which leaks a lease. A standalone
updateSchema with no concurrent reads never leaks, which is why the minimal
repro passed.

Add onUndeliveredElement = { (_, done) -> done.complete(Unit) } so a dropped
lease returns its connection to circulation. CompletableDeferred.complete()
is idempotent, so this is safe alongside the normal finally paths.

Adds ReadPoolLeaseLeakTest, which leaks a lease via concurrent cancelled
reads and then asserts updateSchema completes; it fails before the fix and
passes after.

@simolus3 simolus3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, thanks for the fix!

@simolus3 simolus3 merged commit 0225d44 into powersync-ja:main Jun 8, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PowerSyncDatabase.updateSchema() deadlocks on same-schema call before connect()

2 participants