Fix read-pool lease leak that deadlocks updateSchema (#356)#357
Merged
simolus3 merged 1 commit intoJun 8, 2026
Merged
Conversation
ReadPool's connection-lease handoff channel lacked an onUndeliveredElement
handler. When a read {} (or the withAllConnections acquisition loop) was
cancelled during the channel rendezvous, the leased element was silently
dropped: the receiver never ran done.complete(Unit), so the owning read
worker parked in done.await() forever and that connection was permanently
lost from the pool. Once any lease leaked, updateSchema() (via
withAllConnections -> repeat(size){ available.receive() }) could never
acquire all connections and deadlocked while holding the database mutex.
This is the underlying cause of powersync-ja#356: the reporter's app calls updateSchema
on cold start while concurrent watch() queries are being cancelled (auth
churn over a dead/slow network), which leaks a lease. A standalone
updateSchema with no concurrent reads never leaks, which is why the minimal
repro passed.
Add onUndeliveredElement = { (_, done) -> done.complete(Unit) } so a dropped
lease returns its connection to circulation. CompletableDeferred.complete()
is idempotent, so this is safe alongside the normal finally paths.
Adds ReadPoolLeaseLeakTest, which leaks a lease via concurrent cancelled
reads and then asserts updateSchema completes; it fails before the fix and
passes after.
simolus3
approved these changes
Jun 8, 2026
simolus3
left a comment
Contributor
There was a problem hiding this comment.
Good catch, thanks for the fix!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ReadPool's connection-lease handoff channel (available) lacked anonUndeliveredElementhandler. When aread {}(or thewithAllConnectionsacquisition loop) is cancelled during the channel rendezvous — after a workersends(connection, done)but before the receiver delivers it into itstry/finally— the element is silently dropped.done.complete(Unit)is never called, so that worker parks indone.await()forever and the connection is permanently lost from the pool.Once any lease leaks,
ReadPool.withAllConnections { repeat(size) { available.receive() } }can never acquire all connections and hangs forever.updateSchema()reaches the pool through exactly this path:so a single leaked read lease wedges
updateSchemaindefinitely while it holds the database mutex.Why #356 looked schema-related
It isn't about passing the same
Schemainstance or aboutconnect(). The reporter's app callsupdateSchemaon cold start while concurrentwatch()queries are being cancelled (auth churn over a dead/slow network) — that cancellation is what leaks the lease. A standaloneupdateSchemawith no concurrent reads (the minimal repro) never leaks, which is why it passed.Fix
Add
onUndeliveredElementto theavailablechannel:onUndeliveredElementfires exactly when an element was handed to the channel but not received (cancelled receiver, orclose()/cancel()with an in-flight element). Completingdonereturns the connection to circulation.CompletableDeferred.complete()is idempotent and non-suspending, so there's no double-completion hazard with the normalfinallypaths. This covers both leak windows —read()and thewithAllConnectionsadd(receive())gap — and the close path.How we found this
A thread dump was useless — every thread was idle/parked with no SQLite or native frame (a suspended-coroutine deadlock, not a blocked thread). Coroutine debug probes are inert on Android/ART, so we reflected into the live pool from a debug build at the moment of the hang: all 5 read workers alive, DB mutex held,
writeLockMutexstill free, but one connection checked out and never returned — i.e.withAllConnectionsstuck on its finalavailable.receive()after a leaked read lease. That pointed straight atReadPooland the missingonUndeliveredElement.Test
ReadPoolLeaseLeakTesthammers the read pool with concurrent reads cancelled at varied points, then assertsupdateSchemacompletes under a timeout. It runs on real dispatchers (the race is hidden by the virtual-time test dispatcher).TimeoutCancellationExceptionfromupdateSchema.Full
:common:jvmTestsuite: 130 tests, 0 failures.Fixes #356.