Remove the lock around the ClientsMap in the SubscriptionManager#2821
Merged
Conversation
We were seeing a deadlock where at least one Tokio task was blocked on this lock. @jsdt had some hypotheses about the specific Tokio behaviors that were causing this, but the important thing is that the lock is not necessary at all, and so we can remove it without delving into the dark depths of the Tokio scheduler. As part of this change, I split the send worker into being its own struct. That struct now holds its own copy of the clients map. This also simplifies the code that cleans up the send queue length metric.
Centril
approved these changes
Jun 3, 2025
Centril
left a comment
Centril
left a comment
Contributor
There was a problem hiding this comment.
Some minor comments, mostly requests for more docs, and otherwise LGTM.
Contributor
Author
|
Kim reports out-of-band that in his testing, he could not induce a deadlock with this branch. |
jsdt
reviewed
Jun 3, 2025
jsdt
left a comment
jsdt
left a comment
Contributor
There was a problem hiding this comment.
This gets rid of the lock, but I think the send worker still has a race condition, which I explained in a comment.
bfops
added
the
release-any
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of Changes
We were seeing a deadlock where at least one Tokio task was blocked on this lock. @jsdt had some hypotheses about the specific Tokio behaviors that were causing this, but the important thing is that the lock is not necessary at all, and so we can remove it without delving into the dark depths of the Tokio scheduler.
As part of this change, I split the send worker into being its own struct. That struct now holds its own copy of the clients map. This also simplifies the code that cleans up the send queue length metric.
Closes #2820 .
API and ABI breaking changes
N/a
Expected complexity level and risk
3: I may have missed a place where the client map is updated, which needs to notify the
SendWorker.Testing