Skip to content

fix: resolve concurrency bugs in eventbus and webtransport#3518

Merged
sukunrt merged 8 commits into
libp2p:masterfrom
Sahil-4555:fix/concurrency-safety
Jul 9, 2026
Merged

fix: resolve concurrency bugs in eventbus and webtransport#3518
sukunrt merged 8 commits into
libp2p:masterfrom
Sahil-4555:fix/concurrency-safety

Conversation

@Sahil-4555

Copy link
Copy Markdown
Contributor

Issue

observed two concurrency issues:

  1. Eventbus Deadlock: When multiple emitters concurrently broadcast events to a slow consumer (full queue), they can deadlock. Both the wildcard subscriber node and standard subscriber node shared a single slowConsumerTimer pointer. Concurrent emitters raced on Reset(), causing the timer to fire only once. One emitter consumed the tick and blocked on the channel, leaving other emitters permanently blocked on <-timer.C even after the queue was drained. This hang occurred under a read-lock (RLock), blocking any subscription closure (sub.Close()) or node updates from acquiring a write-lock.
  2. WebTransport Cert Manager Data Race: The SerializedCertHashes() getter read the shared m.serializedCertHashes slice without acquiring a lock. This caused a read/write data race with the background cert manager thread which updates the slice during certificate rotation. Returning the raw slice by reference also caused slice aliasing, exposing the caller to backing memory being concurrently mutated/resized.

Fix

  1. Eventbus Timer Isolation: Removed the shared slowConsumerTimer pointer field from node and wildcardNode. The slow consumer warning logic was refactored to use a local time.Timer allocated on the stack within emitAndLogError, eliminating shared timer state across concurrent emitters.
  2. WebTransport Thread-Safety: Added m.mx.RLock() to SerializedCertHashes() and returned a deep copy of the slice rather than a direct reference to avoid data races and slice aliasing.

Tests

  1. Eventbus: Added TestWildcardSlowConsumerDeadlock in p2p/host/eventbus/basic_test.go which simulates a full queue under concurrent emissions to verify that no deadlocks occur and the emitters safely proceed when the queue is drained.

@lidel lidel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. I'd been debugging the same bug myself, so I'm glad you beat me to it :)

I pushed two small test commits on top:

  • Moved the eventbus slow-consumer test over to synctest. It runs on a fake clock now, so it's fast and won't flake on CI the way the sleep-based version could.
  • Added two tests for SerializedCertHashes: one checks the getter returns a copy, the other runs it against cert rotation under -race.

@MarcoPolo fwiw lgtm, and Kubo could benefit from this, but your call

Comment thread p2p/host/eventbus/basic.go
Comment thread p2p/transport/webtransport/cert_manager.go Outdated
Comment thread p2p/transport/webtransport/cert_manager.go Outdated
@Sahil-4555
Sahil-4555 force-pushed the fix/concurrency-safety branch from 459c98c to 267126b Compare July 3, 2026 07:54
lidel added 2 commits July 3, 2026 13:40
Replace the wall-clock TestWildcardSlowConsumerDeadlock with a synctest
version that fires the warning timeout on a fake clock, so it runs
instantly and cannot flake on a loaded CI runner. Add a close-during-
stall test covering the safety valve where closing a subscription
releases an emit parked under the wildcard node read lock.
Add a test that the getter returns an independent copy, and a -race
test that hammers it during background certificate rotation. Both
fail on the unsynchronized getter and pass with the read lock and copy.
@Sahil-4555
Sahil-4555 force-pushed the fix/concurrency-safety branch from 267126b to efc1305 Compare July 3, 2026 08:12
@sukunrt
sukunrt force-pushed the fix/concurrency-safety branch 2 times, most recently from 48af017 to d557644 Compare July 3, 2026 11:06
@sukunrt
sukunrt requested a review from MarcoPolo July 3, 2026 11:19
@sukunrt
sukunrt force-pushed the fix/concurrency-safety branch 2 times, most recently from 50d34c0 to 38167d7 Compare July 3, 2026 11:40
use slices.DeleteFunc
ensure draining goroutine returns
don't use metrics tracer to test behavior


Assisted-By: Claude Fable 5
@sukunrt
sukunrt force-pushed the fix/concurrency-safety branch from 38167d7 to 8c6fee9 Compare July 3, 2026 11:53
@sukunrt
sukunrt requested a review from lidel July 8, 2026 06:23
lidel added 2 commits July 8, 2026 18:34
Document what SerializedCertHashes promises, and make the tests check
it. The concurrent test only ever guarded the read lock: with the copy
removed but the lock kept, it passed every run under -race. Its clear()
probe lowers to runtime.memclrHasPointers, which carries no race
instrumentation, so the detector never saw the write. An indexed store
is instrumented, and now kills both mutants.

- godoc: the caller owns the outer slice, the hash bytes are shared
  with the manager and read-only
- TestSerializedCertHashesReturnsOuterCopy fails without the copy on
  any run, no race detector needed
- yield in the reader loop, dropping the concurrent test from 8.0s to
  1.1s at GOMAXPROCS=1 under -race
Make these tests fail loudly instead of hanging. A require failure
inside a synctest bubble unwinds the root goroutine, and the deferred
emitter close then parks on the node lock a stalled emit still holds.
Closing the subscription before asserting turns a timeout hang into an
immediate failure that names the subscription that broke.

- TestEmitLogsErrorOnStall: close, wait, then assert; derive the sleep
  from slowConsumerWarningTimeout rather than a bare 3s
- TestWildcardCloseUnblocksStalledEmit: a goroutine blocked on an
  RWMutex is not durably blocked, so synctest cannot call it a
  deadlock; say so, and send the warning to io.Discard

@lidel lidel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM, good to ship from my side.

@sukunrt fyi pushed two commits on top of your removeSink cleanup that make tests more tight:

  • dffa0f3 test(webtransport): assert cert hash ownership
    The concurrent test only guarded the read lock: clear() on a [][]byte compiles to runtime.memclrHasPointers, which has no race instrumentation, so -race never saw the write. An indexed store does, and now catches a missing clone too. Also brought back an ownership test, scoped to the outer slice that slices.Clone actually copies.

  • 4f6c46c test(eventbus): tighten slow-consumer assertions
    A failed require in a synctest bubble unwinds into the deferred emitter close, which parks on the node lock a stalled emit still holds, so failures hung until -timeout rather than failing. Closing the sub first fixes it. Also yielded in the rotation reader loop: 8s at GOMAXPROCS=1 and 2.2s on a 4 vCPU runner, now 1.1s.

@sukunrt
sukunrt merged commit ec408fc into libp2p:master Jul 9, 2026
9 checks passed
@Sahil-4555
Sahil-4555 deleted the fix/concurrency-safety branch July 10, 2026 03:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants