Skip to content

Fix close/listen race in PlainNettyTransport that leaks listener ports#475

Open
codexcoder21 wants to merge 1 commit into
libp2p:developfrom
CodexCoder21Organization:fix-plain-netty-close-listen-race-upstream
Open

Fix close/listen race in PlainNettyTransport that leaks listener ports#475
codexcoder21 wants to merge 1 commit into
libp2p:developfrom
CodexCoder21Organization:fix-plain-netty-close-listen-race-upstream

Conversation

@codexcoder21
Copy link
Copy Markdown

Problem

A concurrent close() racing an in-flight listen() on the same transport can leak the listen socket. The OS file descriptor stays open after both calls return, so the next bind to the same port from this JVM fails with BindException: Address already in use — for the lifetime of the JVM.

Root cause

listen() writes the listener channel into the listeners map under synchronized(this@PlainNettyTransport), but AFTER calling listener.bind(addr). At that point the bind task is already on the boss event loop queue, so the port WILL be bound once the event loop drains it. close() reads the same map WITHOUT acquiring the transport monitor.

In the microsecond window between bind() being submitted and the synchronized listeners += ... running, a concurrent close():

  1. observes an empty listeners map and schedules no channel close;
  2. proceeds straight to shutdownGracefully();
  3. — but Netty's event loop still runs the queued bind task before the loop terminates, so the port DOES get bound; and
  4. nothing ever calls close() on the now-orphaned listener channel.

The OS file descriptor stays open for the lifetime of the JVM, and every subsequent bind to the same port from this process fails.

Fix

  • close() snapshots listeners and channels under synchronized(this@PlainNettyTransport), the same monitor that listen() / dial() / registerChannel() use to mutate them.
  • listen() holds that monitor across the entire closed-check → listener.bind(addr)listeners += ... sequence so the map state is never "bind submitted but listener not recorded".
  • registerChannel() performs its closed-check and channels += ch under one acquire so the dial path has the same atomicity.
  • closed is marked @Volatile for the cheap pre-acquire read in the dial fast path.

The reorder is safe because bind() is called on the calling thread and only queues the actual socket bind for the event loop — bind is microseconds of work to submit the task, and the event loop never acquires the transport monitor, so holding it across the bind submission cannot deadlock.

Regression test

PlainNettyTransportConcurrentListenCloseTest launches listen() and close() from separate threads released by a CountDownLatch barrier and verifies the listen port is bindable after both return. Against the unfixed code the test deterministically leaks the port by the 3rd iteration (198/200 iterations fail with BindException: Address already in use); against the fix it passes consistently across 200 iterations.

Test plan

  • ./gradlew spotlessCheck and ./gradlew :libp2p:detekt pass
  • PlainNettyTransportConcurrentListenCloseTest passes consistently
  • Full TCP / WS / PlainNettyTransport* test suites pass
  • Pre-fix the new test deterministically reproduces the leak at iteration 3

Discovery context

The race was caught downstream by a UrlResolver project where many short-lived libp2p hosts get created and torn down on a fixed listen port. On slow / CPU-contended CI it surfaced as "listen port N not released within 2 seconds of host.stop()" with cascading "Address already in use" on every subsequent iteration. The same fix has been validated end-to-end there.

A concurrent `close()` racing an in-flight `listen()` on the same transport
can leak the listen socket — the OS file descriptor stays open after both
calls return, so the next bind to the same port from this JVM fails with
`BindException: Address already in use`.

## Root cause

`listen()` writes the listener channel into the `listeners` map under
`synchronized(this@PlainNettyTransport)`, but AFTER calling
`listener.bind(addr)`. At that point the bind task is already on the boss
event loop queue (so the port WILL be bound once the event loop drains the
queue). `close()` reads the same map WITHOUT acquiring the transport
monitor.

In the microsecond window between `bind()` being submitted and the
synchronized `listeners += ...` running, a concurrent `close()`:

  1. observes an empty `listeners` map and schedules no channel close;
  2. proceeds straight to `shutdownGracefully()`;
  3. — but Netty's event loop still runs the queued bind task before the
     loop terminates (the queue is drained on shutdown), so the port DOES
     get bound; and
  4. nothing ever calls `close()` on the now-orphaned listener channel.

The OS file descriptor stays open for the lifetime of the JVM, and every
subsequent bind to the same port from this process fails.

## Fix

- `close()` snapshots `listeners` and `channels` under
  `synchronized(this@PlainNettyTransport)`, the same monitor that
  `listen()` / `dial()` / `registerChannel()` use to mutate them.
- `listen()` holds that monitor across the entire closed-check ->
  `listener.bind(addr)` -> `listeners += ...` sequence so the map state is
  never "bind submitted but listener not recorded".
- `registerChannel()` performs its `closed`-check and `channels += ch`
  under one acquire so the dial path has the same atomicity.
- `closed` is marked `@Volatile`. The synchronized blocks already give us
  happens-before; the annotation is for the cheap, lock-free reads that
  still occur inside the dial fast-path before entering `registerChannel`.

The reorder honors the existing constraint that `bind()` is called on the
calling thread (no Netty callback into the transport during bind), so
holding the transport monitor across the bind() submission is safe — bind
is microseconds of work to queue the task on the boss event loop, and the
actual socket bind happens on the event loop where this monitor is never
acquired.

## Regression test

`PlainNettyTransportConcurrentListenCloseTest`'s "parallel listen and close
from separate threads" variant launches `listen()` and `close()` on two
threads released by a `CountDownLatch` barrier and verifies the listen
port is bindable after both return. Against the unfixed code the test
deterministically leaks the port by the 3rd iteration (198/200 iterations
failing); against the fix it passes consistently across 200 iterations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant