fix: force aiohttp's ThreadedResolver to stop the c-ares channel leak#12985
Merged
Conversation
Since aiodns 3.2, aiohttp selects the aiodns/pycares AsyncResolver as its default resolver whenever the package is importable. Every ephemeral ClientSession/TCPConnector then creates a c-ares channel that is never destroyed, which on long-running services accumulates native heap memory (~0.5 MB/h observed on an idle production agent via eBPF memleak) and one leaked /dev/urandom fd per channel (~85,600 open fds on 16.5-day-old agent workers, converging on fd exhaustion). Same root-cause family as aio-libs/aiodns#191. aiohttp offers no configuration knob for the resolver choice and passing resolver= at every connector creation site is easy to miss, so add force_threaded_dns_resolver() to ai.backend.common.networking and call it once at each service entrypoint before any connector is created (the override survives the aiotools worker fork). An A/B test on a production agent confirmed this eliminates both symptoms completely. This intentionally leaves the aiodns/pycares packages installed so the change is backportable without touching release-branch lockfiles; the dependency removal itself lands separately on main (#12979). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
github-actions
Bot
added
size:M
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fregataa
approved these changes
Jul 21, 2026
This was referenced Jul 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Backportable half of the c-ares leak fix (the dependency removal itself lands separately on main in #12979 — split per review discussion so release branches don't need lockfile surgery).
Since aiodns 3.2, aiohttp picks the aiodns/pycares AsyncResolver as its default resolver whenever the package is importable. Every ephemeral
ClientSession/TCPConnectorthen creates a c-ares channel that is never destroyed. On long-running services this accumulates:memleak:ares_malloc_zerooutstanding allocations strictly linear, with zero on-the-wire DNS)/dev/urandomfd per channel — three 16.5-day-old agent workers each held ~85,600 open urandom fds (~216 channels/hour, uniform across idle/busy nodes), converging on fd exhaustionSame root-cause family as aio-libs/aiodns#191 (aiodns ≥ 3.3 additionally exhausts inotify watches, so upgrading aiodns worsens it).
Why a code change at all
Our own code barely touches aiodns — aiohttp picks it up implicitly just because it is installed. aiohttp has no configuration knob for the resolver choice, and injecting
resolver=at every connector creation site is easy to miss. The only lockfile-free way to stop the selection is overriding the module-level defaults once at startup.Changes
ai.backend.common.networking.force_threaded_dns_resolver()— overridesaiohttp.resolver.DefaultResolverandaiohttp.connector.DefaultResolverwithThreadedResolver, with a docstring explaining whyTCPConnectoractually receives aThreadedResolverVerification
pants testgreen for the new tests;pants fmt/lint/checkgreen across all touched files🤖 Generated with Claude Code