fix(engine): hijack multiplexed listener conn to stop pool-slot leak on reconnect (#3694)#4214
Open
AlexlaGuardia wants to merge 1 commit into
Open
Conversation
|
@AlexlaGuardia is attempting to deploy a commit to the Hatchet Team on Vercel. A member of the Team first needs to authorize it. |
gregfurman
reviewed
Jun 25, 2026
| return nil, err | ||
| } | ||
| return poolConn.Conn(), nil | ||
| return poolConn.Hijack(), nil |
…k on reconnect The multiplexer's pgxlisten Connect callback acquired a *pgxpool.Conn and returned its raw *pgx.Conn, letting the pool wrapper fall out of scope without Release(). On every reconnect (e.g. server-side idle-session termination) pgxlisten closes the conn, but pgxpool still counts the orphaned slot as acquired, leaking one slot per reconnect until Acquire() blocks at MaxConns. Hijack() transfers ownership out of the pool so closing the conn leaks nothing. Fixes hatchet-dev#3694
da707bf to
7562bc6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3694.
Problem
pkg/repository/multiplexer.go's pgxlistenConnectcallback acquires a*pgxpool.Connand returns its raw*pgx.Conn, letting the pool wrapper fall out of scope withoutRelease():pgxpool keeps counting that slot as acquired. Every time pgxlisten reconnects (server-side
idle_session_timeout, pgbouncerserver_idle_timeout, or an L7 proxy idle kill), itClose()s the raw conn via itsdefer, but the orphaned pool slot is never reclaimed. So the pool leaks one slot per reconnect untilAcquire()blocks atMaxConnsand the control plane wedges.Fix
Return
poolConn.Hijack()instead ofpoolConn.Conn():Hijack()transfers ownership of the underlying connection out of the pool and removes it from pgxpool's accounting, so when pgxlisten closes it on reconnect, nothing leaks. pgxlisten already owns the connection's lifecycle (it closes the conn on every reconnect), which is exactly the caseHijackexists for. One-line behavioral change plus an explanatory comment.Testing
The leak is integration-level: it only manifests across reconnect cycles under server-side idle termination, so it isn't cleanly covered by the existing repository unit tests. I verified the fix by reasoning about pgxpool's ownership semantics (
Hijack()is the documented way to remove a connection from the pool permanently). Happy to add an integration test that drives the reconnect path if you'd like one before merge.AI usage disclosure
Per CONTRIBUTING.md#ai-usage: this fix was implemented with Claude Code (Opus). It traced the leak from the issue through
multiplexer.go, implemented theHijack()change, and reviewed it against the pgxpool API and pgxlisten's connection lifecycle before opening. Same human-directed workflow and disclosure as #4180.