Skip to content

fix(engine): hijack multiplexed listener conn to stop pool-slot leak on reconnect (#3694)#4214

Open
AlexlaGuardia wants to merge 1 commit into
hatchet-dev:mainfrom
AlexlaGuardia:fix/3694-listener-pool-slot-leak
Open

fix(engine): hijack multiplexed listener conn to stop pool-slot leak on reconnect (#3694)#4214
AlexlaGuardia wants to merge 1 commit into
hatchet-dev:mainfrom
AlexlaGuardia:fix/3694-listener-pool-slot-leak

Conversation

@AlexlaGuardia

Copy link
Copy Markdown

Fixes #3694.

Problem

pkg/repository/multiplexer.go's pgxlisten Connect callback acquires a *pgxpool.Conn and returns its raw *pgx.Conn, letting the pool wrapper fall out of scope without Release():

poolConn, err := m.pool.Acquire(ctx)
if err != nil {
    return nil, err
}
return poolConn.Conn(), nil // *pgxpool.Conn wrapper orphaned, never Release()d

pgxpool keeps counting that slot as acquired. Every time pgxlisten reconnects (server-side idle_session_timeout, pgbouncer server_idle_timeout, or an L7 proxy idle kill), it Close()s the raw conn via its defer, but the orphaned pool slot is never reclaimed. So the pool leaks one slot per reconnect until Acquire() blocks at MaxConns and the control plane wedges.

Fix

Return poolConn.Hijack() instead of poolConn.Conn():

return poolConn.Hijack(), nil

Hijack() transfers ownership of the underlying connection out of the pool and removes it from pgxpool's accounting, so when pgxlisten closes it on reconnect, nothing leaks. pgxlisten already owns the connection's lifecycle (it closes the conn on every reconnect), which is exactly the case Hijack exists for. One-line behavioral change plus an explanatory comment.

Testing

The leak is integration-level: it only manifests across reconnect cycles under server-side idle termination, so it isn't cleanly covered by the existing repository unit tests. I verified the fix by reasoning about pgxpool's ownership semantics (Hijack() is the documented way to remove a connection from the pool permanently). Happy to add an integration test that drives the reconnect path if you'd like one before merge.

AI usage disclosure

Per CONTRIBUTING.md#ai-usage: this fix was implemented with Claude Code (Opus). It traced the leak from the issue through multiplexer.go, implemented the Hijack() change, and reviewed it against the pgxpool API and pgxlisten's connection lifecycle before opening. Same human-directed workflow and disclosure as #4180.

@vercel

vercel Bot commented Jun 16, 2026

Copy link
Copy Markdown

@AlexlaGuardia is attempting to deploy a commit to the Hatchet Team on Vercel.

A member of the Team first needs to authorize it.

@github-actions github-actions Bot added the engine Related to the core Hatchet engine label Jun 16, 2026
return nil, err
}
return poolConn.Conn(), nil
return poolConn.Hijack(), nil

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @mnafees -- wdyt?

…k on reconnect

The multiplexer's pgxlisten Connect callback acquired a *pgxpool.Conn and
returned its raw *pgx.Conn, letting the pool wrapper fall out of scope without
Release(). On every reconnect (e.g. server-side idle-session termination)
pgxlisten closes the conn, but pgxpool still counts the orphaned slot as
acquired, leaking one slot per reconnect until Acquire() blocks at MaxConns.

Hijack() transfers ownership out of the pool so closing the conn leaks nothing.

Fixes hatchet-dev#3694
@AlexlaGuardia AlexlaGuardia force-pushed the fix/3694-listener-pool-slot-leak branch from da707bf to 7562bc6 Compare July 4, 2026 01:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engine Related to the core Hatchet engine

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Multiplexed LISTEN connection leaks pool slots on reconnect under server-side idle connection termination

2 participants