Skip to content

Commit 5dcae5b

Browse files
rdimitrovclaude
andauthored
fix(deploy): bump registry pod memory to match doubled pgxpool size (#1225)
## Summary Follow-up to #1221. That PR doubled pgxpool `MaxConns` from 30 to 60 per pod without bumping the registry pod memory limit. pgxpool keeps per-connection state in the Go process (prepared statement cache, scratch buffers, row iteration state), so memory grew roughly proportionally — peak per-pod usage shifted from ~330 MB to ~450 MB against a 512 MiB limit. ## What surfaced it Pod `tx49f` was OOMKilled at 17:01 UTC on 2026-04-29 (`exitCode 137`, `reason: OOMKilled`). Memory history from Cloud Monitoring confirms the regression: | Pods | Peak 30-min memory | |------|-------------------:| | Pre-#1221 (`MaxConns=30`) | 178–333 MB | | **Post-#1221** (`MaxConns=60`) | **437–449 MB** | | Old limit | **512 MiB** ← right at the edge | Other pod was at 394 MB at the time of inspection — could OOM again. ## Fix `deploy/pkg/k8s/registry.go:217-225`: | | Before | After | |---|-------:|------:| | `requests.memory` | 256 MiB | **512 MiB** | | `limits.memory` | 512 MiB | **1 GiB** | 1 GiB is ~2× the current peak under load — gives room for traffic spikes without immediate OOM risk. Either node has 4–5 GiB free allocatable memory, so the new request fits comfortably. ## Test plan - [x] `go build ./...` clean - [x] `golangci-lint run` clean - [ ] After release + prod promotion: pod memory stabilises around 450 MB peak with no OOMKills - [ ] After release + prod promotion: tx49f restartCount stops climbing (currently at 5) ## Deployment Zero downtime — Deployment template change only, rolling update with `maxSurge: 1, maxUnavailable: 0`. **No PG involvement** (no `max_connections` change, no shared_preload_libraries change). Same shape as a normal v1.7.x image promotion. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 0a97e59 commit 5dcae5b

1 file changed

Lines changed: 10 additions & 2 deletions

File tree

deploy/pkg/k8s/registry.go

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -215,12 +215,20 @@ func DeployMCPRegistry(ctx *pulumi.Context, cluster *providers.ProviderInfo, env
215215
TimeoutSeconds: pulumi.Int(3),
216216
},
217217
Resources: &corev1.ResourceRequirementsArgs{
218+
// Memory was 256Mi/512Mi when pgxpool MaxConns was 30. After
219+
// #1221 doubled MaxConns to 60 per pod, peak per-pod memory
220+
// shifted from ~330MB to ~450MB under scraper load and one
221+
// pod was OOMKilled at 17:01 UTC on 2026-04-29. pgxpool keeps
222+
// per-connection state (prepared statement cache, scratch
223+
// buffers, row iteration state) so memory scales roughly
224+
// linearly with the connection count. Doubled both request
225+
// and limit to match.
218226
Requests: pulumi.StringMap{
219-
"memory": pulumi.String("256Mi"),
227+
"memory": pulumi.String("512Mi"),
220228
"cpu": pulumi.String("100m"),
221229
},
222230
Limits: pulumi.StringMap{
223-
"memory": pulumi.String("512Mi"),
231+
"memory": pulumi.String("1Gi"),
224232
},
225233
},
226234
},

0 commit comments

Comments
 (0)