fix(worker): bound the gRPC port allocator and stop leaking dead backends' ports

[localai-bot]

Fixes two coupled defects in the worker's gRPC port allocator. They ship together because fixing the first without the second converts a benign resource leak into a silent misroute.

The port leak is the real driver of #10961

#10961 as filed estimates the allocator needs ~15,000 concurrent-peak allocations to reach 65535 — effectively unreachable. That estimate is wrong, because of a defect the issue does not mention: startBackend's "process died unexpectedly" branch deleted the process map entry without releasing its port at all. Not to freePorts, not to quarantine. Leaked outright — a fourth removal path against three release paths.

A crash-looping backend therefore leaks one port per restart. A backend dying every 30s burns ~2,880 ports/day and walks 50051 → 65535 in about five days. That is a materially different severity from "concurrent-peak growth", and it is the realistic route to exhaustion on a long-lived worker.

Why #10952's residual had to be fixed in the same change

The unexpected-death path is the one teardown path with no request/reply to carry StoppedProcessKeys back to the controller (the eager row removal added in #10956). So a stale NodeModel row can survive it.

Today that is harmless only because the port is leaked — never rebound, so probes fail and requests fail loudly. Fixing the leak is exactly what makes it live: once the port is recycled, a stale row resolves to a live listener belonging to a different backend, and probeHealth verifies liveness rather than identity, so the request is silently misrouted.

Fixes

Bounded allocator. Explicit [basePort, LOCALAI_GRPC_MAX_PORT] range. Exhaustion returns ErrNoFreePort naming the range, the live backend count, the quarantined count, and the knob to raise — instead of an unbindable integer surfacing as an opaque "backend won't start":

no free gRPC port in range: 50051-50150 is fully consumed by 100 running
backend(s) and 12 port(s) still in quarantine; raise LOCALAI_GRPC_MAX_PORT to
widen the range

Out-of-range config degrades rather than wedges: above 65535 clamps, below the base port falls back to the full range. Both spec'd, because an inverted range would otherwise take every backend start down with it.

Per-key port affinity. A released port is offered back to the process key that last held it before any other key. Process keys (modelID#replica) and NodeModel rows (nodeID, modelName, replicaIndex) are isomorphic, so a port that can only be re-bound by its previous owner can only ever be named by that owner's row — which that key's re-registration overwrites. Misrouting to a different model becomes impossible structurally, rather than by racing a quarantine timer.

Affinity is a preference, not a reservation: under range pressure an owned port is stolen with a warning, because a guaranteed outage is worse than a rare misroute window on a port long out of quarantine. Claiming a port evicts the previous owner's entry, keeping ownership injective over ports, so the affinity map can never exceed the range width regardless of key churn.

Affinity expires after 5 minutes. Ownership held forever would make every distinct model a worker has ever served consume a port permanently — the allocator would climb on distinct-key count rather than concurrency, then steal on every allocation while advising the operator to raise a ceiling that is not the constraint. With expiry, reaching the steal branch genuinely means the worker is out of concurrent capacity, so the warning's advice is correct when it fires.

Known residual — why this does not close #10952

The 5-minute window is set well above the per-model health reaper (~45s at the default cadence), which is what removes a stale row on the death path.

That reaper can be switched off entirely via DisablePerModelHealthCheck. In that configuration a crash death leaves a row nothing removes (eager removal does not apply to this path), affinity lapses after 5 minutes, and the port becomes reusable by another key — reopening the misroute window.

Narrow: it needs the reaper disabled, a crash death, and a subsequent allocation past the expiry. But it is a real gap, so #10952 stays open rather than being auto-closed here. The honest closure for that configuration is eager row removal on the death path, which needs a controller-side subscriber for an unsolicited worker event — deliberately out of scope here.

Alternatives rejected

• Unsolicited "process died" event. No persistent controller-side subscriber to node-published subjects exists to hang it on (the only Subscribe in unloader.go is a per-op progress sub torn down after each call). A new subject plus lifecycle wiring plus cross-version compat surface, for a crash-only path.
• Quarantine plus documentation. Not defensible once the leak is fixed: 15s against a ~45s reaper that can be disabled is a shorter window on a permanent stale row, not a mitigation.
• Identity-verifying probeHealth (distributed: worker recycles a stopped backend's gRPC port, allowing a silent misroute #10952's own Option 2). StatusResponse carries only state and memory — no model identity, and HealthMessage is empty, so there is nowhere to send an expected identity either. Needs a proto change, adoption across ~60 backends, and an extra RPC per probe. Worse, probeHealth short-circuits on a 30s probeCache keyed nodeID|addr, so in the exact failure window the check would not execute at all.

Notes for reviewers

• allocatePort's signature changed. Grepped the whole repo including tests/e2e/: referenced only within core/services/worker. No e2e spec pins it.
• One pre-existing spec changed meaning deliberately: "returns the port to the free pool once the quarantine expires" assumed a global pool; under affinity the port returns to its own key. Re-expressed as "returns the port to its own backend" — same invariant (the port is not lost), correct owner. Flagged explicitly because it is a test edited to pass; what independently guards reachability now is the steal-under-pressure spec and the crash-loop spec.
• LOCALAI_GRPC_MAX_PORT is new user-visible surface, documented in docs/content/features/distributed-mode.md with a "Backend gRPC port range" section explaining that range width caps concurrent backends and that the quarantine needs headroom above true concurrency.

Closes #10961. Advances #10952 — see the residual above.

---

🤖 Generated with Claude Code