distributed: worker recycles a stopped backend's gRPC port, allowing a silent misroute

[localai-bot]

Summary

stopBackendExact returns a stopped backend's gRPC port to freePorts synchronously, and startBackend pops it for the very next backend. A controller holding a cached NodeModel row for the old host:port can then pass probeHealth against a different backend now bound to that port, and dispatch a model load to it.

Why this is newly reachable

Before the reaping fix, backend.delete never actually stopped a process (it resolved to zero process keys and silently no-op'd), so a delete never recycled a port. Once deletes genuinely reap, they do.

The delete case is the worst one: unlike model.unload, the controller has no idea the address just became invalid, so nothing prompts it to re-probe.

Failure shape

1. Backend A is running for model M on node N at port P; the controller has a NodeModel row pointing at N:P.
2. Backend A is deleted. Its process is now correctly reaped, and P returns to freePorts.
3. Backend B starts on node N and is assigned P.
4. The controller routes a request for M, finds the cached row, probeHealth succeeds — because something is listening on P — and dispatches to backend B.

The probe verifies liveness, not identity, so this is silent. The request goes to the wrong backend rather than failing.

Scope

Narrow: requires a delete immediately followed by an install on the same worker, with an in-flight route still holding the old address. It is also strictly better than the status quo it replaces, which was routing into a backend whose directory had been deleted out from under it (observed in production: a certifi path resolving into a removed directory).

But it converts a loud failure into a potentially silent misroute, which is worth closing deliberately rather than leaving implicit.

Options

1. Quarantine freed ports for a grace period before returning them to freePorts. Simple, narrows the window, does not eliminate it.
2. Verify backend identity in probeHealth rather than mere liveness. Fixes the class of bug rather than the window, and would also catch any other stale-address path. Larger change.

Option 2 is the more complete fix; option 1 is the cheaper one.

Related

• Reaping fix: deleting a backend previously orphaned its process (the change that makes this reachable).
• Deliberately deferred alternative: BackendDeleteReply could return the stopped process keys so the controller drops matching NodeModel rows eagerly rather than waiting for the next probeHealth. That was skipped because it adds a message-schema change and cross-version compat surface, but it would also mitigate this.

Found while debugging a distributed LongCat-Video load failure on an ARM64 L4T worker.

[localai-bot]

Investigation update — the scope is wider than described above, and the approach this issue suggested as the more complete fix turns out to be the wrong one.

Upgrade has the identical gap

upgradeBackend (core/services/worker/install.go:186-200) force-stops every process for the backend, and DistributedBackendManager.UpgradeBackend (core/services/nodes/managers_distributed.go:536+) never removes the corresponding NodeModel rows — exactly like delete.

The cleanest framing for a reviewer is the contrast: UnloadModel (core/services/nodes/router.go:1550-1560) calls RemoveAllNodeModelReplicas immediately after StopBackend. Delete and upgrade are the outliers, not a novel case.

The port is released after termination, not during

finishBackendStop (core/services/worker/supervisor.go:437-467) does hold the port reserved while the process dies, but on confirmation it does delete(s.processes, key) then s.freePorts = append(s.freePorts, bp.port). startBackend pops LIFO (supervisor.go:131-137), so the next backend on that worker takes P. The reservation covers the kill window; the window that matters is the one after it, because the controller-side row survives indefinitely.

Three recycling sites are reachable while a controller row is live: finishBackendStop (supervisor.go:458), releaseBackendStart (supervisor.go:242), and startBackend's died-unexpectedly cleanup (supervisor.go:124).

Option 2 (identity in probeHealth) is rejected — with reasoning, so it isn't rediscovered

1. No identity exists to assert on. Health returns Reply (backend/backend.proto:337-346: message/tokens/timings/audio/logprobs) and Status returns StatusResponse (:682-691: state + memory). Neither carries a name, model or PID. HealthMessage is empty (:262), so there is nowhere to send an expected identity either. It would need a proto change plus an implementation in 36 Python, 4 C++ and 27 Go backends.
2. It would be fail-open during rolling upgrades — the controller must treat an empty identity as "allow", so the bug persists for every backend not yet updated.
3. Decisive: it would never execute in the failure window. probeHealth short-circuits on probeCache (core/services/nodes/probe_cache.go:75-92, keyed nodeID|addr, 30s TTL). In this exact sequence the address was probed successfully while the old backend was alive, so for up to 30s after the new one binds P, DoOrCached returns true with no round-trip. Comparing identity on every call would defeat that cache, which exists to fix a real queue-stalling bug (llama.cpp serializes HealthCheck against active Predict).

Also checked and rejected: reusing the gRPC bearer token (LOCALAI_GRPC_AUTH_TOKEN) as an identity. It is set process-globally by the worker via os.Setenv (core/services/worker/worker.go:195), StartProcess has no per-process env parameter (pkg/model/process.go:253-268), only three files under backend/ honour it, and the controller would need a new NodeModel column to carry it.

The stale row is permanent, not transient

HealthMonitor.perModelHealthCheck (core/services/nodes/health.go:181-227) is also pure liveness and needs 3 consecutive misses at a 15s interval (~45s). If the port has been rebound it never fires at all, so nothing eventually cleans this up.

Direction being implemented

#10954 (return the stopped process keys so the controller drops rows eagerly) turns out to be much cheaper than its deferral implied — the adapter already holds registry ModelLocator and RemoveNodeModel already exists, so no new interface method, no DB migration, no proto change, and all three call sites funnel through the adapter. A short quarantine on freePorts closes the remaining NATS round-trip window as an interlock.

Deliberately not tuning that quarantine to the health-check cadence: it is operator-configurable and DisablePerModelHealthCheck can switch the reaper off entirely, so coupling a worker-local constant to it would be fragile.

Separate pre-existing issue noticed

nextPort grows monotonically from effectiveBasePort() with no upper bound — nothing checks 65535. Unrelated to this bug but adjacent, and a quarantine mildly increases port consumption. Worth its own issue.

[localai-bot]

Status: two PRs cover most of this, one residual keeps it open.

Covered by #10956 — delete and upgrade paths

The worker reports StoppedProcessKeys on delete and upgrade; the controller drops the matching NodeModel rows eagerly via the adapter it already had. Plus a 15s port quarantine as an interlock for the NATS round-trip window between the worker freeing the port and the controller dropping the row.

Covered by #10968 — the unexpected-death path

That path had no reply to carry stopped keys, so a stale row could survive it. It turned out to be masked rather than benign: startBackend's died-unexpectedly branch never released the port at all, so it was never rebound and the failure was loud rather than silent. Fixing that leak (required for #10961) is what would have made this live.

Closed structurally with per-key port affinity: a released port is offered back only to the process key that last held it. Process keys (modelID#replica) and NodeModel rows (nodeID, modelName, replicaIndex) are isomorphic, so a port that can only be re-bound by its previous owner can only ever be named by that owner's row, which its re-registration overwrites. Misrouting to a different model becomes impossible by construction rather than by racing a timer.

The residual — why this stays open

Affinity expires after 5 minutes. It has to: ownership held forever would make every distinct model a worker has ever served consume a port permanently, so the allocator would climb on distinct-key count rather than concurrency and steal on every allocation while advising the operator to raise a ceiling that is not the constraint.

The 5-minute window is set well above the per-model health reaper (~45s at the default cadence), which is what removes a stale row on the death path.

But that reaper can be switched off entirely via DisablePerModelHealthCheck. In that configuration:

1. a backend dies unexpectedly — no reply, so no eager row removal on this path
2. nothing reaps the stale NodeModel row, because the reaper is disabled
3. affinity lapses after 5 minutes and the port becomes genuinely unowned
4. another key takes it, and the stale row now names a live listener belonging to a different backend

probeHealth verifies liveness rather than identity, so the request is silently misrouted — the original bug, in a narrower configuration.

It needs the reaper disabled and a crash death and an allocation past the expiry, so it is narrow. But it is a genuine gap rather than a theoretical one, and "fixed except when a config flag is set" is worth naming rather than discovering later.

What would actually close it

Eager row removal on the death path, which means an unsolicited worker → controller event. That was evaluated and deliberately deferred: there is no persistent controller-side subscriber to node-published subjects to hang it on (the only Subscribe in unloader.go is a per-op progress subscription torn down after each call), so it needs a new subject, lifecycle wiring, and cross-version compat surface — for a crash-only path.

Worth revisiting if the crash path proves common in practice, or alongside #10954 if that gets picked up, since both want the same "worker tells the controller what it stopped" channel.

[localai-bot]

Investigation settled the failure mode definitively, and it is worse than described above: a misrouted request returns a silently wrong answer from the wrong model, and persists. It is not a self-limiting errored request.

I had hypothesised that InFlightTrackingClient.reconcile (core/services/nodes/inflight.go:88-104) would self-heal this — it drops the replica row on IsModelNotLoaded, on the request path, independent of the health reaper. That hypothesis is wrong, for a structural reason.

The predict path carries no model identity

PredictOptions (backend/backend.proto:265-318) has 40+ fields — Prompt, Seed, Tokens, Messages, Tools, Metadata — and no model name or ID. Model identity exists only in ModelOptions.Model (backend.proto:353), which belongs to the LoadModel RPC.

Reply (backend.proto:337-346) carries none either, so a response-side check is impossible without a proto change. HealthMessage is empty (:262) and StatusResponse (:682-691) carries only state + memory — no gRPC surface exposes which model a process has loaded, which is also why probeHealth has nothing to compare against.

IsModelNotLoaded means "nothing loaded", not "wrong model"

IsModelNotLoaded (pkg/grpc/grpcerrors/errors.go:27-34) matches FailedPrecondition or the substring model not loaded. Every producer guards a nil/empty loaded-model handle, never an identity mismatch:

• backend/go/parakeet-cpp/goparakeetcpp.go:295-297 — if p.ctxPtr == 0
• backend/go/cloud-proxy/proxy.go:147-150 — if cfg == nil
• backend/python/vllm/backend.py:392-395 — if not hasattr(self,'llm') or self.llm is None
• backend/python/transformers/backend.py:295, backend/python/insightface/backend.py:111

A recycled port hosting a different, successfully loaded model fails none of these. reconcile never runs; the stale row is never dropped.

Every backend family serves blind

• Shared Go server (pkg/grpc/server.go:30-33): struct is { pb.UnimplementedBackendServer; llm AIModel } — single model. Predict (:66-76) calls s.llm.Predict(in) with zero validation, and could not validate if it wanted to, because in has no model field.
• llama.cpp (backend/cpp/llama-cpp/grpc-server.cpp:2183-2188): if (params_base.model.path.empty()) — an "is a model loaded" check on its own state. A process with a different GGUF has a non-empty path and generates.
• Python (vLLM, transformers, diffusers, sglang, insightface, tinygrad): same shape — the only FAILED_PRECONDITION is the self.llm is None check.

Backend processes are single-model

Worker-side the process key is buildProcessKey(req.ModelID, req.Backend, req.ReplicaIndex) (core/services/worker/install.go:64), stored in s.processes[key] (supervisor.go:95-99), with ports recycled LIFO from s.freePorts (supervisor.go:78-84). Server-side there is one llm field. So "wrong backend" means precisely "a live process holding a different model", and it answers.

Nothing in the request path catches it

SmartRouter.Route's cached-hit path (core/services/nodes/router.go:464-508): FindAndLockNodeWithModel → probeHealth (liveness only, :1529-1540) → nodeMatchesScheduling → buildClientForAddr → return. No LoadModel is issued. buildClientForAddr (:1092-1100) wraps only with FileStagingClient. ModelRouterAdapter.Route sets opts.Model, but opts is consumed only by the cold-load path (:297) and discarded on a cached hit.

Identity crosses the wire exactly once: client.LoadModel at router.go:297, cold load only.

One accidental exception, which does not rescue it

TTSRequest (backend.proto:585-587) and SoundGenerationRequest (:653-655) do carry a per-request model string (populated at core/backend/tts.go:30, core/backend/soundgeneration.go:66), and backend/go/piper/piper.go:29-31 passes it straight to synthesis, ignoring its load-time state — so a misrouted piper request happens to produce correct audio. That is accidental statelessness in one backend, not validation. It does not apply to LLM, embedding, image, or transcription paths, and TTS backends that load a voice at LoadModel and ignore request.model would still return the wrong voice.

Consequence

#10968's per-key port affinity remains the right stopgap and closes the common paths, but this class of bug cannot be closed by port hygiene alone — a stale row over a re-bound port will always be indistinguishable from a valid one while liveness is the only check.

A durable fix needs a proto change, in one of two directions:

1. expose a model identity in StatusResponse (or a lightweight new RPC) so probeHealth verifies identity rather than liveness; or
2. add a model field to PredictOptions so backends can reject a mismatch themselves.

(1) also has to contend with probeHealth short-circuiting on probeCache (core/services/nodes/probe_cache.go:75-92, 30s TTL keyed nodeID|addr), which would skip the verification during exactly the window that matters — so it needs the cache keyed on identity too, or bypassed for this check.

Keeping this issue open on that basis.