fix(worker): reap deleted backends and stop models that live on a worker

Three related backend-lifecycle defects, all reachable from the same
production incident on a Jetson/Thor worker: a deleted backend's gRPC
process survived ~40 minutes with its directory removed from disk, a later
model load was routed to that orphan and failed with a certifi path pointing
into the deleted directory, and the admin could not stop the model because
the frontend reported it as not loaded.

1. backend.delete orphaned the process it claimed to delete
------------------------------------------------------------
s.processes is keyed by `modelID#replicaIndex` (buildProcessKey), so the
backend name never appeared in a key and was recorded nowhere on the
process. backend.delete resolved its target via isRunning/stopBackend, whose
prefix path only matches a bare *modelID* - a delete keyed on a backend name
resolved to zero keys, the stop silently no-op'd, and the files were removed
out from under a live process.

The install fast path then handed that orphan back out: it returns any live
process for the (model, replica) slot without checking which backend started
it, so a reinstalled variant inherited the deleted backend's port.

- Record backendName on backendProcess, threaded installBackend ->
  startBackend.
- Add resolveProcessKeysForBackend, matching the recorded name and resolving
  alias <-> concrete via ListSystemBackends *before* DeleteBackendFromSystem
  erases the metadata that carries the alias. Alias resolution failure
  degrades to name-only matching so a delete never fails on it.
- backend.stop goes through resolveStopTargets, which accepts a backend
  name, a model name, or an exact modelID#replica key. Its payload field is
  named "backend" but is published with all three meanings: the admin UI
  sends a backend name, UnloadRemoteModel sends a model name, and the
  router's abandoned-load reap (#10948) sends an exact replica key.
  Narrowing it to backend names alone would strand the latter two.
  backend.delete stays strict - its identifier is unambiguously a backend.
- Gate the install fast path on processMatchesBackend so a slot held by a
  different backend is restarted rather than reused. Processes with no
  recorded name (pre-upgrade) are accepted, so rollout does not restart
  every running backend.
- stopBackendExact reports a real stop failure - the process still being
  alive afterwards, which is precisely what finishBackendStop already
  detects to keep the entry and its port reserved - and backend.delete no
  longer replies success when it knew about a process and could not kill it.
  "No process was running" stays a success but is logged, so the orphan case
  is visible rather than silent.

2. /backend/shutdown reported a running model as missing
---------------------------------------------------------
ModelLoader.deleteProcess short-circuits on a miss in this replica's
in-memory store. In distributed mode the authoritative record of "is this
model loaded" is the shared node registry: a frontend replica that never
served the model itself (load balancer picked a peer, or the replica
restarted) has no local entry. The remote unload path that pkg/model
documents ("when ShutdownModel is called for a model with no local process,
UnloadRemoteModel is called") sat behind that short-circuit, unreachable in
exactly the case it exists for. #10865 reworked this function but kept the
short-circuit at the top, so the gap survived that refactor.

- deleteProcess consults the remote unloader on a local-store miss, via a
  shared unloadRemote helper so this branch and the existing
  no-local-process branch both prefer #10865's RemoteModelContextUnloader,
  preserving force propagation across the distributed boundary.
- UnloadRemoteModelContext reports ErrRemoteModelNotLoaded when no node has
  the model; it previously returned nil, making a no-op stop
  indistinguishable from a real one. The converse case (nodes have it, none
  could be stopped) already errors since #10865 joined the per-node
  failures, so that half of the original fix was dropped as redundant.
- Only when the model is absent locally AND cluster-wide does the endpoint
  report not-found, now 404 naming both scopes rather than a bare 500.
- modelNotFoundErr becomes the exported ErrModelNotFound so the HTTP layer
  can map it without string matching; watchdog's identity comparison becomes
  errors.Is.

3. Coverage for the bounded Free() that #10865 shipped untested
----------------------------------------------------------------
The original branch also bounded the pre-stop Free(), but #10865 landed that
fix first (workerBackendFreeTimeout, applied in both stopBackendExact and
handleModelUnload). That production change is therefore DROPPED here as
superseded - master's version is strictly better, since it also releases the
supervisor mutex across the call and keeps the port reserved until
termination completes.

What #10865 did not ship is a test, and the bound is load-bearing: the
router-side reap in #10948 sends backend.stop for an abandoned load, and
against a wedged backend an unbounded Free would swallow that stop before it
reached the process. Nothing failed if the bound regressed.

The spec stands up a real gRPC backend server whose Free handler never
returns - what a Python backend looks like when its single worker thread
(PYTHON_GRPC_MAX_WORKERS=1 on 37 backends) is occupied by a stuck LoadModel.
A stub socket is not sufficient and was tried first: without a completed
HTTP/2 handshake, gRPC's own ~20s connect timeout ends the call, so that
version passed against the very bug it targets. With the connection READY,
only the caller's deadline can end it, so the spec hangs to its 60s limit if
the timeout is removed and passes with it.

Its fixture process is deliberately never started. go-processmanager v0.1.1
writes Process.pid from readPID() without synchronization, so a live process
races its own monitor goroutine under -race - reproducible with a bare
Run()+Stop() and unrelated to this spec. Since
scripts/model-lifecycle-conformance.sh runs this package with -race and is
fail-closed, starting one would turn that gate red on an upstream defect. An
unstarted process still proves the point: the stop is reached and the slot
released, which is exactly what an unbounded Free prevents.

Verified: make lint (new-from-merge-base origin/master) reports 0 issues;
scripts/model-lifecycle-conformance.sh passes all three stages including the
FizzBee liveness check (1458 states, IsLive: true).

Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

Author: mudler

core/http/endpoints/localai/backend_monitor.go

@@ -1,9 +1,15 @@
 package localai
 
 import (
+	"errors"
+	"fmt"
+	"net/http"
+
 	"github.com/labstack/echo/v4"
 	"github.com/mudler/LocalAI/core/schema"
 	"github.com/mudler/LocalAI/core/services/monitoring"
+	"github.com/mudler/LocalAI/pkg/model"
+	"github.com/mudler/xlog"
 )
 
 // BackendMonitorEndpoint returns the status of the specified backend
@@ -48,7 +54,25 @@ func BackendShutdownEndpoint(bm *monitoring.BackendMonitorService) echo.HandlerF
 		if err := c.Bind(input); err != nil {
 			return err
 		}
+		if input.Model == "" {
+			return echo.NewHTTPError(http.StatusBadRequest, "model is required")
+		}
 
-		return bm.ShutdownModel(input.Model)
+		if err := bm.ShutdownModel(input.Model); err != nil {
+			// "Not loaded" is a client-side condition, not a server fault, so
+			// it should not surface as a 500. In distributed mode this branch
+			// used to be reached for models that were running on a worker —
+			// only this replica's local store was empty. The loader now
+			// consults the node registry first, so reaching it means the
+			// model is loaded neither here nor anywhere in the cluster.
+			if errors.Is(err, model.ErrModelNotFound) {
+				xlog.Info("Shutdown requested for a model that is not loaded", "model", input.Model)
+				return echo.NewHTTPError(http.StatusNotFound,
+					fmt.Sprintf("model %q is not loaded on this instance or on any worker node", input.Model))
+			}
+			xlog.Error("Failed to shut down model", "model", input.Model, "error", err)
+			return err
+		}
+		return c.JSON(http.StatusOK, map[string]string{"message": "model stopped"})
 	}
 }

core/services/nodes/unloader.go

@@ -12,6 +12,7 @@ import (
 
 	"github.com/mudler/LocalAI/core/services/galleryop"
 	"github.com/mudler/LocalAI/core/services/messaging"
+	"github.com/mudler/LocalAI/pkg/model"
 	"github.com/mudler/xlog"
 )
 
@@ -91,8 +92,12 @@ func (a *RemoteUnloaderAdapter) UnloadRemoteModelContext(ctx context.Context, mo
 		return fmt.Errorf("finding nodes with model %q: %w", modelName, err)
 	}
 	if len(nodes) == 0 {
+		// Distinguish "no node has it" from "stopped successfully" so the
+		// caller can report the truth to an operator. Returning nil here made
+		// a no-op stop indistinguishable from a real one, and left the loader
+		// unable to tell a cluster-wide miss from a completed unload.
 		xlog.Debug("No remote nodes found with model", "model", modelName)
-		return nil
+		return model.ErrRemoteModelNotLoaded
 	}
 
 	var unloadErr error

core/services/nodes/unloader_test.go

@@ -14,6 +14,7 @@ import (
 
 	"github.com/mudler/LocalAI/core/services/galleryop"
 	"github.com/mudler/LocalAI/core/services/messaging"
+	"github.com/mudler/LocalAI/pkg/model"
 )
 
 // --- Fakes ---
@@ -127,9 +128,13 @@ var _ = Describe("RemoteUnloaderAdapter", func() {
 	})
 
 	Describe("UnloadRemoteModel", func() {
-		It("with no nodes returns nil", func() {
+		It("with no nodes reports the model is not loaded", func() {
+			// Previously returned nil, making "stopped it" and "there was
+			// nothing to stop" indistinguishable. ShutdownModel relies on
+			// this distinction to choose between success and a genuine 404,
+			// so a cluster-wide miss must be reported, not swallowed.
 			locator.nodes = nil
-			Expect(adapter.UnloadRemoteModel("my-model")).To(Succeed())
+			Expect(adapter.UnloadRemoteModel("my-model")).To(MatchError(model.ErrRemoteModelNotLoaded))
 			Expect(mc.published).To(BeEmpty())
 		})
 

core/services/worker/backend_delete_test.go

@@ -0,0 +1,208 @@
+package worker
+
+import (
+	"github.com/mudler/LocalAI/core/gallery"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// Production incident (Jetson/Thor worker): deleting a backend returned HTTP
+// 200 but left its gRPC process alive for ~40 minutes with its directory
+// removed from disk. A later model load was routed to that orphan and failed
+// with a certifi path pointing into the deleted directory.
+//
+// Root cause: s.processes is keyed by `modelID#replicaIndex`
+// (buildProcessKey), so a delete keyed on the *backend* name resolved to no
+// keys at all and the stop silently no-op'd. These specs pin the two lookups
+// that must work by backend name.
+var _ = Describe("Backend deletion reaps the backend's processes", func() {
+	const (
+		concrete    = "cuda13-nvidia-l4t-arm64-longcat-video"
+		development = "cuda13-nvidia-l4t-arm64-longcat-video-development"
+		alias       = "longcat-video"
+	)
+
+	Describe("resolveProcessKeysForBackend", func() {
+		It("finds processes started for a backend even though they are keyed by modelID", func() {
+			s := &backendSupervisor{
+				processes: map[string]*backendProcess{
+					// Started by a model load: key is modelID#replica, and the
+					// backend name survives only in backendName.
+					"LongCat-Video#0": {addr: "127.0.0.1:30232", backendName: concrete},
+					"LongCat-Video#1": {addr: "127.0.0.1:30234", backendName: concrete},
+					"other-model#0":   {addr: "127.0.0.1:30235", backendName: "llama-cpp"},
+				},
+			}
+
+			Expect(s.resolveProcessKeysForBackend(setOf(concrete))).To(
+				ConsistOf("LongCat-Video#0", "LongCat-Video#1"),
+				"a delete keyed on the backend name must reach every process started for it")
+		})
+
+		It("matches through the alias the model config used at install time", func() {
+			// The model config referenced `backend: longcat-video` (the alias),
+			// so the process recorded the alias; the delete request carries the
+			// concrete directory name. Without alias resolution the orphan
+			// survives exactly as it did in production.
+			s := &backendSupervisor{
+				processes: map[string]*backendProcess{
+					"LongCat-Video#0": {addr: "127.0.0.1:30232", backendName: alias},
+				},
+			}
+
+			Expect(s.resolveProcessKeysForBackend(setOf(concrete, alias))).To(
+				ConsistOf("LongCat-Video#0"),
+				"alias and concrete name must resolve to the same process")
+		})
+
+		It("does not reap processes belonging to a different backend", func() {
+			s := &backendSupervisor{
+				processes: map[string]*backendProcess{
+					"LongCat-Video#0": {addr: "127.0.0.1:30233", backendName: development},
+				},
+			}
+
+			Expect(s.resolveProcessKeysForBackend(setOf(concrete))).To(BeEmpty(),
+				"deleting the non-development backend must not kill the -development process")
+		})
+
+		It("still resolves legacy entries that predate backendName via the modelID prefix", func() {
+			// Installs with an empty modelID key the map by the backend name
+			// itself (buildProcessKey falls back). Those must keep working.
+			s := &backendSupervisor{
+				processes: map[string]*backendProcess{
+					concrete + "#0": {addr: "127.0.0.1:30232"},
+				},
+			}
+
+			Expect(s.resolveProcessKeysForBackend(setOf(concrete))).To(ConsistOf(concrete + "#0"))
+		})
+	})
+
+	Describe("resolveStopTargets", func() {
+		// backend.stop is ambiguous by design: the admin backends UI publishes
+		// a BACKEND name, UnloadRemoteModel publishes a MODEL name, and the
+		// router's abandoned-load reap (#10948) publishes an exact
+		// modelID#replica key. Resolving only one of the three silently
+		// strands the others.
+		newSupervisor := func() *backendSupervisor {
+			return &backendSupervisor{
+				processes: map[string]*backendProcess{
+					"LongCat-Video#0": {addr: "127.0.0.1:30232", backendName: concrete},
+				},
+			}
+		}
+
+		It("stops a process addressed by model ID", func() {
+			Expect(newSupervisor().resolveStopTargets("LongCat-Video")).To(ConsistOf("LongCat-Video#0"),
+				"model-name stop (UnloadRemoteModel) must still reach the process")
+		})
+
+		It("stops a process addressed by backend name", func() {
+			Expect(newSupervisor().resolveStopTargets(concrete)).To(ConsistOf("LongCat-Video#0"),
+				"backend-name stop must reach the process too")
+		})
+
+		It("stops a process addressed by an exact replica key", func() {
+			Expect(newSupervisor().resolveStopTargets("LongCat-Video#0")).To(ConsistOf("LongCat-Video#0"),
+				"the router reaps an abandoned load by exact modelID#replica key")
+		})
+
+		It("does not stop unrelated processes", func() {
+			Expect(newSupervisor().resolveStopTargets("some-other-model")).To(BeEmpty())
+		})
+	})
+
+	Describe("processMatchesBackend", func() {
+		It("rejects reusing a process that was started for a different backend", func() {
+			// The install fast path returns the address of any live process
+			// under this processKey. After the concrete backend was deleted and
+			// the -development variant installed, the same model+replica got
+			// the deleted backend's port handed back to it.
+			s := &backendSupervisor{
+				processes: map[string]*backendProcess{
+					"LongCat-Video#0": {addr: "127.0.0.1:30232", backendName: concrete},
+				},
+			}
+
+			Expect(s.processMatchesBackend("LongCat-Video#0", development)).To(BeFalse(),
+				"a process started for the deleted backend must not be reused for another backend")
+		})
+
+		It("accepts reusing a process started for the same backend", func() {
+			s := &backendSupervisor{
+				processes: map[string]*backendProcess{
+					"LongCat-Video#0": {addr: "127.0.0.1:30232", backendName: concrete},
+				},
+			}
+
+			Expect(s.processMatchesBackend("LongCat-Video#0", concrete)).To(BeTrue())
+		})
+
+		It("accepts a process recorded under the alias of the requested backend", func() {
+			s := &backendSupervisor{
+				processes: map[string]*backendProcess{
+					"LongCat-Video#0": {addr: "127.0.0.1:30232", backendName: alias},
+				},
+			}
+
+			Expect(s.processMatchesBackend("LongCat-Video#0", alias)).To(BeTrue())
+		})
+
+		It("accepts legacy entries with no recorded backend name", func() {
+			// Pre-upgrade processes carry no backendName. Treating them as a
+			// mismatch would restart every running backend once on rollout.
+			s := &backendSupervisor{
+				processes: map[string]*backendProcess{
+					"LongCat-Video#0": {addr: "127.0.0.1:30232"},
+				},
+			}
+
+			Expect(s.processMatchesBackend("LongCat-Video#0", concrete)).To(BeTrue())
+		})
+	})
+
+	Describe("backendIdentitySet", func() {
+		It("maps an alias to its concrete backend and back", func() {
+			backends := gallery.SystemBackends{
+				concrete: {Name: concrete, Metadata: &gallery.BackendMetadata{Name: concrete, Alias: alias}},
+				alias:    {Name: alias, Metadata: &gallery.BackendMetadata{Name: concrete, Alias: alias}},
+			}
+
+			Expect(backendIdentitySet(backends, concrete)).To(HaveKey(alias))
+			Expect(backendIdentitySet(backends, alias)).To(HaveKey(concrete))
+		})
+
+		It("falls back to the bare name when the backend is unknown", func() {
+			// A delete must never fail (or over-reach) because the gallery
+			// listing is unavailable or the entry is already gone from disk.
+			Expect(backendIdentitySet(gallery.SystemBackends{}, concrete)).To(
+				Equal(map[string]struct{}{concrete: {}}))
+		})
+
+		It("does not conflate two concrete backends sharing an alias", func() {
+			// Both variants declare alias "longcat-video"; ListSystemBackends
+			// picks one for the alias row. Deleting the non-chosen concrete
+			// must not pull in the chosen one's identity.
+			backends := gallery.SystemBackends{
+				development: {Name: development, Metadata: &gallery.BackendMetadata{Name: development, Alias: alias}},
+				alias:       {Name: alias, Metadata: &gallery.BackendMetadata{Name: development, Alias: alias}},
+			}
+
+			set := backendIdentitySet(backends, development)
+			Expect(set).To(HaveKey(development))
+			Expect(set).ToNot(HaveKey(concrete))
+		})
+	})
+})
+
+// setOf builds the identity set the resolver consumes, keeping the specs
+// readable without repeating the map literal.
+func setOf(names ...string) map[string]struct{} {
+	set := make(map[string]struct{}, len(names))
+	for _, n := range names {
+		set[n] = struct{}{}
+	}
+	return set
+}