distributed: return stopped process keys in BackendDeleteReply so the controller drops stale NodeModel rows eagerly

[localai-bot]

Summary

When a backend is deleted on a worker, the controller learns that its NodeModel rows are stale only lazily — the next probeHealth against the recorded host:port fails, and the row is then dropped and reloaded. BackendDeleteReply could instead return the process keys that were actually stopped, letting the controller drop matching rows immediately.

Why it was deferred

This was considered and deliberately skipped while fixing the underlying bug (deleting a backend previously left its process running entirely, so nothing was reaped and the rows were stale for a different reason). Reasons for skipping:

• it adds a message-schema change to BackendDeleteReply, and therefore a cross-version compatibility surface between controller and worker
• probe-based self-heal is correct once the process is genuinely dead, which the reaping fix ensures — so this is an optimisation of latency, not a correctness fix

Why it may still be worth doing

Two things make it more attractive than "just an optimisation":

1. It narrows the port-recycling window (see distributed: worker recycles a stopped backend's gRPC port, allowing a silent misroute #10952). Today the controller can hold a stale row pointing at a host:port that a different backend has since been assigned, and probeHealth verifies liveness rather than identity — so the probe succeeds and the request is misrouted. Dropping the rows eagerly at delete time removes the stale row before that can happen.
2. It removes a window where the controller can route to an address it already had reason to know is dead.

Considerations

• Needs to degrade gracefully when the worker is older and does not populate the new field — the controller must keep the probe-based path as a fallback, not depend on the reply.
• Worth deciding whether the same treatment applies to backend.stop replies, not just backend.delete.

Related

• distributed: worker recycles a stopped backend's gRPC port, allowing a silent misroute #10952 — port recycling misroute, which this would partially mitigate
• The reaping fix that made delete actually stop processes (and therefore made stale-row timing matter)

[localai-org-maint-bot]

Already implemented by #10956 and present on current master. The merged change reports stopped process keys from BackendDeleteReply, uses an explicit compatibility flag for older workers, and eagerly removes the matching per-replica NodeModel rows while retaining probe-based cleanup as the fallback.