Summary
POST /models/apply returns HTTP 200 with a job id and a status URL. If the controller replica that owns the job restarts or is replaced, the status becomes permanently unqueryable — from any replica — and the caller has no way to learn whether the install succeeded, failed, or is still running.
Same class as the Responses API cross-replica gap fixed in #11000, in the gallery op path.
Reproduction
POST /models/apply {"id":"localai@longcat-video-avatar-1.5"}
-> HTTP 200 {"uuid":"92911516-8546-11f1-b61e-e21c92fcd5ba", "status":".../models/jobs/92911516-..."}
(controller pods roll — a routine Flux image update)
GET /models/jobs/92911516-8546-11f1-b61e-e21c92fcd5ba
-> HTTP 500 {"error":{"code":500,"message":"could not find any status for ID"}}
On a cluster where the deployment rolls regularly (image automation, autoscaling, node drain), any install taking longer than the interval between rolls is affected. Model materialization routinely takes tens of minutes, so this is the common case, not an edge case.
Why it matters beyond the missing status
The status is the only signal that an install did anything. During the incident that surfaced this:
- an install was submitted and returned
HTTP 200
- the config on disk was never rewritten (verified by mtime — unchanged from a previous day)
- the job status was gone, so there was nothing to consult
- the operator's only evidence of success was the
HTTP 200, which had been true from the start
The stale model config that this install was meant to replace had been wrong for two days without anyone noticing. A reinstall that silently no-ops while reporting success is precisely how that persists.
Checking the filesystem (does the config exist, what is its mtime) turned out to be the only reliable way to determine the outcome, which is not a reasonable API contract.
Scope
/models/apply and /models/jobs/:uuid/backends/apply, /backends/upgrade/:name and /backends/jobs/:uuid appear to share the pattern — a backend upgrade job submitted in the same session was later unqueryable the same way, though I did not isolate that as carefully.
Possible directions
Not proposing an implementation. The obvious options:
- persist job state so any replica can answer (the cluster already has Postgres;
syncstate.SyncedMap already exists and is used for exactly this kind of cross-replica state)
- or make the outcome discoverable without the job: an install that completed should be evident from the installed-model listing, so a caller could reconcile by polling that instead
The second is weaker — it cannot distinguish "still running" from "failed" — but it is much cheaper and would have prevented the specific confusion here.
Summary
POST /models/applyreturnsHTTP 200with a job id and a status URL. If the controller replica that owns the job restarts or is replaced, the status becomes permanently unqueryable — from any replica — and the caller has no way to learn whether the install succeeded, failed, or is still running.Same class as the Responses API cross-replica gap fixed in #11000, in the gallery op path.
Reproduction
On a cluster where the deployment rolls regularly (image automation, autoscaling, node drain), any install taking longer than the interval between rolls is affected. Model materialization routinely takes tens of minutes, so this is the common case, not an edge case.
Why it matters beyond the missing status
The status is the only signal that an install did anything. During the incident that surfaced this:
HTTP 200HTTP 200, which had been true from the startThe stale model config that this install was meant to replace had been wrong for two days without anyone noticing. A reinstall that silently no-ops while reporting success is precisely how that persists.
Checking the filesystem (does the config exist, what is its mtime) turned out to be the only reliable way to determine the outcome, which is not a reasonable API contract.
Scope
/models/applyand/models/jobs/:uuid/backends/apply,/backends/upgrade/:nameand/backends/jobs/:uuidappear to share the pattern — a backend upgrade job submitted in the same session was later unqueryable the same way, though I did not isolate that as carefully.Possible directions
Not proposing an implementation. The obvious options:
syncstate.SyncedMapalready exists and is used for exactly this kind of cross-replica state)The second is weaker — it cannot distinguish "still running" from "failed" — but it is much cheaper and would have prevented the specific confusion here.