fix(distributed): make the cold-load hold scale with progress, not wall-clock

[localai-bot]

The failure

A 70 GB video checkpoint (longcat-video-avatar-1.5) could not be loaded on a distributed cluster. Measured in production:

POST /video   start 11:10:48 -> end 11:35:48
HTTP 500 in 1499.980134s          (= 1500s = exactly 25m00s)

error: ... staging model files for node nvidia-thor: staging model file:
  staging .../base_model_int8/quantized_model-00003-of-00004.safetensors:
  uploading ... failed after 1 attempts within 1h0m0s budget

Staging was healthy and steady the whole time: 26 of 57 files, 39 GB transferred at a sustained ~26 MB/s, zero errors, no stalls. It did not wedge. It was killed by a timer, at exactly the minModelLoadCeiling value.

Root cause

ModelLoadCeilingFor in core/services/nodes/router.go bounds how long one cold load may hold the per-model advisory lock. It covers node selection, backend install, file staging, and the remote LoadModel. Install and load have their own budgets; staging was covered only by a fixed 5-minute margin:

const modelLoadStagingMargin = 5 * time.Minute
const minModelLoadCeiling    = 25 * time.Minute

Staging duration is bytes over available bandwidth, not a constant. 70 GB at 26 MB/s needs ~45 minutes against a 25-minute ceiling, so this failure is deterministic for any sufficiently large model, not a flake.

Verified call path: Route() builds loadCtx with the ceiling, takes the advisory lock, then scheduleAndLoad -> stageModelFiles -> fileStager.EnsureRemote, whose resumeCtx is a child of that context. The ceiling is the binding constraint — the parent's expiry propagates DeadlineExceeded into the upload, which is why the error reads the way it does.

That also explains the corroborating incoherence in the same message: the upload carries a 1h retry budget nested inside the 25m ceiling. The inner budget is unreachable, and failed after 1 attempts shows the retry never got a second try before the parent died.

Why not just raise the constant

The deployment target is checkpoints of 600 GB and larger. A bigger fixed number just moves the cliff to the next model size and we rediscover this later.

The fix: a progress-based deadline

The ceiling's own comment states its purpose — so "a wedged worker can never pin the lock indefinitely". Progress, not elapsed time, is what distinguishes a wedged worker from a large one. The hold now extends whenever the transfer reports bytes and expires a stall window after they stop:

• a large model transferring fine -> continues, for hours if needed
• a worker that died mid-transfer -> still fails within the stall window

Design decisions

What counts as progress, and where it is observed. Byte-level movement on the transfer, via the existing staging progress callback (withStagingCallback), which both the HTTP and S3 stagers already drive. Per-file completion would be far too coarse: a single 600 GB shard would be indistinguishable from a stall for hours. The observation point is the upload body reader, which the transport only reads as fast as it can write to the socket, so it reflects genuine end-to-end movement rather than local disk reads. Observation is coarsened to one timer touch per stall/20 so the per-read (~32 KB) callback stays cheap.

Stall window: 5 minutes. The old modelLoadStagingMargin, reused as a rolling window rather than a one-shot total, so it bounds silence instead of bounding transfer size. It is well beyond any legitimate gap in a healthy stream (a worker fsyncing a multi-GB shard, a LAN blip inside the resumable-upload retry loop, a GC pause) while still releasing the advisory lock fast enough that a dead worker is not felt as an outage. Clamped to the base ceiling so an operator who deliberately tightens the ceiling for fast failure does not silently get it widened back out.

Absolute cap: 24h. The advisory lock is held throughout, so a progress-based hold still has to be bounded against a peer that trickles a few bytes per window forever. 600 GB at the measured 26 MB/s is ~6.5h, and ~17h at a pessimistic 10 MB/s, so 24h sits far above any legitimate transfer while still guaranteeing the lock is eventually released. Progress can never move it.

Operator overrides stay coherent. The base budget is unchanged and still derived from the install and load timeouts, and still covers the steps that report no progress (node selection, install, LoadModel). LOCALAI_NATS_MODEL_LOAD_TIMEOUT therefore behaves exactly as before. ModelLoadCeilingFor keeps its signature and all its existing specs.

The 1h-inside-25m layering. The resumable upload now adopts the caller's deadline when there is one, and applies its fixed budget only when nothing above bounded it. This removes the misleading error text and stops a fixed 1h from reintroducing the size cliff under the now-extendable parent.

The expiry reports context.DeadlineExceeded rather than Canceled, because the upload retry loop branches on that distinction. Deadline() reports the absolute cap, not the rolling expiry, so gRPC children (LoadModel) budget correctly instead of inheriting a few milliseconds.

Tests

TDD, Ginkgo/Gomega, in core/services/nodes/router_staging_deadline_test.go. Red first, on behaviour — both failures were against existing symbols only:

[FAILED] Expected success, but got an error:
    staging model files for node nvidia-thor: staging model file: context deadline exceeded

[FAILED] progress must extend the hold past the base ceiling
Expected <time.Duration>: 201152186 to be > <time.Duration>: 500000000

The first reproduces the production error text; the second shows the hold ending at the base ceiling (201ms) instead of extending. Specs cover: a slow-but-progressing transfer surviving past the base ceiling, a transfer that progresses then goes silent still being killed promptly, the absolute cap bounding a forever-trickling peer, the stall window not outliving a tight ceiling, and Deadline() exposing the cap.

Verification

Command	Exit
go build ./core/... ./pkg/...	0
go test ./core/services/nodes/ (full suite)	0
go test ./core/application/...	0
make lint	0 (0 issues.)

Not verified: the 70 GB cluster transfer itself was not reproduced locally — that requires the multi-node cluster and the real checkpoint. The specs simulate the transfer shape (scaled-down durations) at the router level against a fake stager; the production evidence above is from the incident, not from this branch.

Context

Successor to #10968, where a hardcoded 5-minute LoadModel gRPC timeout was replaced by this derived ceiling. Fixing the inner timeout exposed the outer ceiling as the new binding constraint. The framing for reviewers: the ceiling must scale with work, not wall-clock.

Docs updated in docs/content/features/distributed-mode.md per the docs-with-code rule.