fix(distributed): scale the remote model-load deadline with checkpoint size#11030
Open
localai-bot wants to merge 1 commit into
Open
fix(distributed): scale the remote model-load deadline with checkpoint size#11030localai-bot wants to merge 1 commit into
localai-bot wants to merge 1 commit into
Conversation
…t size
The gRPC deadline for the remote LoadModel call was a fixed 5m. It starts
only after the backend install and file staging have completed, so it
covers the worker's checkpoint read and pipeline init alone - work whose
duration is proportional to the bytes on disk. A fixed value is therefore
a model-size cliff, not a timeout.
Measured in production: a 70 GB video checkpoint (longcat-video-avatar-1.5)
on an NVIDIA Jetson Thor worker failed reproducibly with
"rpc error: code = DeadlineExceeded" after 953.5s of wall clock. Backend
install plus staging consumed ~11m, then LoadModel got its 5m and expired.
The load never had a chance, and the operator saw only a generic
DeadlineExceeded with no hint that a config value was the cause.
Raising the constant does not fix this. It moves the cliff to the next
larger model - the cluster has to support 600 GB checkpoints - and it makes
a genuinely wedged SMALL model hang for the whole inflated duration before
anyone notices, which is a real regression in failure latency.
So derive the budget from the checkpoint size instead:
budget = 5m + 20s/GiB, capped at 6h
2 GiB -> 5m40s, 70 GiB -> 28m20s, 600 GiB -> 3h25m. The per-GiB rate is
deliberately pessimistic (~54 MB/s of weight read) because the errors are
not symmetric: too long costs only failure latency on a load that was going
to fail anyway, too short is a guaranteed false failure on a healthy load.
The size is measured from the frontend's local model files, over the same
path set stageModelFiles uploads. When those files are not present locally -
a backend handed a bare HuggingFace repo id fetches its own weights on the
worker - there is nothing to measure and the budget stays at today's 5m.
An explicit LOCALAI_NATS_MODEL_LOAD_TIMEOUT still wins outright, in both
directions: a shorter override is honoured, so an operator who wants fast
failure is not silently extended by the heuristic.
The cold-load hold needed widening to match. It extends on staging progress,
but LoadModel reports none, so once the last byte lands the hold expires a
stall window later and would cancel a load still well inside its own budget.
scheduleAndLoad now extends the hold by the load budget plus the staging
margin as it enters the load phase; ModelLoadCeilingFor stays the hold's
starting budget rather than its maximum.
Finally, a deadline that does expire now names the budget, the checkpoint
size it was derived from, and the knob that overrides it, instead of
surfacing a bare "context deadline exceeded".
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Contributor
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
DefaultModelLoadTimeoutwas a fixed5 * time.Minute— the gRPC deadline on the remoteLoadModelcall. That deadline starts only after backend install and file staging complete, so it covers the worker's checkpoint read and pipeline init alone. That work is proportional to the bytes on disk, which makes a fixed value a model-size cliff, not a timeout.Measured in production: a 70 GB video checkpoint (
longcat-video-avatar-1.5) on an NVIDIA Jetson Thor worker failed reproducibly.Backend install + staging consumed ~11m of wall clock, then
LoadModelgot its 5m and expired. The load never had a chance, and the operator saw only a genericDeadlineExceededwith no hint that a config value was the cause.LOCALAI_NATS_MODEL_LOAD_TIMEOUT=30munblocked the cluster — but requiring every operator to discover that after a confusing failure is the bug.Why not just raise the constant
The cluster has to support checkpoints of 600 GB and beyond. A bigger fixed number:
Same defect shape as #11019, which replaced a fixed wall-clock staging margin with a progress-based deadline.
The fix
Derive the budget from the checkpoint's on-disk size:
The per-GiB rate is deliberately pessimistic — it corresponds to reading weights at ~54 MB/s, below what any supported medium sustains — because the two errors are not symmetric: too long costs only failure latency on a load that was going to fail anyway, too short is a guaranteed false failure on a load that was perfectly healthy. The 5m floor is today's default, so small models keep today's fast-failure behaviour.
Where the size comes from
modelPayloadBytessums the frontend's local model files over the same path setstageModelFilesuploads (dedup'd; directories walked). Not available for a backend handed a bare HuggingFace repo id — that path was never materialized locally and the worker fetches its own weights — in which case the budget falls back to today's plain 5m rather than guessing.Override
An explicit
LOCALAI_NATS_MODEL_LOAD_TIMEOUTwins outright, in both directions: a shorter override is honoured verbatim, so an operator who deliberately wants fast failure is not silently extended by the heuristic. Zero (unset) now means "derive";core/application/distributed.gopasses the raw value rather thanModelLoadTimeoutOrDefault()so that distinction survives.Ceiling composition
ModelLoadCeilingForneeded a companion. The cold-load hold extends on staging progress, butLoadModelreports none — so once the last byte lands the hold expires a stall window later and would cancel a load still well inside its own budget (the #11026 shape: work being done with no progress signal). A 70 GB checkpoint's 28m20s budget under a 25m default ceiling is exactly that case.scheduleAndLoadnow callsextendLoadDeadline(ctx, loadTimeout + modelLoadStagingMargin)as it enters the load phase.ModelLoadCeilingForkeeps its signature and meaning as the hold's starting budget rather than its maximum; the 24h absolute cap still bounds everything.Error message
An expired budget now reports the budget, the checkpoint size it was derived from, and the knob that overrides it, instead of a bare
context deadline exceeded. That cost real debugging time on the night this was found.Tests (TDD, behavioural red first)
core/config/model_load_budget_test.goandcore/services/nodes/router_load_budget_test.go(Ginkgo/Gomega). Written against a stub returning the old fixed value, so they failed on behaviour, not on missing symbols:The router specs drive a real
Route()through toLoadModeland assert the deadline the gRPC client actually receives. The 70 GB and 600 GB fixtures are sparse files (Truncate), so they report the right size while occupying no blocks.Verification
go build ./core/... ./pkg/...go test ./core/services/nodes/ ./core/config/make lint0 issues.)Docs updated in the same change (
docs/content/features/distributed-mode.md) per the docs-with-code rule, plus the--model-load-timeoutCLI help text.Not verified: the 70 GB load itself cannot be reproduced locally — the specs prove the budget is granted and not clipped, not that a real 70 GB checkpoint completes within 28m20s on Jetson Thor hardware. The 20s/GiB rate is a deliberately conservative engineering choice, not a measured fit.