fix(model-artifacts): persist companion artifacts so remote workers get the base_model option#11075
Merged
Merged
Conversation
A managed model can declare companion artifacts (LongCat-Video-Avatar-1.5
pulls its tokenizer, text encoder and VAE from the separate LongCat-Video
base repo via a target: companion artifact). preloadOne resolves the whole
set in memory, but the binding written back to disk carried only the
primary: persistArtifactBinding marshalled []Spec{result.Spec} and replaced
the entire artifacts: list with it, silently dropping every companion.
In a single process the loss is invisible because the in-memory config keeps
the companion. It bites on the next controller restart: the config reloads
from the mangled file with the primary alone, so withCompanionArtifactOptions
finds no resolved companion and synthesizes no base_model option. The remote
longcat-video backend then never receives base_model, falls back to
BASE_MODEL_ID and downloads the repo itself ("Downloading required files for
meituan-longcat/LongCat-Video"), failing the load with "base_model must point
to a LongCat-Video checkpoint".
This is why an explicit base_model:<path> added to the config options works
where the managed companion does not: an explicit option lives in options:,
which is never rewritten, while the managed companion lives in artifacts:,
which the binding overwrote.
Persist the full resolved set (primary + every companion), and widen
bindingNeedsPersistence to compare the whole artifact list so a companion
resolving for the first time still triggers a write. The single-node path is
unaffected: there the in-memory config already carried the companion, and the
staging/ModelPath resolution for a remote worker (nested per-model staged
root, #10949) is unchanged and already correct once the option is generated.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
Distributed loads of a model with a companion artifact (e.g.
longcat-video-avatar-1.5, whosebase_modelcompanion points atmeituan-longcat/LongCat-Video) fail on the remote worker with:and the backend log shows it fetching the base repo itself instead of using the staged companion:
Root cause
preloadOneresolves the full artifact set (primary + every companion) in memory, but the binding written back to disk carried only the primary:setMappingValueoverwrites the entireartifacts:list, so everytarget: companionentry was silently dropped from the config file.In a single process this is invisible (the in-memory config still holds the companion). It bites on the next controller restart: the config reloads from the mangled file with the primary alone, so
withCompanionArtifactOptionsfinds no resolved companion and synthesizes nobase_modeloption. The remotelongcat-videobackend never receivesbase_model, falls back toBASE_MODEL_ID, and downloads the repo itself — failing validation.This also explains the discriminating observation from live testing: adding an explicit
base_model:<path>to the model'soptions:works, while the managed companion does not. Explicit options live inoptions:, which the binding never rewrites; the managed companion lives inartifacts:, which the binding overwrote.What this is NOT
The companion path nesting (bare
/models/.artifacts/...vs per-model staged/models/longcat-video-avatar-1.5/.artifacts/...) is already handled by #10949: staging derives the workerModelPathas the nested per-model staged root, and the relative companion option resolves under it (router_companionstage_test.gopins this). Rewriting the option with a per-model prefix would double the nesting and break it, so staging is left untouched. Once the option is generated again (this fix), the existing staging/resolution path works.The fix
persistArtifactBindingnow writes the full resolved artifact set (primary + all companions).bindingNeedsPersistencecompares the whole artifact list, so a companion resolving for the first time still triggers a write.Single-node is unaffected (the in-memory config already carried the companion). The gallery-install path (
bindPrimaryArtifact) already persisted the full list; only the preload write-back was lossy.Tests (TDD, Ginkgo/Gomega)
keeps every resolved artifact in the persisted file across a reload(models a controller restart with a fresh loader): red before, green after.persistArtifactBindingunit spec proving both primary and companion survive the write.Verification (exit codes read separately)
go build ./core/... ./pkg/...→ 0go test ./core/backend/→ 0go test ./core/services/nodes/→ 0 (196s)go test ./core/config/→ 0golangci-lint run --new-from-merge-base=origin/master ./core/config/→ 0This is the last blocker in the distributed video-model bring-up (after #10986, #11029, #11030, #11053, #11071): the backend now receives the managed
base_modelcompanion and resolves it from the staged snapshot instead of downloading it.🤖 Generated with Claude Code
https://claude.ai/code/session_0156CbKpDh8qbAYzJZJDZ2yk