fix(vl): strip model.visual.* in qwen2_vl/qwen3_vl/qwen3_vl_moe sanitize#1473
Open
Jonathangadeaharder wants to merge 2 commits into
Open
Conversation
9eb5800 to
7d24881
Compare
HuggingFace ForConditionalGeneration VL checkpoints (Qwen3_5MoeForConditionalGeneration,
Qwen3VLForConditionalGeneration, Qwen2VLForConditionalGeneration) nest the
language model and vision tower under a top-level 'model' dict:
- model.language_model.* (language model weights)
- model.visual.* (vision tower weights)
The existing sanitize() helpers only popped bare 'visual'/'vision_tower'
top-level keys, which matches the mlx-vlm conversion layout but not the HF
layout. On HF checkpoints:
- qwen3_vl_moe.sanitize crashed with KeyError('model') because
weights['language_model']['model'] doesn't exist (the language model
is nested under model.language_model.model.*).
- qwen3_vl and qwen2_vl leaked model.visual.* into the sanitized weights,
re-prefixed as language_model.model.visual.* — causing 'parameters not
in model' errors downstream.
Fix: after tree_unflatten, pop the top-level 'model' dict, re-root its
'language_model' to the top level, and drop its 'visual'/'vision_tower'
subtrees.
Verified on deepreinforce-ai/Ornith-1.0-35B (333 model.visual.* tensors,
full vision_config): the grafted checkpoint now loads without the
'768 parameters not in model' error that blocked it before.
Tests: TestVLSanitize covers all three model classes against the HF
ForConditionalGeneration layout — each must (a) not crash, (b) leave zero
visual/vision_tower keys, (c) preserve the language model tensors.
7d24881 to
04b2bd8
Compare
Hook only read args ($@), silently no-opped when piped filenames via stdin without xargs. Now falls back to 'while IFS= read -r f' when no args given. Pre-commit flow (xargs -0 -n1) unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
HuggingFace
ForConditionalGenerationVL checkpoints (Qwen3_5MoeForConditionalGeneration, Qwen3VLForConditionalGeneration, Qwen2VLForConditionalGeneration) nest the language model and vision tower under a top-levelmodeldict:The existing
sanitize()helpers only popped barevisual/vision_towertop-level keys — which matches the mlx-vlm conversion layout but not the HF layout. On HF checkpoints:qwen3_vl_moe.sanitizecrashes withKeyError: 'model'becauseweights["language_model"]["model"]doesn't exist (the language model is nested undermodel.language_model.model.*, so aftertree_unflattenit lives atweights["model"]["language_model"]["model"], notweights["language_model"]["model"]).qwen3_vlandqwen2_vlleakmodel.visual.*into the sanitized weights, re-prefixed aslanguage_model.model.visual.*— causing"parameters not in model"errors downstream.Reproducer
Fix
After
tree_unflatten, pop the top-levelmodeldict, re-root itslanguage_modelsubtree to the top level, and drop itsvisual/vision_towersubtrees. Three model classes changed identically:mlx_lm/models/qwen3_vl_moe.pymlx_lm/models/qwen3_vl.pymlx_lm/models/qwen2_vl.pyTests
TestVLSanitizeintests/test_models.pycovers all three model classes against the HFForConditionalGenerationlayout (9-key fixture mirroring Ornith-1.0's real weight_map). Each test asserts:sanitize()does not crash,visual/vision_towerkeys remain,language_model.*.All three fail on
main(Red), pass after the fix (Green). Fulltests/test_models.pysuite: 75 passed, 1 skipped (pre-existingtest_ssmfailure unchanged, unrelated).Verification
Verified end-to-end on
deepreinforce-ai/Ornith-1.0-35B(333model.visual.*tensors, fullvision_config): with this fix, the model loads cleanly where it previously crashed withKeyError: 'model'duringsanitize().