Skip to content

fix(vl): strip model.visual.* in qwen2_vl/qwen3_vl/qwen3_vl_moe sanitize#1473

Open
Jonathangadeaharder wants to merge 2 commits into
ml-explore:mainfrom
Jonathangadeaharder:fix/vl-sanitize-model-visual-prefix
Open

fix(vl): strip model.visual.* in qwen2_vl/qwen3_vl/qwen3_vl_moe sanitize#1473
Jonathangadeaharder wants to merge 2 commits into
ml-explore:mainfrom
Jonathangadeaharder:fix/vl-sanitize-model-visual-prefix

Conversation

@Jonathangadeaharder

Copy link
Copy Markdown

Problem

HuggingFace ForConditionalGeneration VL checkpoints (Qwen3_5MoeForConditionalGeneration, Qwen3VLForConditionalGeneration, Qwen2VLForConditionalGeneration) nest the language model and vision tower under a top-level model dict:

model.language_model.*   (language model weights)
model.visual.*           (vision tower weights)

The existing sanitize() helpers only popped bare visual / vision_tower top-level keys — which matches the mlx-vlm conversion layout but not the HF layout. On HF checkpoints:

  • qwen3_vl_moe.sanitize crashes with KeyError: 'model' because weights["language_model"]["model"] doesn't exist (the language model is nested under model.language_model.model.*, so after tree_unflatten it lives at weights["model"]["language_model"]["model"], not weights["language_model"]["model"]).
  • qwen3_vl and qwen2_vl leak model.visual.* into the sanitized weights, re-prefixed as language_model.model.visual.* — causing "parameters not in model" errors downstream.

Reproducer

from mlx.utils import tree_unflatten
flat = [
    ("model.visual.pos_embed.weight", None),
    ("model.language_model.model.layers.0.self_attn.q_proj.weight", None),
    ("model.language_model.lm_head.weight", None),
]
tree = tree_unflatten(flat)
# top keys: ["model"] — no bare "visual" or "language_model"
tree.pop("visual", None)  # no-op
tree["language_model"]["model"]  # KeyError: "model"

Fix

After tree_unflatten, pop the top-level model dict, re-root its language_model subtree to the top level, and drop its visual / vision_tower subtrees. Three model classes changed identically:

  • mlx_lm/models/qwen3_vl_moe.py
  • mlx_lm/models/qwen3_vl.py
  • mlx_lm/models/qwen2_vl.py

Tests

TestVLSanitize in tests/test_models.py covers all three model classes against the HF ForConditionalGeneration layout (9-key fixture mirroring Ornith-1.0's real weight_map). Each test asserts:

  1. sanitize() does not crash,
  2. zero visual / vision_tower keys remain,
  3. the language model tensors survive, re-rooted under language_model.*.

All three fail on main (Red), pass after the fix (Green). Full tests/test_models.py suite: 75 passed, 1 skipped (pre-existing test_ssm failure unchanged, unrelated).

Verification

Verified end-to-end on deepreinforce-ai/Ornith-1.0-35B (333 model.visual.* tensors, full vision_config): with this fix, the model loads cleanly where it previously crashed with KeyError: 'model' during sanitize().

@Jonathangadeaharder Jonathangadeaharder force-pushed the fix/vl-sanitize-model-visual-prefix branch 3 times, most recently from 9eb5800 to 7d24881 Compare July 5, 2026 08:13
HuggingFace ForConditionalGeneration VL checkpoints (Qwen3_5MoeForConditionalGeneration,
Qwen3VLForConditionalGeneration, Qwen2VLForConditionalGeneration) nest the
language model and vision tower under a top-level 'model' dict:
  - model.language_model.* (language model weights)
  - model.visual.*        (vision tower weights)

The existing sanitize() helpers only popped bare 'visual'/'vision_tower'
top-level keys, which matches the mlx-vlm conversion layout but not the HF
layout. On HF checkpoints:
  - qwen3_vl_moe.sanitize crashed with KeyError('model') because
    weights['language_model']['model'] doesn't exist (the language model
    is nested under model.language_model.model.*).
  - qwen3_vl and qwen2_vl leaked model.visual.* into the sanitized weights,
    re-prefixed as language_model.model.visual.* — causing 'parameters not
    in model' errors downstream.

Fix: after tree_unflatten, pop the top-level 'model' dict, re-root its
'language_model' to the top level, and drop its 'visual'/'vision_tower'
subtrees.

Verified on deepreinforce-ai/Ornith-1.0-35B (333 model.visual.* tensors,
full vision_config): the grafted checkpoint now loads without the
'768 parameters not in model' error that blocked it before.

Tests: TestVLSanitize covers all three model classes against the HF
ForConditionalGeneration layout — each must (a) not crash, (b) leave zero
visual/vision_tower keys, (c) preserve the language model tensors.
@Jonathangadeaharder Jonathangadeaharder force-pushed the fix/vl-sanitize-model-visual-prefix branch from 7d24881 to 04b2bd8 Compare July 5, 2026 08:39
Hook only read args ($@), silently no-opped when piped filenames
via stdin without xargs. Now falls back to 'while IFS= read -r f'
when no args given. Pre-commit flow (xargs -0 -n1) unaffected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants