[Bugfix] Enable MTP for the official Qwen3.5 NVFP4 checkpoint

[mmangkad]

Purpose

Enable MTP for the official Qwen3.5 NVFP4 checkpoint, which currently fails to initialize with MTP because the Qwen3.5 MTP branch is stored in BF16 rather than modelopt_fp4.

Test Plan

8xB300

vllm serve nvidia/Qwen3.5-397B-A17B-NVFP4 \
  --tensor-parallel-size 8 \
  --language-model-only \
  --reasoning-parser qwen3 \
  --enable-prefix-caching \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'

Test Result

python tests/evals/gsm8k/gsm8k_eval.py

Running GSM8K evaluation: 1319 questions, 5-shot
Evaluating: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1319/1319 [01:38<00:00, 13.44it/s]

Results:
Accuracy: 0.879
Invalid responses: 0.024
Total latency: 98.163 s
Questions per second: 13.437
Total output tokens: 193341
Output tokens per second: 1969.593

[gemini-code-assist]

Code Review

This pull request introduces a specialized quantization configuration handler for Qwen3.5 MTP models, ensuring that NVFP4 checkpoints use bf16 weights for the MTP branch. The implementation uses a helper function and a try...finally block to temporarily override the global configuration during layer initialization. Feedback suggests improving the safety of this implementation by using a shallow copy of the configuration object instead of mutating shared state, which would also simplify the code by removing the need for manual state restoration.

[vadiklyutiy]

I think it should be fixed on checkpoint level.

There is exclude_modules in hf_quant_config.json where MPT layers should be added.

[mmangkad]

I think it should be fixed on checkpoint level.

There is exclude_modules in hf_quant_config.json where MPT layers should be added.

@vadiklyutiy exactly, hf_quant_config.json already excludes the MTP path, but vllm still initializes Qwen3.5 MTP with modelopt_fp4 and applies that quant config during MTP construction. Because of that, the official Qwen3.5 NVFP4 checkpoint cannot start with method="mtp" before this change.

[gaby]

👍 looking forward to this

[vadiklyutiy]

this is the fix from modelopt team NVIDIA/Model-Optimizer#1124 (review)

[mmangkad]

this is the fix from modelopt team NVIDIA/Model-Optimizer#1124 (review)

The issue here is still on the vllm side: vllm does not currently respect the MTP exclusion when constructing Qwen3.5 MTP, so a re-export alone would not fix this.

[ZJY0516]

I think it should be fixed on checkpoint level.

There is exclude_modules in hf_quant_config.json where MPT layers should be added.

We can accept this workaroud temporarily, because I think many users need this

[vadiklyutiy]

this is the fix from modelopt team NVIDIA/Model-Optimizer#1124 (review)

The issue here is still on the vllm side: vllm does not currently respect the MTP exclusion when constructing Qwen3.5 MTP, so a re-export alone would not fix this.

I think the problem is not that vLLM ignores the exclusion, but rather that mtp.fc.weight is missing in the exclusion list, no?

[vadiklyutiy]

I think it should be fixed on checkpoint level.
There is exclude_modules in hf_quant_config.json where MPT layers should be added.

We can accept this workaroud temporarily, because I think many users need this

I am ok with accepting workaround temporary, but seem here the thing is overcomplicated because assumes that vllm doesn't respect exclusion list, but it does (see is_layer_excluded() in modelopt.py)

[vadiklyutiy]

#38832 is a bit simpler and smaller change that only fix problematic place