[DeepSeek-V4] Loading nvidia/DeepSeek-V4-Pro-NVFP4 crashes: 'Linear' object has no attribute 'weight_scale' in fused_a loader (MIXED_PRECISION attn excluded → built BF16 but stored as FP8)

[d3nb]

System / environment

• Images (both reproduce, empirically verified 2026-07-09): nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc15.post1 and 1.3.0rc20 (NVIDIA Release 26.02, PyTorch 2.11.0a0). Note the DeepSeek-V4 model file was refactored between the two: in rc15.post1 the loader lives in _torch/models/modeling_deepseekv4.py; in rc20 it was folded into _torch/models/modeling_deepseekv3.py (there is no modeling_deepseekv4.py in rc20). The bug survives the refactor — see line refs for both below.
• Hardware: single node, 8× B300. Failure is during weight loading (per-rank, at Loading weights 0%), so it is hardware/parallelism independent — reproduces with TP4 and reproduces identically under both monolithic trtllm-serve serve and PD-disaggregated serving.
• Checkpoint: nvidia/DeepSeek-V4-Pro-NVFP4 — the ModelOpt experts-only NVFP4 re-quantization (hf_quant_config.json → producer.version: "dsv4-nvfp4-experts").

Reproduce

trtllm-serve serve /path/to/DeepSeek-V4-Pro-NVFP4 \
  --tp_size 4 --host 0.0.0.0 --port 8001
# crashes before serving, during model weight loading — no tokenizer/parser flags needed
# to reproduce. (In rc20 the `--tool_parser deepseek_v4` value was also removed; the valid
# set is deepseek_v3/deepseek_v31/deepseek_v32 — orthogonal to this bug.)

Actual behavior

File ".../tensorrt_llm/_torch/models/modeling_deepseekv4.py", line 1030, in load_weights
    _copy_deepseek_v4_fused_a_weight_scale(module, fused_a, fused_a_scale)
File ".../tensorrt_llm/_torch/models/modeling_deepseekv4.py", line 240, in _copy_deepseek_v4_fused_a_weight_scale
    if tuple(module.weight_scale.shape) == tuple(fused_a_scale.shape):
File ".../torch/nn/modules/module.py", line 1967, in __getattr__
    raise AttributeError(...)
AttributeError: 'Linear' object has no attribute 'weight_scale'

Root cause (traced through the code)

This checkpoint is MIXED_PRECISION: only the routed MoE experts are NVFP4; the attention projections are stored as native DeepSeek block-scale FP8 — every layers.N.attn.{wq_a,wkv,wo_a,wo_b,wq_b} leaf carries both .weight (fp8) and .scale. Its hf_quant_config.json:

{
  "producer": {"name": "modelopt", "version": "dsv4-nvfp4-experts"},
  "quantization": {
    "quant_algo": "MIXED_PRECISION",
    "quantized_layers": { "layers.0.ffn.experts": {"quant_algo": "NVFP4", "group_size": 16}, "...": "... (61 layers)" },
    "exclude_modules": ["*.attn.*", "*.ffn.shared_experts.*", "head", "mtp.*"]
  }
}

1. The attention Linears (q_a_proj / kv_a_proj_with_mqa, and MTP e_proj/h_proj) are built with model_config.get_quant_config() called with no name (e.g. modeling_deepseekv4.py:2177/2185/2197/2208). ModelConfig.get_quant_config(name=None) returns the global QuantConfig (model_config.py:249-250).
2. The per-layer table (quantized_layers → per_layer_quant_configs) is consulted only when a name is passed, and raises ValueError on a miss (model_config.py:252-254). In practice it is used only for the MoE experts (which are wired through a separate override_quant_config, modeling_deepseekv4.py:1467/1504). So no hf_quant_config.json / quant_cfg.json entry can reach the attention modules.
3. The global algo is MIXED_PRECISION + exclude_modules: "*.attn.*" (matched via fnmatch, quantization/quantize.py:51) → the attn Linear is built unquantized (BF16), with no weight_scale.
4. In load_weights, nvfp4_fused_a is False (attn weights are FP8, not FP4), so the code takes the else branch (~modeling_deepseekv4.py:1015-1032). That branch finds kv_a_proj_with_mqa.weight_scale_inv in the checkpoint and calls _copy_deepseek_v4_fused_a_weight_scale(module, fused_a, fused_a_scale), which dereferences module.weight_scale → AttributeError.

Note the same branch also does module.weight.data.copy_(fused_a) where fused_a is the FP8 weight and module.weight is BF16 — i.e. even if the weight_scale access were guarded, the FP8 weight would be loaded into a BF16 module without applying the block scale, silently producing wrong values. The branch implicitly assumes the fused_a module is FP8_BLOCK_SCALES.

Expectation / suggested fix

When the fused_a target module is unquantized (excluded → BF16) but the checkpoint stores it as block-scale FP8, the else branch should dequantize (weight_dequant, already present at modeling_deepseekv4.py:97-147) the FP8 fused_a with its fused_a_scale and copy the resulting BF16 into module.weight, instead of copying the scale onto a non-existent module.weight_scale. (Equivalently: guard on hasattr(module, "weight_scale") and branch to dequant.)

What we ruled out

• Image version — reproduced on both 1.3.0rc15.post1 and 1.3.0rc20 (empirically). rc20 refactored the V4 loader into modeling_deepseekv3.py and reworked model_config.py, but still sets exclude_modules from the HF quant config and crashes identically (rc20 crash site modeling_deepseekv3.py:~557).
• PD-disaggregated vs monolithic — identical crash (both workers run trtllm-serve serve; loading is the same code path).
• Removing *.attn.* from exclude_modules — still crashes; under a MIXED_PRECISION global there is no per-module algo for the now-un-excluded attn, so it is not built as FP8.
• Missing quant_cfg.json — neither the checkpoint nor the ModelOpt extended file exists in the repo; adding per-layer entries there cannot help, since attn queries get_quant_config() without a name (point 2 above).

Additional note

The native DeepSeek checkpoint deepseek-ai/DeepSeek-V4-Pro (FP8 + MXFP4 routed experts; config-level quantization_config: {quant_method: fp8, weight_block_size: [128,128]}, no exclude_modules) loads and serves fine — verified end-to-end on rc15.post1, TP4 on 4× B300: weights load to 100%, Application startup complete, and chat completions generate correctly. Because the global algo is FP8_BLOCK_SCALES (not MIXED_PRECISION) and attn is not excluded, every attn Linear is built as FP8_BLOCK_SCALES and has a weight_scale. So TensorRT-LLM can serve DeepSeek-V4-Pro on this hardware; the crash is specific to the ModelOpt experts-only NVFP4 packaging (nvidia/DeepSeek-V4-Pro-NVFP4), where attn is exclude_modules'd (→ built BF16) yet physically stored as FP8 with a scale.

[waynehacking8]

I tried to reproduce this on current main (8bcec87) with a tiny synthetic replica of the checkpoint (same hf_quant_config.json, tensor names, fp8-e4m3 weights + ue8m0 block scales, experts as ModelOpt NVFP4 quadruples) on an RTX PRO 6000, and main now fails earlier than your trace: DeepseekV4DecoderLayer hits the "MIXED_PRECISION is ambiguous" assert (modeling_deepseekv4.py:1762, added with #15414) during model construction, before load_weights ever runs. So on main this packaging needs layer-level MIXED_PRECISION resolution at construction first (the FIXME right above that assert acknowledges the gap), and then your fused_a fix -- the unguarded weight_scale deref plus raw-fp8 copy you diagnosed is still there in the else branch (modeling_deepseekv4.py:1032, same pattern at modeling_deepseekv3.py:557 for the V3/V32 arches). I have the guard+weight_dequant half ready on a branch (https://github.com/waynehacking8/TensorRT-LLM/tree/wayne/fix-16196-fused-a-dequant) if maintainers want it as a follow-on; caveats on my repro: main python over the 1.3.0rc20 wheel binaries, synthetic tiny checkpoint, not the real weights.

[d3nb]

Thanks @waynehacking8 — I tested your fused-A dequant fix against the real nvidia/DeepSeek-V4-Pro-NVFP4 weights (851 GB, 64 shards) on 4× B300, TP4, and it clears the loading crash.

Setup: since the real checkpoint's deepseek_v4 model type isn't recognized by the transformers in the rc20 wheel (ValueError: ... does not recognize this architecture at config_utils.py:569, before load), I ran on 1.3.0rc15.post1 (which loads deepseek_v4 fine — the official deepseek-ai/DeepSeek-V4-Pro FP8+MXFP4 checkpoint loads + serves + generates end-to-end on the exact same setup, as a control). I ported your guard to rc15.post1's modeling_deepseekv4.py else-branch (line 1030), i.e. the same hasattr(module, "weight_scale") → weight_dequant(...) change.

Result: the 'Linear' object has no attribute 'weight_scale' crash is gone — weights load to 100%. So the fused-A guard+dequant is confirmed on real weights, not just the synthetic replica. 🎉

But a second, downstream blocker then surfaces during executor init (MLA attention warmup), and I believe it's the same root cause showing up at runtime:

File ".../tensorrt_llm/_torch/modules/attention.py", line 2358, in _q_branch
    assert not self._is_fused_q_fp8_quant_enabled(num_generations=num_generations), (
AssertionError: CuTe DSL q_b_proj path is incompatible with the fused FP8 q-quant branch

_q_branch takes the CuTe-DSL path because q_b_proj.weight.dtype == torch.bfloat16 (attn is exclude_modules'd → built unquantized BF16), while _is_fused_q_fp8_quant_enabled() is still True — exactly the "future config drift" the assert's comment warns about. So dequantizing fused-A to BF16 makes the model load, but leaves the attention modules BF16 while the rest of the MLA path still assumes FP8 q-quant.

This lines up with your point that the real fix is the construction-time, layer-level MIXED_PRECISION resolution (the MIXED_PRECISION is ambiguous assert + the FIXME above it): if the excluded-but-FP8 attn modules were built as FP8_BLOCK_SCALES at construction, they'd carry a weight_scale (so the fused-A else-branch never triggers) and the runtime _q_branch FP8-q-quant contract would hold. The guard+dequant is a good defensive fallback, but for this ModelOpt experts-only NVFP4 packaging it isn't sufficient on its own — the attn modules really need to end up FP8, not BF16.

Happy to test a construction-time resolution patch on the real weights + B300 if you or the maintainers put one up. For reference, my env: rc15.post1 image, real nvidia/DeepSeek-V4-Pro-NVFP4, TP4 on 4× B300, moe_config.backend=TRTLLM.

[waynehacking8]

Thanks for running the real thing -- that closes the loading half on actual weights, and your downstream _q_branch assert points at the same place my construction repro does. I think the underlying semantic bug is that this packaging's exclude_modules means "left at the base model's precision", and DeepSeek-V4-Pro's base attn IS block-scale FP8 (the repack's embedded quantization_config still says quant_method=fp8 with weight_block_size [128,128] alongside the NVFP4 moe_quant_algo), while TRT-LLM's exclude handling builds excluded modules as unquantized BF16 -- so the construction-time resolution should build the excluded attn Linears as FP8_BLOCK_SCALES, which gives them a weight_scale (the fused-A else branch then never fires) and keeps the MLA fused-q-quant contract intact, with the layer-level MIXED_PRECISION assert (modeling_deepseekv4.py:1762 on main) resolved the same way. I'll draft that construction-time patch on my branch and will gladly take you up on the B300 offer once it's up.

[waynehacking8]

@d3nb the construction-time resolution is up on the same branch (https://github.com/waynehacking8/TensorRT-LLM/tree/wayne/fix-16196-fused-a-dequant, commit 3972f5e, on top of the loader guard). It does three things: derives an FP8_BLOCK_SCALES config from the checkpoint's embedded base quantization_config and builds the excluded attention / shared experts / MTP e_proj+h_proj with it (matching the native FP8 flow), resolves the decoder-layer algo from the per-layer quantized_layers table under MIXED_PRECISION (so the ambiguity assert goes away and the MoE stays NVFP4), and accepts the checkpoint-native layers.N.ffn.experts key form that these repacks emit. On my tiny replica: construction passes, weights load to 100%, the fused_a fp8 payload is bit-exact vs the checkpoint, and q_b_proj / shared experts end up FP8 with weight_scale present -- which should also satisfy the _q_branch fused-q-quant contract you hit. Not covered: runtime on real weights (your B300 loop), and MTP spec decoding (the MTP routed experts are stored at the base model's MXFP4 -- worth leaving spec-dec off for the first run).

[d3nb]

Update from the B300 real-weights side, @waynehacking8 — the construction-time resolution works end-to-end on the real nvidia/DeepSeek-V4-Pro-NVFP4, and the MTP path you flagged is now working too.

Construction-time resolution — validated on real weights + B300

I implemented the same idea you described (independently, on 1.3.0rc15.post1 in model_config.py rather than modeling_deepseekv4.py, since that's the wheel that loads deepseek_v4 configs for us): when quant_algo == MIXED_PRECISION and the checkpoint's base quantization_config is quant_method=fp8 / weight_block_size=[128,128], resolve the global to FP8_BLOCK_SCALES (reusing the native-fp8 path) so the excluded attention / shared-expert projections build as FP8 with a weight_scale, keep the per-layer NVFP4 experts, and keep the genuinely-unquantized modules (head, mtp) excluded.

Result on 4× B300, TP4, moe_config.backend=TRTLLM:

• weights load 100%, Application startup complete, chat completions generate correctly ("我是DeepSeek…");
• q_b_proj / shared experts end up FP8 with weight_scale present — so the runtime _q_branch fused-FP8-q-quant assert I hit earlier is gone, exactly as you predicted.

So both halves (fused-A dequant guard, and construction-time FP8 resolution) are now confirmed on the actual 851 GB checkpoint, not just synthetic replicas.

The MTP piece: dense experts are NVFP4, MTP experts are MXFP4 (same checkpoint)

You were right that MTP needed separate handling — the reason turned out to be a per-region layout split inside one checkpoint:

• dense routed experts layers.0..60.ffn.experts → dtype U8 = NVFP4 (ModelOpt re-quantized these);
• MTP routed experts mtp.0.ffn.experts → dtype I8 = MXFP4 (left at the base model's precision).

The routed-MoE layout resolver detects a single layout (from layers.0.ffn.experts.0.w1.weight = NVFP4) and applies it to all MoE layers including the MTP layer, so the MTP experts get NVFP4 and die in fused_moe/quantization.py load_quant_scales. Detecting the MTP expert dtype separately (mtp.0.ffn.experts.0.w1.weight → I8 → W4A8_MXFP4_*) and covering the MTP layer indices with that config fixes it.

With that, on 4× B300:

• MTP=1 (checkpoint-native num_nextn_predict_layers): loads + serves + generates. Log: MTP routed experts layout=mxfp4 over 1 layer(s) / dense routed experts NVFP4.
• MTP=3 (NVIDIA's B300 config — the checkpoint's single MTP layer is duplicated ×3): also loads + serves + generates, once the duplicated MTP layer indices model.layers.{num_hidden+i} are covered too. (One wrinkle: spec_config is None in from_pretrained in our path, so _get_mtp_num_layers returns 0 there — I currently cover a small generous range of MTP indices as a stopgap; the clean version would thread the model-side num_nextn_predict_layers in, or extend coverage where the MTP layers are duplicated.)

Our model_config.py diff (rc15.post1) is small — happy to share it or fold the MTP-layout piece into your branch, whichever you prefer. We can re-run any revision of your branch against the real weights + B300 (both MTP off and MTP=1/3). For reference, env: 1.3.0rc15.post1, nvidia/DeepSeek-V4-Pro-NVFP4, TP4 on 4× B300, moe_config.backend=TRTLLM.

[d3nb]

@waynehacking8 I put the MTP-layout piece up as a PR against your branch so it stays consolidated: waynehacking8#1 (single change to _set_deepseek_v4_routed_moe_quant_config, +38/-6, on top of your 3972f5e). It detects the MTP routed-expert dtype separately (mtp.0.ffn.experts.0.w1.weight → I8/MXFP4) so the MTP layers stop inheriting the dense NVFP4 config. Validated on the real nvidia/DeepSeek-V4-Pro-NVFP4 + 4× B300 (MTP=1 and MTP=3, accept_len ≈ 2.86). Feel free to squash/reword or fold it in however you like.