Attention-only LoRA fine-tuning on NVFP4-quantized 122B MoE — community implementation + preprint

[ddreeselogs]

[ddreeselogs]

Hey team. Built an NVFP4 LoRA training pipeline for Qwen 3.5 122B MoE on a single
DGX Spark (GB10, 128GB unified memory). Wanted to surface it in case it's useful
or overlaps with work your team is doing.

TL;DR

• First publicly documented NVFP4 LoRA fine-tuning at 100B+ MoE scale on
  single-node unified-memory hardware.
• Workaround for the absent FP4 dequant-matmul backward kernel: restrict LoRA
  targets to attention only, exploit the NVFP4-mtp variant's bf16 attention
  so standard autograd works without touching FP4 weights.
• 500-scenario held-out benchmark: 89.0% base → 95.3% fine-tuned (+6.3pt).
• Zero-adapter adversarial merge control scores 88.8% (within sampling noise
  of base), confirming the lift is LoRA-attributable and not merge-pipeline
  artifact.

Preprint

Zenodo (CC BY 4.0): https://zenodo.org/records/19646701
PDF attached to this issue.

Context

No training library supports NVFP4 right now. PEFT, Unsloth, Axolotl, TRL all
need an FP4 dequant-matmul kernel with PyTorch-tracked gradients, which hasn't
shipped yet. Practitioners with NVFP4-only model distributions (like the Qwen
3.5 122B you ship) have no supported path to fine-tune them.

Workaround I landed on: target LoRA at attention projections only (q/k/v/o).
The NVFP4-mtp variant you distribute keeps attention in bf16 and packs
everything else NVFP4. That means standard bf16 autograd handles attention
plus LoRA math, and the frozen NVFP4 MoE experts only see forward pass. No
FP4 backward kernel needed.

Questions

Three things I'd love input on if you have a minute:

1. Is there internal work on the FP4 dequant-matmul backward kernel that would
   obsolete this approach? If so, I'd rather cite and adapt than keep shipping
   a workaround.

2. Would a proper NVFP4 backend for PEFT or Unsloth be something modelopt wants
   community contribution on? I'd contribute if it's aligned with your direction.

3. Scope-restricting LoRA to attention-only is a hack, not a method. Any sense
   of whether it's the right hack, or if you'd steer me somewhere else?

Code, training data, benchmark harness all available if useful. Not looking
for anything specific out of this, just want to make sure I'm not building
parallel to your team.