Skip to content

Commit fb5923c

Browse files
Add Megatron-Bridge pruning example scripts (#800)
## What does this PR do? **Type of change:** new example <!-- Use one of the following: Bug fix, new feature, new example, new tests, documentation. --> Megatron-Bridge pruning example scripts (HF input, HF / Megatron output). Also defined some utility functions we can reuse for adding examples for quantization or other optimizations: - `modelopt.torch.utils.plugins.mbridge.load_mbridge_model_from_hf`: Load HF to MBridge with ModelOpt spec in desired TP/PP/etc configuration - `modelopt.torch.utils.plugins.mbridge.get_hf_mbridge_calibration_loop`: Create `forward_loop` for calibration on a HF dataset - Supports all datasets available in `modelopt.torch.utils.dataset_utils` (`cnn_dailymail`, `nemotron-post-training-dataset-v2`, etc) - Support applying chat template for chat-based data ## Usage <!-- You can potentially add a usage example below. --> From `nvcr.io/nvidian/nemo:26.02.rc1` container (mount latest code to `/opt/Megatron-Bridge` and `/opt/Model-Optimizer`) ```python torchrun --nproc_per_node 2 /opt/Model-Optimizer/examples/megatron_bridge/prune_minitron.py \ --hf_model_name_or_path Qwen/Qwen3-8B \ --prune_target_params 6e9 \ --hparams_to_skip num_attention_heads \ --output_hf_path /tmp/Qwen3-8B-Pruned-6B ``` ## Testing <!-- Mention how have you tested your change if applicable. --> - [x] Manually ran pruning script in nemo:25.11 container (plus modelopt and mbridge mounted to latest) for Qwen3-8B and Nemotron-Nano-9B-v2 with PP=8 and PP=4 - [ ] Added per-PR CI/CD test for example script Results when pruning Qwen3 8B -> 6B (10 different configurations) with and without chat template on the dataset samples. Perhaps MMLU is not the right metric to look at. | Layers | Hidden Size | FFN Hidden Size | Params | MMLU (Concatenated messages) | MMLU (Applied Chat Template) | |--------|-------------|-----------------|--------|------------------|----------------------| | 34 | 3328 | 11264 | 5.99B | 0.401 | 0.393 | | 30 | 3584 | 11776 | 5.99B | 0.588 | 0.576 | | 36 | 3840 | 8192 | 5.98B | 0.507 | 0.518 | | 36 | 3584 | 9216 | 5.98B | 0.477 | 0.469 | | 36 | 3072 | 11776 | 5.97B | 0.255 | 0.249 | | 32 | 3584 | 10752 | 5.96B | 0.554 | 0.549 | | 28 | 4096 | 10240 | 5.94B | 0.400 | 0.438 | | 36 | 4096 | 7168 | 5.93B | 0.461 | 0.438 | | 36 | 3328 | 10240 | 5.92B | 0.362 | 0.359 | | 34 | 3840 | 8704 | 5.91B | 0.515 | 0.546 | ## Before your PR is "*Ready for review*" <!-- If you haven't finished some of the above items you can still open `Draft` PR. --> - **Make sure you read and follow [Contributor guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)** and your commits are signed. - **Is this change backward compatible?**: Yes <!--- If No, explain why. --> - **Did you write any new necessary tests?**: Yes - **Did you add or update any necessary documentation?**: ‼️ TODO - **Did you update [Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?**: Yes <!--- Only for new features, API changes, critical bug fixes or bw breaking changes. --> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes **New Features** * Added new Megatron-Bridge pruning example demonstrating Minitron-based model optimization with advanced pruning configurations. **Documentation** * Updated core project documentation to highlight Megatron-Bridge as a supported optimization framework. * Added comprehensive example documentation for Megatron-Bridge workflows including pruning, distillation, and quantization. * Updated pruning guides with Megatron-Bridge integration examples and best practices. <sub>✏️ Tip: You can customize this high-level summary in your review settings.</sub> <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Keval Morabia <28916987+kevalmorabia97@users.noreply.github.com>
1 parent 23278a4 commit fb5923c

11 files changed

Lines changed: 791 additions & 42 deletions

File tree

.github/CODEOWNERS

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ modelopt/torch/utils @NVIDIA/modelopt-torch-utils-codeowners
4444
/examples/llm_ptq @NVIDIA/modelopt-examples-llm_ptq-codeowners
4545
/examples/llm_qat @NVIDIA/modelopt-examples-llm_qat-codeowners
4646
/examples/llm_sparsity @NVIDIA/modelopt-torch-sparsity-codeowners
47+
/examples/megatron_bridge @NVIDIA/modelopt-examples-megatron-codeowners
4748
/examples/model_hub @NVIDIA/modelopt-examples-model_hub-codeowners
4849
/examples/nemo_run @NVIDIA/modelopt-examples-megatron-codeowners
4950
/examples/onnx_ptq @NVIDIA/modelopt-onnx-codeowners

CHANGELOG.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ NVIDIA Model Optimizer Changelog (Linux)
1313
- Add standalone type inference option (``--use_standalone_type_inference``) in ONNX AutoCast as an alternative to ONNX's ``infer_shapes``. This experimental feature performs type-only inference without shape inference, useful as a workaround when shape inference fails or to avoid unnecessary shape inference overhead.
1414
- Add support for Kimi K2 Thinking model quantization from the original int4 checkpoint.
1515
- Add support for ``params`` constraint based automatic neural architecture search in Minitron pruning (``mcore_minitron``) as an alternative to manual pruning (using ``export_config``). See `examples/pruning/README.md <https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/pruning>`_ for more details on its usage.
16+
- New example for Minitron pruning with Megatron-Bridge framework along with advanced pruning usage with new ``params`` constraint based pruning. Check `examples/megatron_bridge/README.md <https://github.com/NVIDIA/Model-Optimizer/tree/main/examples/megatron_bridge>`_ for example scripts.
1617
- Add support for calibration data with multiple samples in ``npz`` format in the ONNX Autocast workflow.
1718
- Add ``--opset`` option to ONNX quantization CLI to specify the target opset version for the quantized model.
1819
- Add support for context parallelism in Eagle speculative decoding for huggingface and megatron core models.

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ ______________________________________________________________________
2020
**[Input]** Model Optimizer currently supports inputs of a [Hugging Face](https://huggingface.co/), [PyTorch](https://github.com/pytorch/pytorch) or [ONNX](https://github.com/onnx/onnx) model.
2121

2222
**[Optimize]** Model Optimizer provides Python APIs for users to easily compose the above model optimization techniques and export an optimized quantized checkpoint.
23-
Model Optimizer is also integrated with [NVIDIA NeMo](https://github.com/NVIDIA-NeMo/NeMo), [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [Hugging Face Accelerate](https://github.com/huggingface/accelerate) for training required inference optimization techniques.
23+
Model Optimizer is also integrated with [NVIDIA Megatron-Bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge), [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) and [Hugging Face Accelerate](https://github.com/huggingface/accelerate) for training required inference optimization techniques.
2424

2525
**[Export for deployment]** Seamlessly integrated within the NVIDIA AI software ecosystem, the quantized checkpoint generated from Model Optimizer is ready for deployment in downstream inference frameworks like [SGLang](https://github.com/sgl-project/sglang), [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/quantization), [TensorRT](https://github.com/NVIDIA/TensorRT), or [vLLM](https://github.com/vllm-project/vllm).
2626

examples/megatron_bridge/README.md

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# Megatron Bridge
2+
3+
This directory contains examples of using Model Optimizer with [NeMo Megatron-Bridge](https://github.com/NVIDIA-Nemo/Megatron-Bridge) framework for pruning, distillation, quantization, etc.
4+
5+
<div align="center">
6+
7+
| **Section** | **Description** | **Link** | **Docs** |
8+
| :------------: | :------------: | :------------: | :------------: |
9+
| Pre-Requisites | Development environment setup | \[[Link](#pre-requisites)\] | |
10+
| Pruning | Examples of pruning a model using Minitron algorithm | \[[Link](#pruning)\] | |
11+
| Distillation | Examples of distillation a pruned or quantized model | \[[Link](#distillation)\] | |
12+
| Quantization | Examples of quantizing a model | \[[Link](#quantization)\] | |
13+
| Resources | Extra links to relevant resources | \[[Link](#resources)\] | |
14+
15+
</div>
16+
17+
## Pre-Requisites
18+
19+
Running these examples requires many additional dependencies to be installed (e.g., Megatron-Bridge, Megatron-core, etc.), hence we strongly recommend directly using the NeMo container (e.g., `nvcr.io/nvidia/nemo:26.02`) which has all the dependencies installed.
20+
21+
To get the latest ModelOpt features and examples, you can mount your latest ModelOpt cloned repository to the container at `/opt/Model-Optimizer` or pull the latest changes once inside the docker container (`cd /opt/Model-Optimizer && git checkout main && git pull`).
22+
23+
## Pruning
24+
25+
This section shows how to prune a HuggingFace model using Minitron algorithm in Megatron-Bridge framework. Checkout other available pruning algorithms, supported frameworks and models, and general pruning getting-started in the [pruning README](../pruning/README.md).
26+
27+
Example usage to prune Qwen3-8B to 6B on 2-GPUs (Pipeline Parallelism = 2) while skipping pruning of `num_attention_heads` using following defaults:
28+
1024 samples from [`nemotron-post-training-dataset-v2`](https://huggingface.co/datasets/nvidia/Nemotron-Post-Training-Dataset-v2) for calibration,
29+
at-most 20% depth (`num_layers`) and 40% width is pruned per prunable hparam (`hidden_size`, `ffn_hidden_size`, ...),
30+
top-10 candidates are evaluated for MMLU score (5% sampled data) to select the best model.
31+
32+
```bash
33+
torchrun --nproc_per_node 2 /opt/Model-Optimizer/examples/megatron_bridge/prune_minitron.py \
34+
--hf_model_name_or_path Qwen/Qwen3-8B \
35+
--prune_target_params 6e9 \
36+
--hparams_to_skip num_attention_heads \
37+
--output_hf_path /tmp/Qwen3-8B-Pruned-6B
38+
```
39+
40+
To see the full usage for advanced configurations, run:
41+
42+
```bash
43+
python /opt/Model-Optimizer/examples/megatron_bridge/prune_minitron.py --help
44+
```
45+
46+
> [!TIP]
47+
> If number of layers in the model is not divisible by number of GPUs i.e. pipeline parallel (PP) size, you can configure
48+
> uneven PP by setting `--num_layers_in_first_pipeline_stage` and `--num_layers_in_last_pipeline_stage`.
49+
> E.g. for Qwen3-8B with 36 layers and 8 GPUs, you can set both to 3 to get 3-5-5-5-5-5-5-3 layers per GPU.
50+
51+
## Distillation
52+
53+
TODO
54+
55+
## Quantization
56+
57+
TODO
58+
59+
## Resources
60+
61+
- 📅 [Roadmap](https://github.com/NVIDIA/Model-Optimizer/issues/146)
62+
- 📖 [Documentation](https://nvidia.github.io/Model-Optimizer)
63+
- 💡 [Release Notes](https://nvidia.github.io/Model-Optimizer/reference/0_changelog.html)
64+
- 🐛 [File a bug](https://github.com/NVIDIA/Model-Optimizer/issues/new?template=1_bug_report.md)
65+
-[File a Feature Request](https://github.com/NVIDIA/Model-Optimizer/issues/new?template=2_feature_request.md)

0 commit comments

Comments
 (0)