Commit 04fcf24
authored
Fix LLM deploy test failure by defaulting expert parallelism to 1 (#1273)
### What does this PR do?
Type of change: Bug fix
Fixes TRT-LLM DeepEP kernel failures during LLM deployment on
unsupported GPUs (e.g. Blackwell SM 12.0) by defaulting expert
parallelism (`ep`) to 1 instead of auto-setting it to the GPU count for
MoE models.
Previously, when the model config contained expert-related keys, `ep`
was automatically set to `torch.cuda.device_count()`, which triggered
DeepEP kernel failures on GPUs that don't support it. Now `ep` defaults
to 1 while still enabling attention data parallelism for MoE models.
Expert parallelism can be enabled explicitly by the caller when the
environment is known to support it.
### Testing
- [x] Verified that the `llm_ptq` test passes with this fix on Blackwell
GPUs.
- [x] 2-gpu CI test triggered:
https://github.com/NVIDIA/Model-Optimizer/actions/runs/24495054531/job/71588037727
### Before your PR is "*Ready for review*"
Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)
and your commits are signed (`git commit -s -S`).
Make sure you read and follow the [Security Best
Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors)
(e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(...,
weights_only=False)`, `pickle`, etc.).
- Is this change backward compatible?: ✅
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: N/A
- Did you write any new necessary tests?: N/A
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
N/A
Signed-off-by: Chenjie Luo <chenjiel@nvidia.com>1 parent 6ded36b commit 04fcf24
1 file changed
Lines changed: 3 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
109 | 109 | | |
110 | 110 | | |
111 | 111 | | |
112 | | - | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
113 | 115 | | |
114 | 116 | | |
115 | 117 | | |
116 | 118 | | |
117 | | - | |
118 | 119 | | |
119 | 120 | | |
120 | 121 | | |
| |||
0 commit comments