Commit c4b662f
authored
[Bug fix] Fake quantized model save after HF accelerate hooks are added (#906)
## What does this PR do?
**Type of change:** Bug fix
**Overview:** Fix `AttributeError: Can't get local object
'add_hook_to_module.<locals>.new_forward'` when saving a quantized model
a second time after restoring it with `device_map="auto"`.
When a model is loaded with `device_map="auto"`, accelerate's
`add_hook_to_module` patches every submodule (including
`TensorQuantizer` instances) and injects three instance attributes:
`_hf_hook`, `_old_forward`, and `forward` (a `functools.partial`
wrapping a local function). These are not picklable and were leaking
into the modelopt state dict collected by `get_modelopt_state()`,
causing `torch.save` to fail.
This PR adds the three accelerate-injected attributes to
`TensorQuantizer._skip_properties_for_save_restore` so they are excluded
from the serialized state, matching the existing pattern used for
HuggingFace and DeepSpeed attributes.
## Usage
```python
mto.enable_huggingface_checkpointing()
# Quantize and save
model = AutoModelForCausalLM.from_pretrained(name, device_map="auto")
model = mtq.quantize(model, mtq.FP8_DEFAULT_CFG, forward_loop=forward_loop)
model.save_pretrained(save_dir)
# Restore and save again (this previously failed)
model2 = AutoModelForCausalLM.from_pretrained(save_dir, device_map="auto")
model2.save_pretrained(save_dir_round2) # now works
```
## Testing
- Added unit test
`test_tensor_quantizer_modelopt_state_with_accelerate_hook` in
`tests/unit/torch/quantization/plugins/test_accelerate.py` that verifies
accelerate hook attributes are excluded from modelopt state and the
state dict remains picklable.
## Before your PR is "*Ready for review*"
- **Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)**
and your commits are signed.
- **Is this change backward compatible?**: Yes — only adds entries to a
skip set; existing saved checkpoints are unaffected.
- **Did you write any new necessary tests?**: Yes
- **Did you add or update any necessary documentation?**: No (internal
fix, no API change)
- **Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?**:
No
## Additional Information
The root cause is in accelerate's `add_hook_to_module`, which defines
`new_forward` as a local function and binds it via `functools.partial`
onto `module.forward`. Since local functions cannot be pickled, any
`TensorQuantizer` that has been hooked by accelerate becomes
unserializable unless these attributes are excluded.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Enhanced compatibility with accelerate library by excluding
framework-specific hooks and attributes from model state serialization,
preventing issues during save/restore operations.
* **Tests**
* Added test to validate that accelerate-related attributes are properly
excluded from model state and that the state remains picklable.
* **Public API**
* TensorQuantizer is now publicly exported.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: realAsma <akuriparambi@nvidia.com>1 parent eb99488 commit c4b662f
File tree
2 files changed
+34
-1
lines changed- modelopt/torch/quantization/nn/modules
- tests/unit/torch/quantization/plugins
2 files changed
+34
-1
lines changedLines changed: 4 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
156 | 156 | | |
157 | 157 | | |
158 | 158 | | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
159 | 163 | | |
160 | 164 | | |
161 | 165 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
| 17 | + | |
16 | 18 | | |
17 | 19 | | |
18 | 20 | | |
19 | 21 | | |
20 | 22 | | |
21 | | - | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
| |||
51 | 53 | | |
52 | 54 | | |
53 | 55 | | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
0 commit comments