[tests] fix autoencoderdc ml training fix.#14129
Conversation
| if torch.device(torch_device).type == "cuda" and not torch.cuda.is_bf16_supported( | ||
| including_emulation=False | ||
| ): |
There was a problem hiding this comment.
Why do we need to use including_emulation=False here? My understanding is that this will cause the test to be skipped on older CUDA devices for which BF16 is not natively supported but could be emulated, but I'm not sure whether "emulation but no native support" implies that BF16 training is not supported.
There was a problem hiding this comment.
The default is_bf16_supported() only verifies that a bf16 tensor can be constructed on the device (that's the emulation fallback in the torch source) — it doesn't guarantee op coverage.
Emulated bf16 covers some cases like storage, casting, etc. but cuDNN has no bf16 convolution engines on compute capability < 8.0, and autocast routes convs to cuDNN in bf16. This is what happened in https://github.com/huggingface/diffusers/actions/runs/28765062110/job/85287707829?pr=14113.
including_emulation=False reduces to a compute-capability ≥ 8.0 check, which matches where cuDNN bf16 support actually begins — so for conv models, "emulated but not native" does mean bf16 autocast training doesn't work.
dg845
left a comment
dg845
left a comment
There was a problem hiding this comment.
Thanks for the PR! Left one comment.
|
/diffusers-bot pytest tests/models/autoencoders/test_models_autoencoder_dc.py::TestAutoencoderDCTraining::test_mixed_precision_training |
|
✅ |
2 participants
No description provided.