Skip to content

Commit d1b492b

Browse files
sayakpaulstevhliu
andauthored
Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
1 parent b761e8c commit d1b492b

1 file changed

Lines changed: 10 additions & 5 deletions

File tree

docs/source/en/optimization/fp16.md

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -150,11 +150,14 @@ pipeline(prompt, num_inference_steps=30).images[0]
150150

151151
Compilation is slow the first time, but once compiled, it is significantly faster. Try to only use the compiled pipeline on the same type of inference operations. Calling the compiled pipeline on a different image size retriggers compilation which is slow and inefficient.
152152

153-
### Compilation on shape changes
153+
### Dynamic shape compilation
154154

155-
`torch.compile()` maintains a stack of "guards" for the shapes and conditions it sees when it is triggered. When that is violated, the compiler triggers recompilation. This means that if a model was compiled on the 1024x1024 resolution, for example, it will trigger recompilation if it is called on a different resolution.
155+
> [!TIP]
156+
> Make sure to always use the nightly version of PyTorch for better support.
157+
158+
`torch.compile` keeps track of input shapes and conditions, and if these are different, it recompiles the model. For example, if a model is compiled on a 1024x1024 resolution image and used on an image with a different resolution, it triggers recompilation.
156159

157-
In these cases, it's beneficial to compile with `dynamic=True`:
160+
To avoid recompilation, add `dynamic=True` to try and generate a more dynamic kernel to avoid recompilation when conditions change.
158161

159162
```diff
160163
+ torch.fx.experimental._config.use_duck_shape = False
@@ -163,9 +166,11 @@ In these cases, it's beneficial to compile with `dynamic=True`:
163166
)
164167
```
165168

166-
Make sure to always use the nightly version of PyTorch for this. Specifying `use_duck_shape` to be `False` instructs the compiler if it should use the same symbolic variable to represent input sizes that are the same. For more details, check out [this comment](https://github.com/huggingface/diffusers/pull/11327#discussion_r2047659790).
169+
Specifying `use_duck_shape=False` instructs the compiler if it should use the same symbolic variable to represent input sizes that are the same. For more details, check out this [comment](https://github.com/huggingface/diffusers/pull/11327#discussion_r2047659790).
170+
171+
Not all models may benefit from dynamic compilation out of the box and may require changes. Refer to this [PR](https://github.com/huggingface/diffusers/pull/11297/) that improved the [`AuraFlowPipeline`] implementation to benefit from dynamic compilation.
167172

168-
All models might not benefit from this out of the box and may require changes. Refer to [this PR](https://github.com/huggingface/diffusers/pull/11297/) that improved the implementation of [`AuraFlowPipeline`] to benefit from compilation with `dynamic=True`. Feel free to open an issue if dynamic compilation doesn't work expected for a model inside Diffusers.
173+
Feel free to open an issue if dynamic compilation doesn't work as expected for a Diffusers model.
169174

170175
### Regional compilation
171176

0 commit comments

Comments
 (0)