Commit 33f785b
Add Helios-14B Video Generation Pipelines (#13208)
* [1/N] add helios
* fix test
* make fix-copies
* change script path
* fix cus script
* update docs
* fix documented check
* update links for docs and examples
* change default config
* small refactor
* add test
* Update src/diffusers/models/transformers/transformer_helios.py
Co-authored-by: YiYi Xu <yixu310@gmail.com>
* remove register_buffer for _scale_cache
* fix non-cuda devices error
* remove "handle the case when timestep is 2D"
* refactor HeliosMultiTermMemoryPatch and process_input_hidden_states
* Update src/diffusers/pipelines/helios/pipeline_helios.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* Update src/diffusers/models/transformers/transformer_helios.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* Update src/diffusers/pipelines/helios/pipeline_helios.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* fix calculate_shift
* Update src/diffusers/pipelines/helios/pipeline_helios.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* rewritten `einops` in pure `torch`
* fix: pass patch_size to apply_schedule_shift instead of hardcoding
* remove the logics of 'vae_decode_type'
* move some validation into check_inputs()
* rename helios scheduler & merge all into one step()
* add some details to doc
* move dmd step() logics from pipeline to scheduler
* change to Python 3.9+ style type
* fix NoneType error
* refactor DMD scheduler's set_timestep
* change rope related vars name
* fix stage2 sample
* fix dmd sample
* Update src/diffusers/models/transformers/transformer_helios.py
Co-authored-by: YiYi Xu <yixu310@gmail.com>
* Update src/diffusers/models/transformers/transformer_helios.py
Co-authored-by: YiYi Xu <yixu310@gmail.com>
* remove redundant & refactor norm_out
* Update src/diffusers/pipelines/helios/pipeline_helios.py
Co-authored-by: YiYi Xu <yixu310@gmail.com>
* change "is_keep_x0" to "keep_first_frame"
* use a more intuitive name
* refactor dynamic_time_shifting
* remove use_dynamic_shifting args
* remove usage of UniPCMultistepScheduler
* separate stage2 sample to HeliosPyramidPipeline
* Update src/diffusers/models/transformers/transformer_helios.py
Co-authored-by: YiYi Xu <yixu310@gmail.com>
* Update src/diffusers/models/transformers/transformer_helios.py
Co-authored-by: YiYi Xu <yixu310@gmail.com>
* Update src/diffusers/models/transformers/transformer_helios.py
Co-authored-by: YiYi Xu <yixu310@gmail.com>
* Update src/diffusers/models/transformers/transformer_helios.py
Co-authored-by: YiYi Xu <yixu310@gmail.com>
* fix transformer
* use a more intuitive name
* update example script
* fix requirements
* remove redudant attention mask
* fix
* optimize pipelines
* make style .
* update TYPE_CHECKING
* change to use torch.split
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* derive memory patch sizes from patch_size multiples
* remove some hardcoding
* move some checks into check_inputs
* refactor sample_block_noise
* optimize encoding chunks logits for v2v
* use num_history_latent_frames = sum(history_sizes)
* Update src/diffusers/pipelines/helios/pipeline_helios.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* remove redudant optimized_scale
* Update src/diffusers/pipelines/helios/pipeline_helios_pyramid.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* use more descriptive name
* optimize history_latents
* remove not used "num_inference_steps"
* removed redudant "pyramid_num_stages"
* add "is_cfg_zero_star" and "is_distilled" to HeliosPyramidPipeline
* remove redudant
* change example scripts name
* change example scripts name
* correct docs
* update example
* update docs
* Update tests/models/transformers/test_models_transformer_helios.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* Update tests/models/transformers/test_models_transformer_helios.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* separate HeliosDMDScheduler
* fix numerical stability issue:
* Update src/diffusers/schedulers/scheduling_helios_dmd.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* Update src/diffusers/schedulers/scheduling_helios_dmd.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* Update src/diffusers/schedulers/scheduling_helios_dmd.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* Update src/diffusers/schedulers/scheduling_helios_dmd.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* Update src/diffusers/schedulers/scheduling_helios_dmd.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* remove redudant
* small refactor
* remove use_interpolate_prompt logits
* simplified model test
* fallbackt to BaseModelTesterConfig
* remove _maybe_expand_t2v_lora_for_i2v
* fix HeliosLoraLoaderMixin
* update docs
* use randn_tensor for test
* fix doc typo
* optimize code
* mark torch.compile xfail
* change paper name
* Make get_dummy_inputs deterministic using self.generator
* Set less strict threshold for test_save_load_float16 test for Helios pipeline
* make style and make quality
* Preparation for merging
* add torch.Generator
* Fix HeliosPipelineOutput doc path
* Fix Helios related (optimize docs & remove redudant) (#13210)
* fix docs
* remove redudant
* remove redudant
* fix group offload
* Removed fixes for group offload
---------
Co-authored-by: yuanshenghai <yuanshenghai@bytedance.com>
Co-authored-by: Shenghai Yuan <140951558+SHYuanBest@users.noreply.github.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>
Co-authored-by: SHYuanBest <shyuan-cs@hotmail.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>1 parent 06ccde9 commit 33f785b
File tree
33 files changed
+5655
-2
lines changed- docs/source
- en
- api
- loaders
- models
- pipelines
- schedulers
- using-diffusers
- zh
- using-diffusers
- src/diffusers
- loaders
- models
- transformers
- pipelines
- helios
- schedulers
- utils
- tests
- lora
- models/transformers
- pipelines/helios
33 files changed
+5655
-2
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
194 | 194 | | |
195 | 195 | | |
196 | 196 | | |
| 197 | + | |
| 198 | + | |
197 | 199 | | |
198 | 200 | | |
199 | 201 | | |
| |||
350 | 352 | | |
351 | 353 | | |
352 | 354 | | |
| 355 | + | |
| 356 | + | |
353 | 357 | | |
354 | 358 | | |
355 | 359 | | |
| |||
625 | 629 | | |
626 | 630 | | |
627 | 631 | | |
628 | | - | |
629 | 632 | | |
630 | 633 | | |
631 | 634 | | |
| |||
674 | 677 | | |
675 | 678 | | |
676 | 679 | | |
| 680 | + | |
| 681 | + | |
677 | 682 | | |
678 | 683 | | |
679 | 684 | | |
| |||
745 | 750 | | |
746 | 751 | | |
747 | 752 | | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
748 | 757 | | |
749 | 758 | | |
750 | 759 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
23 | 23 | | |
24 | 24 | | |
25 | 25 | | |
| 26 | + | |
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
| |||
86 | 87 | | |
87 | 88 | | |
88 | 89 | | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
89 | 94 | | |
90 | 95 | | |
91 | 96 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
0 commit comments