Skip to content

Fix Flux IP-Adapter masks, CFG embeds, and modular helper coupling#13708

Open
taivu1998 wants to merge 1 commit intohuggingface:mainfrom
taivu1998:tdv/issue-13584-flux-fixes
Open

Fix Flux IP-Adapter masks, CFG embeds, and modular helper coupling#13708
taivu1998 wants to merge 1 commit intohuggingface:mainfrom
taivu1998:tdv/issue-13584-flux-fixes

Conversation

@taivu1998
Copy link
Copy Markdown

What this PR does

Addresses #13584 by fixing several Flux pipeline and model issues found in the Flux review:

  • Applies ip_adapter_masks in FluxIPAdapterAttnProcessor instead of accepting and silently ignoring them.
  • Enables true CFG when users pass precomputed negative_prompt_embeds and negative_pooled_prompt_embeds to Flux img2img, inpaint, and ControlNet pipelines.
  • Validates positive and negative prompt embedding shape mismatches in base Flux and Flux Kontext.
  • Fixes Flux ControlNet img2img dimension validation so it checks divisibility by vae_scale_factor * 2.
  • Hardens Flux Prior Redux batch validation for tensor/NumPy image batches, prompt lists, prompt embeddings, and both prompt scale lists.
  • Removes hidden Flux modular pipeline dependencies on classic Flux pipeline private helpers and QwenImage helpers by adding Flux-local modular helper utilities.

Root cause

The affected Flux code paths had drifted across pipeline variants:

  • Flux attention already accepted ip_adapter_masks, but the processor never consumed them.
  • Some true-CFG-capable pipelines only checked negative_prompt, even though their input validation allowed negative embeddings.
  • Base Flux/Kontext validation missed a shape check that sibling Flux pipelines already had.
  • ControlNet img2img used height % self.vae_scale_factor * 2, which Python parses as (height % self.vae_scale_factor) * 2.
  • Prior Redux only counted list images when validating batch-coupled prompt and scale inputs.
  • Modular Flux reused private helpers from the classic Flux pipeline and a helper from QwenImage, creating cross-family coupling.

Validation

Ran the focused regression suite for the changed behavior:

uvx ruff format src/diffusers/models/transformers/transformer_flux.py src/diffusers/modular_pipelines/flux/before_denoise.py src/diffusers/modular_pipelines/flux/decoders.py src/diffusers/modular_pipelines/flux/encoders.py src/diffusers/modular_pipelines/flux/inputs.py src/diffusers/modular_pipelines/flux/pipeline_helpers.py src/diffusers/pipelines/flux/pipeline_flux.py src/diffusers/pipelines/flux/pipeline_flux_controlnet.py src/diffusers/pipelines/flux/pipeline_flux_controlnet_image_to_image.py src/diffusers/pipelines/flux/pipeline_flux_img2img.py src/diffusers/pipelines/flux/pipeline_flux_inpaint.py src/diffusers/pipelines/flux/pipeline_flux_kontext.py src/diffusers/pipelines/flux/pipeline_flux_prior_redux.py tests/models/transformers/test_models_transformer_flux.py tests/modular_pipelines/flux/test_modular_pipeline_flux.py tests/pipelines/controlnet_flux/test_controlnet_flux.py tests/pipelines/controlnet_flux/test_controlnet_flux_img2img.py tests/pipelines/flux/test_pipeline_flux.py tests/pipelines/flux/test_pipeline_flux_img2img.py tests/pipelines/flux/test_pipeline_flux_inpaint.py tests/pipelines/flux/test_pipeline_flux_kontext.py tests/pipelines/flux/test_pipeline_flux_redux.py
uvx ruff check src/diffusers/models/transformers/transformer_flux.py src/diffusers/modular_pipelines/flux/before_denoise.py src/diffusers/modular_pipelines/flux/decoders.py src/diffusers/modular_pipelines/flux/encoders.py src/diffusers/modular_pipelines/flux/inputs.py src/diffusers/modular_pipelines/flux/pipeline_helpers.py src/diffusers/pipelines/flux/pipeline_flux.py src/diffusers/pipelines/flux/pipeline_flux_controlnet.py src/diffusers/pipelines/flux/pipeline_flux_controlnet_image_to_image.py src/diffusers/pipelines/flux/pipeline_flux_img2img.py src/diffusers/pipelines/flux/pipeline_flux_inpaint.py src/diffusers/pipelines/flux/pipeline_flux_kontext.py src/diffusers/pipelines/flux/pipeline_flux_prior_redux.py tests/models/transformers/test_models_transformer_flux.py tests/modular_pipelines/flux/test_modular_pipeline_flux.py tests/pipelines/controlnet_flux/test_controlnet_flux.py tests/pipelines/controlnet_flux/test_controlnet_flux_img2img.py tests/pipelines/flux/test_pipeline_flux.py tests/pipelines/flux/test_pipeline_flux_img2img.py tests/pipelines/flux/test_pipeline_flux_inpaint.py tests/pipelines/flux/test_pipeline_flux_kontext.py tests/pipelines/flux/test_pipeline_flux_redux.py
git diff --check
uv run --python 3.10 python -m py_compile src/diffusers/models/transformers/transformer_flux.py src/diffusers/modular_pipelines/flux/before_denoise.py src/diffusers/modular_pipelines/flux/decoders.py src/diffusers/modular_pipelines/flux/encoders.py src/diffusers/modular_pipelines/flux/inputs.py src/diffusers/modular_pipelines/flux/pipeline_helpers.py src/diffusers/pipelines/flux/pipeline_flux.py src/diffusers/pipelines/flux/pipeline_flux_controlnet.py src/diffusers/pipelines/flux/pipeline_flux_controlnet_image_to_image.py src/diffusers/pipelines/flux/pipeline_flux_img2img.py src/diffusers/pipelines/flux/pipeline_flux_inpaint.py src/diffusers/pipelines/flux/pipeline_flux_kontext.py src/diffusers/pipelines/flux/pipeline_flux_prior_redux.py tests/models/transformers/test_models_transformer_flux.py tests/modular_pipelines/flux/test_modular_pipeline_flux.py tests/pipelines/controlnet_flux/test_controlnet_flux.py tests/pipelines/controlnet_flux/test_controlnet_flux_img2img.py tests/pipelines/flux/test_pipeline_flux.py tests/pipelines/flux/test_pipeline_flux_img2img.py tests/pipelines/flux/test_pipeline_flux_inpaint.py tests/pipelines/flux/test_pipeline_flux_kontext.py tests/pipelines/flux/test_pipeline_flux_redux.py
uv run --python 3.10 python -m pytest tests/models/transformers/test_models_transformer_flux.py tests/pipelines/flux/test_pipeline_flux_redux.py tests/modular_pipelines/flux/test_modular_pipeline_flux.py tests/pipelines/flux/test_pipeline_flux.py tests/pipelines/flux/test_pipeline_flux_kontext.py tests/pipelines/controlnet_flux/test_controlnet_flux_img2img.py tests/pipelines/flux/test_pipeline_flux_img2img.py tests/pipelines/flux/test_pipeline_flux_inpaint.py tests/pipelines/controlnet_flux/test_controlnet_flux.py -k "ip_adapter_masks_are_applied or ip_adapter_masks_validate_num_images or ip_adapter_no_mask_keeps_3d_hidden_state_support or FluxReduxFastTests or modular_blocks_do_not_import or latent_helpers_roundtrip or negative_prompt_embeds_shape_mismatch or dimension_check_uses_packed_latent_multiple or true_cfg_with_negative_prompt_embeds"

Targeted pytest result:

15 passed, 497 deselected

Notes

This PR adds focused fast regression coverage for the fixed behavior. The broader slow/nightly real-checkpoint coverage discussed in the issue is intentionally left for a separate CI-focused PR because it spans many Flux variants and checkpoint/runtime choices.

@taivu1998 taivu1998 marked this pull request as ready for review May 11, 2026 03:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant