Skip to content

Fix the attention mask in ulysses SP for QwenImage #13278

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dg845 merged 6 commits into from
Mar 24, 2026
Merged

Fix the attention mask in ulysses SP for QwenImage #13278

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
832b4d2
fix mask in SP
zhtmike Mar 17, 2026
1e535f0
change the modification to qwen specific
zhtmike Mar 17, 2026
3e212e7
Merge branch 'main' into fix_sp
sayakpaul Mar 17, 2026
c76fc30
Merge branch 'main' into fix_sp
sayakpaul Mar 24, 2026
ea43309
drop xfail since qwen-image mask is fixed
zhtmike Mar 24, 2026
0a5c6ad
Merge branch 'main' into fix_sp
sayakpaul Mar 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/diffusers/models/transformers/transformer_qwenimage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -933,6 +933,7 @@ def forward(
batch_size, image_seq_len = hidden_states.shape[:2]
image_mask = torch.ones((batch_size, image_seq_len), dtype=torch.bool, device=hidden_states.device)
joint_attention_mask = torch.cat([encoder_hidden_states_mask, image_mask], dim=1)
joint_attention_mask = joint_attention_mask[:, None, None, :]
Copy link
Copy Markdown
Member

@sayakpaul sayakpaul Mar 17, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this okay for non-CP?

Copy link
Copy Markdown
Contributor Author

@zhtmike zhtmike Mar 17, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The image is same w/o this change.

block_attention_kwargs["attention_mask"] = joint_attention_mask

for index_block, block in enumerate(self.transformer_blocks):
Expand Down
1 change: 0 additions & 1 deletion tests/models/testing_utils/parallelism.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,7 +200,6 @@ def test_context_parallel_inference(self, cp_type, batch_size: int = 1):
f"Context parallel inference failed: {return_dict.get('error', 'Unknown error')}"
)

@pytest.mark.xfail(reason="Context parallel may not support batch_size > 1")
@pytest.mark.parametrize("cp_type", ["ulysses_degree", "ring_degree"], ids=["ulysses", "ring"])
def test_context_parallel_batch_inputs(self, cp_type):
self.test_context_parallel_inference(cp_type, batch_size=2)
Expand Down
Loading