Skip to content

Commit ecb6f53

Browse files
fix(wan): apply patchify fix in consumers, keep signature
Per @xiuyuan18 review: the bug should be fixed in forward (and the other tuple-unpacking consumers), not by changing patchify's return signature. This matches the official Wan 2.1 implementation, which also flattens x in forward, not in patchify. Reverts patchify back to returning just `x` (post-Conv3d tensor shape (B, dim, f, h, w)). Adds the flatten + (f, h, w) extraction at each tuple-unpacking consumer: - wan_video_dit.py: WanModel.forward - xdit_context_parallel.py: usp_dit_forward - wan_video.py: model_fn_wans2v (two sites: main x and ref_latents) WanToDance callers in wan_video.py (lines 1412/1414) continue to work unchanged because they always wanted the raw post-Conv3d tensor and do their own flattening downstream. Pre-existing S2V double-embedding pattern at model_fn_wans2v (caller pre-runs patch_embedding, then patchify re-runs patch_embedding internally) is preserved as-is — out of scope for this PR. Refs #1063
1 parent 5d8a8bf commit ecb6f53

3 files changed

Lines changed: 15 additions & 13 deletions

File tree

diffsynth/models/wan_video_dit.py

Lines changed: 5 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -498,13 +498,7 @@ def patchify(self, x: torch.Tensor, control_camera_latents_input: Optional[torch
498498
y_camera = self.control_adapter(control_camera_latents_input)
499499
x = [u + v for u, v in zip(x, y_camera)]
500500
x = x[0].unsqueeze(0)
501-
# After Conv3d the tensor is (B, dim, f, h, w). The forward expects
502-
# (x, (f, h, w)) where x has been flattened to (B, f*h*w, dim) ready
503-
# for the transformer blocks; the grid size tuple drives the RoPE
504-
# frequency cat and the unpatchify rearrange at the end.
505-
f, h, w = x.shape[-3], x.shape[-2], x.shape[-1]
506-
x = x.flatten(2).transpose(1, 2)
507-
return x, (f, h, w)
501+
return x
508502

509503
def unpatchify(self, x: torch.Tensor, grid_size: torch.Tensor):
510504
return rearrange(
@@ -533,8 +527,10 @@ def forward(self,
533527
clip_embdding = self.img_emb(clip_feature)
534528
context = torch.cat([clip_embdding, context], dim=1)
535529

536-
x, (f, h, w) = self.patchify(x)
537-
530+
x = self.patchify(x)
531+
f, h, w = x.shape[-3], x.shape[-2], x.shape[-1]
532+
x = x.flatten(2).transpose(1, 2)
533+
538534
freqs = torch.cat([
539535
self.freqs[0][:f].view(f, 1, 1, -1).expand(f, h, w, -1),
540536
self.freqs[1][:h].view(1, h, 1, -1).expand(f, h, w, -1),

diffsynth/pipelines/wan_video.py

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1658,11 +1658,15 @@ def model_fn_wans2v(
16581658

16591659
# x and s2v_pose_latents
16601660
s2v_pose_latents = torch.zeros_like(x) if s2v_pose_latents is None else s2v_pose_latents
1661-
x, (f, h, w) = dit.patchify(dit.patch_embedding(x) + dit.cond_encoder(s2v_pose_latents))
1661+
x = dit.patchify(dit.patch_embedding(x) + dit.cond_encoder(s2v_pose_latents))
1662+
f, h, w = x.shape[-3], x.shape[-2], x.shape[-1]
1663+
x = x.flatten(2).transpose(1, 2)
16621664
seq_len_x = seq_len_x_global = x.shape[1] # global used for unified sequence parallel
16631665

16641666
# reference image
1665-
ref_latents, (rf, rh, rw) = dit.patchify(dit.patch_embedding(origin_ref_latents))
1667+
ref_latents = dit.patchify(dit.patch_embedding(origin_ref_latents))
1668+
rf, rh, rw = ref_latents.shape[-3], ref_latents.shape[-2], ref_latents.shape[-1]
1669+
ref_latents = ref_latents.flatten(2).transpose(1, 2)
16661670
grid_sizes = dit.get_grid_sizes((f, h, w), (rf, rh, rw))
16671671
x = torch.cat([x, ref_latents], dim=1)
16681672
# mask

diffsynth/utils/xfuser/xdit_context_parallel.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -81,8 +81,10 @@ def usp_dit_forward(self,
8181
clip_embdding = self.img_emb(clip_feature)
8282
context = torch.cat([clip_embdding, context], dim=1)
8383

84-
x, (f, h, w) = self.patchify(x)
85-
84+
x = self.patchify(x)
85+
f, h, w = x.shape[-3], x.shape[-2], x.shape[-1]
86+
x = x.flatten(2).transpose(1, 2)
87+
8688
freqs = torch.cat([
8789
self.freqs[0][:f].view(f, 1, 1, -1).expand(f, h, w, -1),
8890
self.freqs[1][:h].view(1, h, 1, -1).expand(f, h, w, -1),

0 commit comments

Comments
 (0)