Skip to content

Commit 81c4cb3

Browse files
csy2077claude
andcommitted
Fix per-frame timestep allocation for video2world mode in CosmosPredict2
In video2world (image2world) mode, conditioning frames were receiving the same noisy timestep as all other frames. This caused the transformer to treat the clean conditioning frame as a fully-noised input, breaking temporal coherence and causing severe quality degradation. Fix: after replacing conditioning frames in model_input, expand t to shape (B, T) and zero out timesteps for frames where condition_mask=1, signaling to the model that those frames are already clean (t=0). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 123e6a2 commit 81c4cb3

1 file changed

Lines changed: 7 additions & 0 deletions

File tree

fastgen/networks/cosmos_predict2/network.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1352,6 +1352,13 @@ def forward(
13521352
conditioning_latents_full = conditioning_latents
13531353
# Replace conditioning frames
13541354
model_input = conditioning_latents_full * condition_mask_C + x_t * (1 - condition_mask_C)
1355+
# Per-frame timesteps: assign timestep 0 to conditioning (clean) frames so the
1356+
# model knows they are noise-free and should not be denoised. Without this, the
1357+
# model treats the clean first frame as a fully-noised frame, causing incoherent
1358+
# video2world (image2world) generation.
1359+
t_expanded = t.unsqueeze(1).expand(B, T)
1360+
mask_B_T = condition_mask[:, 0, :, 0, 0] # (B, T)
1361+
t = t_expanded * (1 - mask_B_T)
13551362

13561363
model_outputs = self.transformer(
13571364
x_B_C_T_H_W=model_input,

0 commit comments

Comments
 (0)