Skip to content

I can not get the final mp4 when using multi gpus #586

Description

@kaihekaihe

I can not get the final mp4 when using multi gpus,but it works well when using single gpu
/workspace/huggingface/ComfyUI/.venv/bin/python /workspace/huggingface/ComfyUI/custom_nodes/ComfyUI-SeedVR2_VideoUpscaler/inference_cli.py /workspace/huggingface/ComfyUI/custom_nodes/ComfyUI-SeedVR2_VideoUpscaler/fisheys170.mp4 --output /workspace/huggingface/ComfyUI/output/fisheys170_frames_single --output_format png --cuda_device 0,1 --resolution 1280 --batch_size 5 --chunk_size 32 --temporal_overlap 2 --vae_encode_tiled --vae_decode_tiled --vae_encode_tile_size 512 --vae_decode_tile_size 512 --dit_offload_device cpu --vae_offload_device cpu --color_correction none --debug
📊 Initial CUDA memory: 36.61GB free / 39.38GB total
/workspace/huggingface/ComfyUI/.venv/lib/python3.10/site-packages/requests/init.py:113: RequestsDependencyWarning: urllib3 (2.6.3) or chardet (7.2.0)/charset_normalizer (3.4.6) doesn't match a supported version!
warnings.warn(
⚠️ SeedVR2 optimizations check: SageAttention ❌ | Flash Attention ❌ | Triton ✅
💡 Optional: pip install sageattention flash-attn
🔧 Conv3d workaround active: PyTorch 2.9.1, cuDNN 91002 (fixing VAE 3x memory bug)
flash_attn is installed but failed to import: No module named 'flash_attn.flash_attn_interface'. Falling back to native PyTorch attention.

███████╗███████╗███████╗██████╗ ██╗ ██╗██████╗ ██████╗ ███████╗
██╔════╝██╔════╝██╔════╝██╔══██╗██║ ██║██╔══██╗ ╚════██╗ ██╔════╝
███████╗█████╗ █████╗ ██║ ██║██║ ██║██████╔╝ █████╔╝ ███████╗
╚════██║██╔══╝ ██╔══╝ ██║ ██║╚██╗ ██╔╝██╔══██╗ ██╔═══╝ ╚════██║
███████║███████╗███████╗██████╔╝ ╚████╔╝ ██║ ██║ ███████╗ ██╗ ███████║
╚══════╝╚══════╝╚══════╝╚═════╝ ╚═══╝ ╚═╝ ╚═╝ ╚══════╝ ╚═╝ ╚══════╝
💻 CLI · v2.5.24 © ByteDance Seed · NumZ · AInVFX
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[08:00:49.873] ℹ️ OS: Ubuntu 24.04 | GPU: NVIDIA A100-PCIE-40GB (39GB)
[08:00:49.873] ℹ️ Python: 3.10.19 | PyTorch: 2.9.1+cu128 | FlashAttn: ✗ | SageAttn: ✗ | Triton: ✓
[08:00:49.873] ℹ️ CUDA: 12.8 | cuDNN: 91002
[08:00:49.873]
[08:00:49.873] 🔧 Arguments:
[08:00:49.873] input: /workspace/huggingface/ComfyUI/custom_nodes/ComfyUI-SeedVR2_VideoUpscaler/fisheys170.mp4
[08:00:49.873] output: /workspace/huggingface/ComfyUI/output/fisheys170_frames_single
[08:00:49.873] output_format: png
[08:00:49.873] video_backend: opencv
[08:00:49.873] use_10bit: False
[08:00:49.873] model_dir: None
[08:00:49.873] dit_model: seedvr2_ema_3b_fp8_e4m3fn.safetensors
[08:00:49.873] resolution: 1280
[08:00:49.873] max_resolution: 0
[08:00:49.873] batch_size: 5
[08:00:49.873] uniform_batch_size: False
[08:00:49.873] seed: 42
[08:00:49.873] skip_first_frames: 0
[08:00:49.873] load_cap: 0
[08:00:49.873] chunk_size: 32
[08:00:49.873] prepend_frames: 0
[08:00:49.873] temporal_overlap: 2
[08:00:49.873] color_correction: none
[08:00:49.873] input_noise_scale: 0.0
[08:00:49.873] latent_noise_scale: 0.0
[08:00:49.873] cuda_device: 0,1
[08:00:49.873] dit_offload_device: cpu
[08:00:49.873] vae_offload_device: cpu
[08:00:49.873] tensor_offload_device: cpu
[08:00:49.873] blocks_to_swap: 0
[08:00:49.873] swap_io_components: False
[08:00:49.873] vae_encode_tiled: True
[08:00:49.873] vae_encode_tile_size: 512
[08:00:49.873] vae_encode_tile_overlap: 128
[08:00:49.873] vae_decode_tiled: True
[08:00:49.873] vae_decode_tile_size: 512
[08:00:49.873] vae_decode_tile_overlap: 128
[08:00:49.873] tile_debug: false
[08:00:49.874] attention_mode: sdpa
[08:00:49.874] compile_dit: False
[08:00:49.874] compile_vae: False
[08:00:49.874] compile_backend: inductor
[08:00:49.874] compile_mode: default
[08:00:49.874] compile_fullgraph: False
[08:00:49.874] compile_dynamic: False
[08:00:49.874] compile_dynamo_cache_size_limit: 64
[08:00:49.874] compile_dynamo_recompile_limit: 128
[08:00:49.874] cache_dit: False
[08:00:49.874] cache_vae: False
[08:00:49.874] debug: True
[08:00:49.874] 🖥️ CUDA_VISIBLE_DEVICES: Not set (all)
[08:00:49.874] 🖥️ torch.cuda.device_count(): 4
[08:00:49.874] 🖥️ Using device index 0 inside script (mapped to selected GPU)
[08:00:49.874] 🖥️ Using devices: ['0', '1']
[08:00:49.874] 🔧 DiT model found: ./models/SEEDVR2/seedvr2_ema_3b_fp8_e4m3fn.safetensors
[08:00:49.874] 🔧 DiT model already validated (cache): ./models/SEEDVR2/seedvr2_ema_3b_fp8_e4m3fn.safetensors
[08:00:49.874] 🔧 VAE model found: ./models/SEEDVR2/ema_vae_fp16.safetensors
[08:00:49.874] 🔧 VAE model already validated (cache): ./models/SEEDVR2/ema_vae_fp16.safetensors
[08:00:49.875] 🎬 Processing video: fisheys170.mp4
[08:00:49.875] ℹ️ Requested output format: png
[08:00:49.891] ℹ️ Video info: 271 frames, 1280x720, 60.00 FPS
[08:00:49.891] ℹ️ Streaming mode: chunks of 32 frames, overlap=2
📊 Initial CUDA memory: 38.97GB free / 39.38GB total
📊 Initial CUDA memory: 36.14GB free / 39.38GB total
/workspace/huggingface/ComfyUI/.venv/lib/python3.10/site-packages/requests/init.py:113: RequestsDependencyWarning: urllib3 (2.6.3) or chardet (7.2.0)/charset_normalizer (3.4.6) doesn't match a supported version!
warnings.warn(
/workspace/huggingface/ComfyUI/.venv/lib/python3.10/site-packages/requests/init.py:113: RequestsDependencyWarning: urllib3 (2.6.3) or chardet (7.2.0)/charset_normalizer (3.4.6) doesn't match a supported version!
warnings.warn(
flash_attn is installed but failed to import: No module named 'flash_attn.flash_attn_interface'. Falling back to native PyTorch attention.
flash_attn is installed but failed to import: No module named 'flash_attn.flash_attn_interface'. Falling back to native PyTorch attention.
[08:00:57.694] 🎬 GPU 0: frames 0-138 (138 frames, chunks of 32)
[08:00:57.755] 🎬 GPU 1: frames 136-271 (135 frames, chunks of 32)
[08:00:58.020]
[08:00:58.020] 🎬 [GPU 0] Chunk 1/5: 32 new + 0 context frames
[08:00:58.020]
[08:00:58.047]
[08:00:58.047] 🎬 [GPU 1] Chunk 1/5: 32 new + 0 context frames
[08:00:58.047]
[08:00:58.056] 🔧 Generation context initialized: DiT=cuda:0, VAE=cuda:0, Offload=[DiT offload=cpu, VAE offload=cpu, Tensor offload=cpu], LOCAL_RANK=0
[08:00:58.056] 🎯 Unified compute dtype: torch.bfloat16 across entire pipeline for maximum performance
[08:00:58.056] 🏃 Configuring inference runner...
[08:00:58.056] 🏃 Creating new runner: DiT=seedvr2_ema_3b_fp8_e4m3fn.safetensors, VAE=ema_vae_fp16.safetensors
[08:00:58.091] 🔧 Generation context initialized: DiT=cuda:0, VAE=cuda:0, Offload=[DiT offload=cpu, VAE offload=cpu, Tensor offload=cpu], LOCAL_RANK=0
。。。。。
[08:10:47.342] ⚡ └─ VAE decode: 5.66s
[08:10:47.342] ⚡ └─ Decoded batch 8: 5.67s
[08:10:47.342] ⚡ └─ VAE decode: 5.66s
[08:10:47.342] ⚡ └─ VAE moved to CUDA:0: 0.08s
[08:10:47.342] ⚡ └─ VAE moved to CPU: 0.06s
[08:10:47.343] 📊 After phase 3 (VAE decoding):
[08:10:47.343] 📊 [VRAM] 0.01GB allocated / 7.72GB reserved / Peak: 5.60GB / 28.30GB free / 39.38GB total
[08:10:47.343] 📊 [RAM] 7.96GB process / 42.57GB others / 452.98GB free / 503.51GB total
[08:10:47.343] 📊 Memory changes: VRAM -0.47GB, RAM +0.01GB
[08:10:47.343] 📊 Resetting VRAM peak memory statistics
[08:10:47.343]
[08:10:47.343] ━━━━━━━━ Phase 4: Post-processing ━━━━━━━━
[08:10:47.343] 📹 Post-processing batch 1/4
[08:10:47.343] 🔄 Moving sample_1 from CPU to CUDA:0 (post-processing)
[08:10:47.351] 📹 Color correction disabled (set to none)
[08:10:47.352] 🔄 Moving sample_1_final from CUDA:0 to CPU (writing processed result to final_video)
[08:10:47.359] 📹 Post-processing batch 2/4
[08:10:47.359] 🔄 Moving sample_2 from CPU to CUDA:0 (post-processing)
[08:10:47.364] 📹 Color correction disabled (set to none)
[08:10:47.364] 🔄 Moving sample_2_final from CUDA:0 to CPU (writing processed result to final_video)
[08:10:47.369] 📹 Post-processing batch 3/4
[08:10:47.369] 🔄 Moving sample_3 from CPU to CUDA:0 (post-processing)
[08:10:47.374] 📹 Color correction disabled (set to none)
[08:10:47.374] 🔄 Moving sample_3_final from CUDA:0 to CPU (writing processed result to final_video)
[08:10:47.380] 📹 Post-processing batch 4/4
[08:10:47.380] 🔄 Moving sample_4 from CPU to CUDA:0 (post-processing)
[08:10:47.382] 📹 Color correction disabled (set to none)
[08:10:47.382] 🔄 Moving sample_4_final from CUDA:0 to CPU (writing processed result to final_video)
[08:10:47.384] 🎬 Output assembled: 12 frames (20 computed with 2 padding + 6 overlap removed), Resolution: 2274x1280px, Channels: RGB
[08:10:47.384] ⚡ Phase 4: Post-processing complete: 0.04s
[08:10:47.384] ⚡ └─ Post-processed batch 1: 0.02s
[08:10:47.384] ⚡ └─ Post-processed batch 8: 0.01s
[08:10:47.384] ⚡ └─ Post-processed batch 5: 0.01s
[08:10:47.384] ⚡ └─ Post-processed batch 6: 0.01s
[08:10:47.384] ⚡ └─ Post-processed batch 7: 0.01s
[08:10:47.384] ⚡ └─ Post-processed batch 10: 0.01s
[08:10:47.384] ⚡ └─ Post-processed batch 9: 0.01s
[08:10:47.384] ⚡ └─ Post-processed batch 3: 0.01s
[08:10:47.385] 📊 After phase 4 (Post-processing):
[08:10:47.385] 📊 [VRAM] 0.01GB allocated / 7.72GB reserved / Peak: 0.09GB / 28.30GB free / 39.38GB total
[08:10:47.385] 📊 [RAM] 7.96GB process / 42.56GB others / 452.98GB free / 503.51GB total
[08:10:47.385] 📊 Resetting VRAM peak memory statistics
[08:10:47.419] 🧹 Clearing memory caches (deep)...
Then it gets nothing

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions