feat(models): add diffusers-based world-model generation backends (Wan 2.1/2.2, Cosmos, LTX-Video, MAGI-1, fal.ai)

[Luodian]

Summary

Adds a family of diffusers/API-based world-model generation backends to lmms-eval, covering text/image-to-video and video-to-video generation.

New model backends (registered in AVAILABLE_SIMPLE_MODELS):

model_id	class	backend
wan2_1_t2v	Wan2_1_T2V	Wan 2.1 text-to-video (diffusers WanPipeline)
wan2_1_t2i	Wan2_1_T2I	Wan 2.1 text-to-image (T2V pipeline, num_frames=1)
wan2_2	Wan2_2	Wan 2.2 image-to-video
wan2_2_t2v	Wan2_2_T2V	Wan 2.2 text-to-video
ltx_video	LTXVideo	LTX-Video text-to-video (diffusers LTXPipeline)
cosmos_wm	CosmosWorldModel	NVIDIA Cosmos Predict2/2.5/3 (nim / local / vllm_omni / transformers_reasoner / passthrough)
magi1_wm	Magi1WorldModel	Sand-AI MAGI-1 autoregressive T2V/I2V/V2V
generation_api	GenerationApi	fal.ai queue API (t2v / v2v), FAL_KEY env

A shared DiffusersWMBase consolidates per-rank device selection, lazy pipeline load, the dual-expert (transformer/transformer_2) device-placement fix, the UniPCMultistepScheduler sigma-device patch, cache-path hashing, and the DP generation loop.

Notes

• Defaults use public HF checkpoint ids (Wan-AI/*, Lightricks/LTX-Video, nvidia/Cosmos-*, sand-ai/MAGI-1).
• No new dependencies: diffusers, fal_client, and imageio/imageio-ffmpeg are lazy-imported with clear, actionable ImportError messages so the package imports fine without them.
• Data-parallel only: DiffusersWMBase enforces tp_size == 1, and MAGI-1 raises a clear error for model-parallel configs (cp_size*pp_size > 1) under the stock evaluator — its sharding is per-rank data-parallel, which would desync model-parallel collectives.
• The generation-eval task YAMLs these backends target (vbench, physics_iq, geneval, ...) ship separately in a follow-up PR series (their datasets are being staged for public HF hosting); the usage docstrings reference those task names ahead of that.

Review note

diffusers_wm_base.py inlines a minimal ParallelPlan dataclass. #1382 (gemini SDK) separately ships model_utils/parallel.py with the full version — if that lands first, the inline copy here can switch to importing it (one-line follow-up).

Review pass (2026-07-06)

Self-review follow-up commit, all backends:

• Per-sample failure isolation: cosmos's threaded dispatch no longer lets a single failed sample (reasoner HTTP error, corrupt visual) abort the whole run — failures come back as [GENERATION_FAILED] sentinels; MAGI-1's fallback image pre-processing likewise returns per-request [ERROR] sentinels.
• Multi-GPU wiring: cosmos and generation_api now set _rank/_world_size/device (cuda:{LOCAL_RANK} / Accelerator-based respectively) like the sibling backends, fixing pad/gather under accelerate launch/torchrun.
• MAGI-1 model-parallel guard: removed a dead "data-parallel rank routing" layer built on an evaluator API that doesn't exist, and replaced it with an explicit unsupported error for MP configs (single-GPU 4.5B and pure DP unaffected).
• Atomic downloads in generation_api (.part + os.replace), so an interrupted download can't be resume-hit as a finished video.
• Removed an unreachable generate_until_agentic path from wan2_2; unified all seed defaults on 42; stable sha1-based output ids for non-int doc_ids (was hash(), PYTHONHASHSEED-dependent); actionable ImportErrors for lazy diffusers/imageio imports; _nim_submit annotation corrected.

[pufanyi]

I'm currently doing #1303 and #1314, not sure if these 3 can work together