feat(datasets): lmms_video_utils video backend for LLaVA-OneVision-2 codec stream

[kcz358]

Summary

Plumbs the new lmms-video-utils codec-stream video frontend into lmms-engine so LLaVA-OneVision-2 can train on the same patch-positions / per-canvas-timestamp format the model expects, but without depending on the closed-source cv-preinfer binary.

Changes

• datasets.config: video_backend literal gains "lmms_video_utils".
• datasets.multimodal_mixin: adds load_video_lmms_video_utils which returns (canvases, fps, CodecVideoOutput). Decoder backend / device read from extra_kwargs.video_decode_backend / extra_kwargs.video_decode_device (defaults pyav / cpu; "cuda" resolves to cuda:LOCAL_RANK).
• datasets.iterable.llava_ov2_iterable_dataset (new): registers llava_ov2_iterable dataset that propagates the CodecVideoOutput metadata into the processor.
• datasets.processor.llava_onevision2_processor: process() learns a video_metadata kwarg. When supplied, the processor bypasses OV2's bundled video_processor and constructs pixel_values_videos / video_grid_thw / patch_positions / frame_timestamps directly from the codec output, preserving real source-frame coordinates instead of falling back to arange(T). The frame-sampling path is unchanged.
• pyproject.toml: adds [video] extra (lmms-video-utils[all]>=0.1.0) and rolls it into [all].

Example config

dataset_config:
  dataset_type: llava_ov2_iterable
  video_backend: lmms_video_utils
  extra_kwargs:
    video_decode_backend: torchcodec
    video_decode_device: cuda

Validation

• 3-step max_steps run on LLaVA-Video-178K/llava_video_0_30_s_cap_oe.parquet with the OV-2 8B checkpoint completed end-to-end.
• DataLoader must run with num_workers=0 when video_decode_device=cuda (forked workers can't share the parent's CUDA context). Multi-worker GPU decoding would need a multiprocessing_context='spawn' toggle, which is a separate change.

Backward compatibility

Existing video backends (decord, qwen_vl_utils, qwen_omni_utils) and all other dataset / processor paths are untouched. Base install is unchanged; users must opt into pip install .[video] (or .[all]) to pull in lmms-video-utils.

[Luodian]

@kcz358 can we extract an abstract class, and name it to something like codec native input.

also for processor, ov2 is a specification in both data and processor.

[Luodian]

I think there should not be max_frames, more like we set a visual tokens budget?

[Luodian]

maybe should not consider qwen_to_ours, we can convert all data offline, prepare all things needed using ffprobe.

[kcz358]

@Luodian Refactored: extracted the codec video loading logic into a reusable CodecVideoLoadingMixin (in datasets/codec_video_mixin.py, alongside multimodal_mixin). LlavaOv2IterableDataset now inherits this mixin instead of inlining the codec orchestration.

The backend implementation (load_video_lmms_video_utils) stays in MultiModalDataLoadingMixin; the new mixin only handles the dataset-level orchestration (collecting canvases + their CodecVideoOutput metadata across a message list). This way any future codec-aware dataset can reuse it. (714856f)