[Video Generation] Add VAE encoder support to AutoencoderKLLTXVideo

[goyaladitya05]

This PR implements AutoencoderKLLTXVideo::encode(), enabling Image-to-Video workflows where a conditioning frame is encoded into latent space and passed to the diffusion pipeline.

This is phase 1 of Image-to-Video support in LTX Video Generation pipeline.

Changes

• VAE encoder now compiles and runs. compile() and reshape() had // TODO: for img2video. Both are now implemented.
• encode(video, generator) - takes a [B, C, F, H, W] video tensor, runs the encoder model, and returns a normalized latent ready for the diffusion transformer. Handles both model output variants:
  • latent_parameters - samples z from the predicted distribution
  • latent_sample - uses the output directly (no sampling needed)
• Added 7 tests which cover construction, error messages, output shape, determinism with the same seed, and variation across seeds. Tests that require the encoder skip if vae_encoder is not present in the test model.

Testing

Encode decode roundtrip on an image (512×512, CPU):

Step	Value
Input	[1, 3, 1, 512, 512]
Latent	[1, 128, 1, 16, 16] (32× spatial compression)
Output	[1, 1, 512, 512, 3]

Checklist:

• This PR follows GenAI Contributing guidelines.
• Tests have been updated or added to cover the new code.
• This PR fully addresses the ticket.
• I have made corresponding changes to the documentation. No necessary changes.

[Copilot]

Pull request overview

This PR adds VAE encoder support to the LTX video autoencoder, enabling encoding of a conditioning frame/video into latent space for upcoming Image-to-Video workflows.

Changes:

• Implemented encoder compilation/reshape and added AutoencoderKLLTXVideo::encode() with support for latent_parameters sampling and latent_sample passthrough + normalization.
• Exposed encode() in Python bindings.
• Added Python tests covering encoder construction, error paths, output shape, and seed determinism/variation.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.

File	Description
tests/python_tests/test_video_generation.py	Adds encoder-focused tests (construction, error messages, shape, determinism/seed variation).
src/python/py_video_generation_models.cpp	Exposes AutoencoderKLLTXVideo.encode() via pybind11 with GIL release.
src/cpp/src/video_generation/models/autoencoder_kl_ltx_video.cpp	Implements encoder compile/reshape and encode() including sampling + latent normalization.
src/cpp/include/openvino/genai/video_generation/autoencoder_kl_ltx_video.hpp	Adds public encode() declaration and stores encoder output name in class state.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

[goyaladitya05]

Interesting. This should also affect image generation when models are exported in full fp16 and bf16 using GPU. I'll verify this once.

[goyaladitya05]

#3865

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

[goyaladitya05]

cc @sgonorov @likholat

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

src/cpp/src/video_generation/models/autoencoder_kl_ltx_video.cpp:295

• encode() asserts the latent is f32 and normalizes via latent.data() in-place. On GPU (or with inference-precision hints) the encoder output/latent may be f16/bf16, which will currently fail at runtime. To make encode() robust across devices, consider either converting the sampled/copied latent to f32 before normalization, or performing normalization in a dtype-generic way similar to denormalize_latents() in video_generation/ltx_pipeline.hpp.

    OPENVINO_ASSERT(shape.size() == 5, "Encoder output expected to be [B, C, F, H, W]");
    OPENVINO_ASSERT(latent.get_element_type() == ov::element::f32,
        "Latent normalization requires f32, got ", latent.get_element_type());
    const size_t B = shape[0], C = shape[1], spatial = shape[2] * shape[3] * shape[4];

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

src/cpp/src/video_generation/models/autoencoder_kl_ltx_video.cpp:295

• encode() asserts that the produced latent must be f32 and then normalizes it via latent.data(). On GPU / fp16 or bf16 exports this will currently fail even though inference succeeds, and it blocks Image-to-Video usage on those configurations. Consider making the normalization path dtype-generic (f16/bf16/f32) or explicitly converting the latent to f32 before normalization, similar to the robustness work tracked in #3865.

    const ov::Shape shape = latent.get_shape();
    OPENVINO_ASSERT(shape.size() == 5, "Encoder output expected to be [B, C, F, H, W]");
    OPENVINO_ASSERT(latent.get_element_type() == ov::element::f32,
        "Latent normalization requires f32, got ", latent.get_element_type());
    const size_t B = shape[0], C = shape[1], spatial = shape[2] * shape[3] * shape[4];

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

src/cpp/src/video_generation/models/autoencoder_kl_ltx_video.cpp:295

• encode() hard-requires the latent tensor to be f32 for normalization (latent.get_element_type() == f32). This makes encode() unusable for fp16/bf16 IR exports or GPU inference-precision hints, even though the decode path already supports f16/bf16 (denormalize_latents in ltx_pipeline.hpp is dtype-generic). Consider normalizing in a dtype-generic way (e.g., via OpenVINO ops/broadcast like denormalize_latents does) or explicitly converting the latent to f32 before normalization and documenting the output dtype.

    OPENVINO_ASSERT(shape.size() == 5, "Encoder output expected to be [B, C, F, H, W]");
    OPENVINO_ASSERT(latent.get_element_type() == ov::element::f32,
        "Latent normalization requires f32, got ", latent.get_element_type());
    const size_t B = shape[0], C = shape[1], spatial = shape[2] * shape[3] * shape[4];

[sgonorov]

Can also refactor tests a little bit - extract more fixtures for less boilerplate, but overall looks good.