Skip to content

feat: Implements memory-safe temporal VAE slicing within the LTXPipeline#3760

Open
Ashitpatel001 wants to merge 3 commits into
openvinotoolkit:masterfrom
Ashitpatel001:feat/vae-temporal-slicer
Open

feat: Implements memory-safe temporal VAE slicing within the LTXPipeline#3760
Ashitpatel001 wants to merge 3 commits into
openvinotoolkit:masterfrom
Ashitpatel001:feat/vae-temporal-slicer

Conversation

@Ashitpatel001
Copy link
Copy Markdown

@Ashitpatel001 Ashitpatel001 commented Apr 24, 2026

Description

This PR implements memory-safe temporal VAE slicing within the LTXPipeline to prevent Out-Of-Memory (OOM) crashes during large video generation tasks. Previously, decoding full latent sequences at once caused massive activation memory spikes in the 3D VAE.

Key Implementations:

  • Integrated decode_temporal_slices directly into the monolithic LTXPipeline architecture to process latents in manageable chunks (default 32 frames) with a temporal overlap (default 8 frames).
  • Implemented blend_temporal_overlap using OpenVINO's ov::parallel_for to apply a linear cross-fade across overlapping frames. This prevents visual seams without causing CPU bottlenecks.
  • Added a dedicated integration test (test_text2video_pipeline.cpp) that utilizes the public Text2VideoPipeline API to verify anchored stride mathematics and frame alignment. (Note: I am currently updating the mock model initialization based on reviewer feedback so the test executes fully in CI instead of skipping).

Fixes #3690

Checklist:

  • This PR follows GenAI Contributing guidelines.
  • Tests have been updated or added to cover the new code.
  • This PR fully addresses the ticket.
  • I have made corresponding changes to the documentation.

@Ashitpatel001
Copy link
Copy Markdown
Author

Hi @likholat,

I saw you were assigned to this PR a few days ago. Just dropping a gentle ping to see if you might have some bandwidth to review the temporal VAE slicing implementation this week.

I am fully available to address any feedback or make architectural adjustments as soon as you are ready.
Thanks

Copy link
Copy Markdown
Contributor

@goyaladitya05 goyaladitya05 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description mentions making changes to the CMake file, but I don't see it in the files changed. Also, in its current state, the test always skips and verifies nothing.

Comment thread src/cpp/src/video_generation/ltx_pipeline.hpp
@Ashitpatel001
Copy link
Copy Markdown
Author

Ashitpatel001 commented May 14, 2026

Hi @goyaladitya05, thanks for the review!

  1. Unrelated changes: Apologies for that, my editor/formatter must have stripped that TODO by accident. I have restored it and pushed the fix.
  2. CMake mismatch: Actually I originally planned to add it manually, but realized GLOB in CMakeLists.txt picked up test_text2video_pipeline.cpp automatically. I have updated the PR description to remove that confusion.
  3. Skipping test: You are absolutely right. I hardcoded a dummy path to prevent CI hard-failures since we obviously don't want to load gigabytes of weights for a unit test. What is the preferred pattern in this repository for testing pipeline math/slicing?
    • Should I dynamically generate a temporary folder with mock config.json and model_index.json files during the test setup?
    • Or is there an existing dummy model fixture/CLI argument in the test suite that I should be using instead?

Let me know how you'd like me to mock the initialization, and I will update the test fixture immediately!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Implement Temporal VAE Slicing in LTXPipeline to prevent OOM on long videos

3 participants