Could you please provide an example of a C++ runtime that supports multimodal inference? Thanks!

[zhangts20]

[Advait251206]

At the moment, I don't believe there is a complete C++ multimodal inference example comparable to the Python examples provided in TensorRT-LLM.

Most of the officially documented multimodal workflows (e.g., LLaVA, Qwen-VL, VILA, and similar models) are demonstrated using the Python APIs, where the vision encoder, projector, prompt preparation, and LLM are orchestrated together. The C++ runtime primarily exposes the inference engine and execution APIs, but the higher-level multimodal preprocessing pipeline is not demonstrated in a standalone C++ example.

If your goal is to build a native C++ multimodal application, the general pipeline would be:

1. Run the vision encoder (and any projector, if required) to produce visual embeddings.
2. Prepare the prompt embeddings or multimodal inputs in the format expected by the LLM.
3. Submit those embeddings (along with any text tokens) to the TensorRT-LLM runtime.
4. Decode the generated tokens from the LLM output.

The exact implementation details depend on the multimodal architecture (e.g., LLaVA, Qwen-VL, Pixtral, etc.), since each has its own prompt formatting and embedding fusion strategy.

One question for the TensorRT-LLM team:

• Is there an official C++ end-to-end multimodal example (or one planned) that demonstrates the complete pipeline—from image preprocessing and vision encoder inference through to passing the embeddings into the Executor/Runtime API? I think such an example would be very helpful for users building native C++ applications.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206