How to input multimodal embeddings to an LLM

[RuABraun]

I'm using this function to try and pass multimodal embeddings: https://github.com/NVIDIA/TensorRT-LLM/blob/release/0.20/tensorrt_llm/llmapi/llm.py#L341

I'm assuming the fusing behaviour to be like here: https://github.com/NVIDIA/TensorRT-LLM/blob/release/0.20/tensorrt_llm/_torch/models/modeling_multimodal_utils.py#L31

Is my thinking accurate? Based on what nonsense my model outputs I assume no. I have not been able to find relevant docs or figure out where the C++ is

[Advait251206]

Your understanding is mostly correct, but there is an important distinction.

From the implementation and documentation, prompt_embedding_table is not automatically fused into the token embeddings in the same way as the PyTorch helper (fuse_input_embeds). Instead, it is treated as a block of prepended embeddings, analogous to prompt tuning or multimodal prefix embeddings. The multimodal README describes both max_multimodal_len and max_prompt_embedding_table_size as representing prepended/concatenated embeddings that are supplied to the LLM. The visual encoder output is flattened and passed through this mechanism.

So conceptually the runtime expects something like:

[prompt_embedding_table] + [token_embeddings]

rather than taking input_ids and replacing special placeholder tokens with your embeddings like fuse_input_embeds() does in the PyTorch path.

The fuse_input_embeds() helper is used by the PyTorch models before inference, whereas the TensorRT runtime receives the already-prepared prompt embeddings. They serve similar purposes, but they are not necessarily implemented identically.

If you're simply passing arbitrary embeddings through prompt_embedding_table while also providing normal input_ids, a few things need to line up:

• the embedding dimension must exactly match the LLM hidden size;
• the runtime must know how many prompt embeddings belong to each request;
• the text prompt/tokenization must be consistent with where those embeddings are expected to appear.

If any of those are incorrect, the model will happily run but typically produce nonsensical output, which sounds similar to what you're observing.

I also couldn't find much documentation describing the C++ execution path for this API. The release notes only mention that a multimodal embedding field was added to LlmRequest, but don't explain how it is consumed internally.

One question that would help narrow this down:

• Are these embeddings coming from one of the supported TensorRT-LLM vision encoders (e.g. LLaVA/Qwen-VL), or are they custom embeddings from your own encoder?

If they're custom embeddings, there may be additional assumptions (positioning, projector, special tokens, etc.) that the runtime does not enforce automatically.

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206