How do I use idefics2-8b with TensorRT LLM?

[Iven2132]

I'm a big fan of @kaiyux's work on TensorRT LLM, Thank you very much Kaiyu! I'm currently trying to figure out How to deploy an API with idefics2-8b, the latest model launched out-performs LLava. Are there any plans to add support for this model anytime soon?

Also Kaiyu, How can I connect with you? I want to chat with you.

[Advait251206]

Short answer: you can’t currently run IDEFICS2-8B end-to-end in TensorRT-LLM, and there is no official converter or runtime support for it yet. You can only use TensorRT-LLM for the decoder LLM part (if you isolate it), but not the full multimodal model.

---

1. Why IDEFICS2-8B is not supported

IDEFICS2 is not a “plain LLM” like LLaMA or GPT. It is:

• Vision encoder (e.g. SigLIP / ViT variant)
• Perceiver / multimodal projector
• LLM decoder (based on Mistral-style architecture in IDEFICS2)
• Cross-modal attention / fusion logic

TensorRT-LLM currently supports:

✔ Decoder-only LLMs (LLaMA, Mistral, GPT-like)
✔ KV cache + attention + MLP optimization
✔ Tensor/pipeline parallel inference

TensorRT-LLM does NOT support:

❌ Vision encoders
❌ Multimodal fusion blocks
❌ Perceiver resamplers / cross-attention stacks
❌ Hugging Face multimodal forward() graphs

So IDEFICS2 is outside the supported graph boundary.

---

2. What can be done today

You have 3 realistic options:

---

✔ Option A: Hybrid deployment (recommended)

Split the model:

Image → Vision encoder (PyTorch / HF / ONNX / TRT)
      → Multimodal projector (PyTorch or TRT)
      → Text embeddings
      → TensorRT-LLM (LLM only)
      → Output tokens

TensorRT-LLM is used only for:

the decoder (Mistral-like backbone inside IDEFICS2)

---

✔ Option B: Run full model outside TRT-LLM

Use:

• Hugging Face Transformers
• vLLM (if supported)
• or custom PyTorch inference

This is the simplest path for IDEFICS2 today.

---

✔ Option C: Partial TensorRT conversion

You can try:

• Export vision tower → TensorRT engine
• Export LLM → TensorRT-LLM engine
• Keep multimodal glue in Python

But you must manually implement:

• tensor formatting
• embedding injection
• token-image alignment logic

---

3. Why TensorRT-LLM does not support it yet

The core issue is:

TensorRT-LLM expects a pure causal decoder graph

IDEFICS2 introduces:

• cross-modal attention
• dynamic sequence composition
• non-transformer blocks (Perceiver pooling)
• modality routing logic

These break assumptions like:

• fixed KV cache layout
• single-token autoregressive decoding graph
• static attention masks

---

4. Is support planned?

As of now:

• NVIDIA TensorRT-LLM roadmap has focused on:
  • LLaMA family
  • Mistral
  • GPT-style models
  • some multimodal extensions (LLaVA-style, partially)

But:

IDEFICS2-specific support is not part of the standard TensorRT-LLM model zoo yet.

So if support comes, it will likely be:

• via a LLaVA-like abstraction layer
• not native full IDEFICS2 graph conversion

---

5. Practical recommendation

If your goal is deployment:

Best current architecture:

• Use IDEFICS2 in Hugging Face or vLLM for multimodal part
• Optionally extract only the LLM for TensorRT-LLM acceleration

Why:

• vision + fusion is still rapidly evolving
• TRT-LLM is optimized for stable decoder-only graphs
• multimodal support is still partial and model-specific

---

6. About contacting the developer

I can’t help facilitate direct contact with specific individuals, but:

• NVIDIA TensorRT-LLM discussions happen via:
  • GitHub issues/discussions
  • NVIDIA Developer forums

That’s the correct channel for model support requests and roadmap questions.

---

Bottom line

• ❌ No direct IDEFICS2-8B TensorRT-LLM support today
• ✔ You can only use TRT-LLM for the decoder portion
• ✔ Full multimodal pipeline must stay hybrid or PyTorch-based
• ⚠ Native support would require major graph-level multimodal integration changes

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206