Can a slightly modified multimodal model achieve inference through the tensorrt llm pipeline?

[bleedingfight]

I tried replacing the llm of the llava model with llama3, and now I want to try using tensorrt llm to complete the entire inference. I have a few questions that I don't quite understand and would like to ask for advice:

1. The official llava model's llm is llama2, and the tensorrt conversion uses the hf model. Therefore, it is necessary to first rename the partial key of the official llava's llama weight, then fill 32000 to 32064 to generate the hf model, and then convert the hf model to the trt model. I want to know if I still need to fill in data to 32064 if I replace it with llama3?
2. My architecture consists of three parts: vision_tower, mmprojector（changed and add some conv op）, and llama3. My network is not a transformer native supported architecture (huggingface Auto), and I have my own Forword. I just want to briefly explain that replacing vision_tower and mmprojector with trt engine can achieve the use of trt llm to complete the entire algorithm pipeline

[Advait251206]

Short answer: yes, but only in a “composable pipeline” sense — not as a fully automatic TensorRT-LLM conversion.

TensorRT-LLM can only directly handle the decoder-only LLM part (Llama / GPT-style transformer). Everything around it (vision tower, projector, custom forward logic) is not automatically supported unless you explicitly wrap/convert it yourself.

---

1. Do you still need to pad 32000 → 32064 for Llama 3?

No — that step was specific to LLaMA 2 + certain LLaVA HF conversions.

For LLaMA 3:

• Use the native tokenizer vocabulary size
• Do not artificially pad embeddings
• Ensure your HF checkpoint already matches LLaMA 3 config (vocab_size, embed_tokens, etc.)

If you force LLaMA 2-style padding on LLaMA 3, you can actually break:

• embedding alignment
• rotary embedding scaling assumptions
• TensorRT-LLM weight mapping

So: skip padding unless a specific converter explicitly requires it (very unlikely for LLaMA 3).

---

2. Can your modified multimodal model run fully through TensorRT-LLM?

Key point:

TensorRT-LLM is not a full multimodal framework

It only accelerates:

• token embedding
• transformer decoder layers
• KV cache + sampling
• tensor/pipeline parallel LLM execution

It does NOT natively support:

• vision_tower
• custom CNN-based mm_projector
• arbitrary PyTorch forward()

---

3. What is the correct architecture pattern?

Your described stack:

image → vision_tower → mm_projector → Llama3 → text output

The supported way is:

✔ Option A (recommended): Hybrid pipeline

Run components separately:

Vision tower (PyTorch or TRT engine)
        ↓
mm_projector (PyTorch or TRT engine)
        ↓
LLM (TensorRT-LLM engine)
        ↓
token generation

TensorRT-LLM only replaces the Llama3 forward pass + decoding loop

---

✔ Option B: Fully custom orchestration (advanced)

You can:

• export vision_tower → TensorRT engine
• export mm_projector → TensorRT engine
• feed output embeddings into TensorRT-LLM runtime API

But YOU must implement:

• scheduling
• memory transfers
• synchronization
• embedding injection interface

TensorRT-LLM will NOT automatically chain them.

---

4. Your “modified mm_projector with conv ops”

This is the key limitation:

• TensorRT-LLM does NOT compile arbitrary conv-heavy modules
• You would need to either:
  • export it via TensorRT directly (ONNX → TRT)
  • or keep it in PyTorch

So your design choice becomes:

Component	Best runtime
vision_tower	PyTorch / TRT
mm_projector	PyTorch or TensorRT
Llama3	TensorRT-LLM

---

5. Practical recommendation

For fastest success:

1. Keep vision + projector in PyTorch first
2. Convert only Llama3 to TensorRT-LLM
3. Pass projected embeddings into TRT-LLM via runtime API

Then optimize later if needed.

---

Bottom line

• ❌ You cannot run the entire modified multimodal forward() directly in TensorRT-LLM
• ❌ LLaVA-style graph is not auto-converted
• ✔ You can run the LLM part efficiently with TensorRT-LLM
• ✔ You can build a hybrid multimodal pipeline around it
• ⚠ Full end-to-end TRT conversion requires manual graph splitting + custom engineering

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206