My llava1.5 7b can get output, but how to set it up to get the same output as transformers

[bleedingfight]

I can use the transformer interface to input text and images for output。The parameters passed in by tensorrt-llm are not the same. How to ensure that the same model can obtain same results that are consistent with transformers or tensorrt-llm.trt model.generate()'s param not same with transformers LlavaForConditionalGeneration.Is there a one-to-one corresponding parameter pair?

[Advait251206]

This is a very common issue when comparing Hugging Face Transformers vs TensorRT-LLM outputs for LLaVA. Even when using the same model weights, getting bitwise or even token-identical outputs is non-trivial because the two runtimes differ in decoding, sampling, and sometimes even preprocessing.

---

1. Why outputs differ in the first place

Even if the model is identical, differences come from:

🔴 A. Tokenization + prompt formatting

Transformers LLaVA typically uses:

• apply_chat_template()
• image token insertion order
• special tokens (<image>, <|im_start|> etc.)

TensorRT-LLM uses:

• a custom prompt builder
• sometimes different system prompt injection
• different image embedding insertion logic

👉 Even small prompt differences = different outputs

---

🔴 B. Sampling implementation differences

Transformers:

model.generate(
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1
)

TensorRT-LLM:

• has its own SamplingConfig
• uses different internal kernels
• may differ in:
  • RNG seeding
  • softmax precision
  • top-k/top-p ordering

---

🔴 C. Numerical differences (FP16 / FP32 / fused kernels)

TensorRT-LLM uses:

• fused attention kernels
• optimized GEMM kernels
• different layernorm implementations

👉 small numeric drift → different token choices

---

2. Is there a 1-to-1 mapping of parameters?

❌ No exact 1-to-1 mapping exists

But there is a functional equivalence mapping.

---

3. Approximate parameter mapping

Transformers → TensorRT-LLM

Hugging Face generate()	TensorRT-LLM equivalent
max_new_tokens	max_new_tokens
temperature	temperature
top_p	top_p
top_k	top_k
repetition_penalty	repetition_penalty
do_sample=True/False	sampling vs greedy in TRT
num_beams	beam_width (if enabled in engine)
eos_token_id	configured in tokenizer / engine
pad_token_id	engine config

---

4. Critical differences for LLaVA specifically

LLaVA adds extra complexity:

🔴 Image processing must match exactly

You must ensure:

• same image resize (e.g., 336×336 or 224×224)
• same normalization mean/std
• same patch ordering
• same vision tower output dtype

Even a slight mismatch → different text output

---

🔴 Image token placement

Transformers:

<image> + text prompt

TensorRT-LLM:

• must match exact embedding injection position
• otherwise attention context shifts

---

5. How to get closest possible matching outputs

✔ Step 1: Disable randomness

Make both deterministic:

Transformers:

do_sample=False

TensorRT-LLM:

temperature = 0
top_k = 1

---

✔ Step 2: Match prompt exactly

Ensure identical:

• chat template
• system prompt
• image token format

Even whitespace matters.

---

✔ Step 3: Use greedy decoding in both

This is the only practical way to get near-identical outputs.

---

✔ Step 4: Match precision

Try:

• Transformers: float16
• TensorRT-LLM: fp16

Avoid mixing fp32/fp16.

---

✔ Step 5: Fix RNG seed (if sampling)

If sampling:

• set same seed in both frameworks

But note:
👉 TRT-LLM RNG ≠ PyTorch RNG internally

So still not guaranteed identical.

---

6. Why perfect matching is not realistic

Even if everything is identical:

• different kernel fusion order
• different reduction precision
• different attention softmax scheduling

👉 leads to divergence after a few tokens

This is expected in LLM inference engines.

---

7. What you can achieve in practice

Goal	Achievable?
identical tokens (greedy)	⚠️ sometimes
similar outputs	✔ yes
identical sampling outputs	❌ rarely
identical logits	❌ no

---

8. Best practice recommendation

If your goal is validation:

👉 Use Transformers as ground truth
👉 Use TRT-LLM for performance deployment

And compare at:

• qualitative level
• or top-k similarity

---

Bottom line

• ❌ No strict 1:1 parameter mapping exists
• ✔ Most generation parameters do have equivalents
• ⚠ Exact output matching is not guaranteed due to kernel + numerical differences
• ✔ Greedy decoding + identical preprocessing gives closest match

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206