With the new TensorRT-LLM 0.9.0, ModelConfig in tesnorrt_llm.runtime.generation now has new args max_batch_size, max_beam_width - how do we set these?

[digitalmonkey]

The new version of Tensorrt_llm introduced new arguments to ModelConfig - max_batch_size, max_beam_width.
How do we set these? Specifically, this breaks the trt_llama_api.py script that is used to build the rag on windows. I'm trying to run that on Ubuntu and wish to stay up to date with tensorrt_llm's latest release.

[Advait251206]

In TensorRT-LLM 0.9.0, the addition of max_batch_size and max_beam_width to ModelConfig is mainly about making runtime constraints explicit at the engine configuration level, rather than introducing new independent tuning knobs.

---

1. Where do these values come from?

You generally do NOT manually invent them at runtime.

They come from one of three places:

✔ Option A: From the engine/build config (recommended)

When you build the engine:

trtllm-build \
  --max_batch_size X \
  --max_beam_width Y

Those values are embedded into the serialized engine and later reused automatically in runtime ModelConfig. :contentReference[oaicite:0]{index=0}

---

✔ Option B: Passed via high-level APIs (LLM API)

If using Python LLM:

build_config = BuildConfig()
build_config.max_batch_size = 8
build_config.max_beam_width = 4

llm = LLM(model_dir, build_config=build_config)

Then runtime config inherits it. :contentReference[oaicite:1]{index=1}

---

✔ Option C: Auto-loaded from engine (most common runtime path)

If you're using:

ModelRunnerCpp.from_dir(engine_dir)

then:

• max_batch_size
• max_beam_width

are read directly from the engine config file and passed into ModelConfig internally. :contentReference[oaicite:2]{index=2}

So in this case:

You usually do NOT set them manually at all.

---

2. What do these parameters actually mean?

🔹 max_batch_size

This is:

Maximum number of simultaneous requests the engine is allowed to handle.

It defines:

• KV cache allocation
• scheduler limits
• memory reservation

⚠️ It is NOT the runtime batch size you “choose per request”
It is the upper bound baked into the engine

---

🔹 max_beam_width

This is:

Maximum beam size supported by the engine for beam search decoding.

It defines:

• how many candidate sequences can be tracked per request
• KV cache duplication factor
• sampling memory layout

If set to 1, beam search is effectively disabled (greedy / sampling only).

---

3. Why did 0.9.0 introduce this into ModelConfig?

Before 0.9.0:

• these values lived only in build-time config
• runtime assumed them implicitly

After 0.9.0:

TensorRT-LLM made runtime fully explicit and validated, meaning:

• runtime cannot exceed engine limits
• avoids silent mismatches between build/runtime
• supports stricter executor validation

---

4. Why your RAG script broke (trt_llama_api.py)

Old scripts often did:

ModelConfig(vocab_size=..., num_layers=...)

But now require:

ModelConfig(
    max_batch_size=engine_max_batch,
    max_beam_width=engine_max_beam,
    vocab_size=...
)

So legacy code breaks because:

It assumed runtime didn’t need to know engine constraints.

---

5. Practical fix for your case

If you're updating old scripts:

Minimal safe approach:

Extract from engine config:

engine_config = engine.config

model_config = ModelConfig(
    max_batch_size=engine_config.build_config.max_batch_size,
    max_beam_width=engine_config.build_config.max_beam_width,
    vocab_size=...,
    num_layers=...,
    ...
)

---

6. Rule of thumb

Value	Who sets it	When
max_batch_size	build time (trtllm-build)	engine creation
max_beam_width	build time	engine creation
ModelConfig fields	runtime loader	engine execution

---

Bottom line

• You typically do not manually tune these at runtime
• They are inherited from engine build config
• Their purpose is safety + consistency enforcement in runtime API (0.9+)
• Old scripts break because they assumed runtime was independent of build constraints

---

If this answer helped or pointed you in the right direction, I'd appreciate it if you could mark it as the accepted answer so it's easier for others with the same issue to find.

Also, if you found my contribution useful, I'd appreciate it if you could check out my GitHub profile, follow me, and star any repositories you find interesting.

GitHub: https://github.com/Advait251206