Skip to content

Commit 9653450

Browse files
[None][chore] Update AD model list (#14686)
Signed-off-by: Yueh-Ting (eop) Chen <yuehtingc@nvidia.com> Signed-off-by: tcherckez-nvidia <127761168+tcherckez-nvidia@users.noreply.github.com> Co-authored-by: Yueh-Ting (eop) Chen <yuehtingc@nvidia.com>
1 parent 66262cf commit 9653450

6 files changed

Lines changed: 407 additions & 319 deletions

File tree

examples/auto_deploy/model_registry/configs/granite_4.0_h_small.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@
77
# MoE experts distributed via EP across 4 GPUs. This avoids the TP incompatibility
88
# where n_groups=1 makes Mamba B/C SSM dimensions non-shardable.
99
attn_backend: flashinfer
10-
max_batch_size: 16
10+
cuda_graph_config:
11+
max_batch_size: 16
1112
kv_cache_config:
1213
free_gpu_memory_fraction: 0.5
1314
transforms:

examples/auto_deploy/model_registry/configs/granite_4.0_micro.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
# Configuration for granite-4.0-micro (and similar GraniteMoeHybrid models)
55
# Uses flashinfer attention backend (trtllm backend produces garbled output).
66
attn_backend: flashinfer
7-
max_batch_size: 16
7+
cuda_graph_config:
8+
max_batch_size: 16
89
kv_cache_config:
910
free_gpu_memory_fraction: 0.5

examples/auto_deploy/model_registry/configs/granite_4.0_tiny_preview.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,8 @@
55
# Uses flashinfer attention backend (trtllm backend produces garbled output with muP scaling)
66
# dtype is missing from config for this model, so we provide it explicitly.
77
attn_backend: flashinfer
8-
max_batch_size: 16
8+
cuda_graph_config:
9+
max_batch_size: 16
910
kv_cache_config:
1011
free_gpu_memory_fraction: 0.5
1112
model_kwargs:

examples/auto_deploy/model_registry/configs/hunyuan_a13b_ep.yaml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,8 @@
22
# SPDX-License-Identifier: Apache-2.0
33

44
# Hunyuan A13B MoE: replicate attention/dense layers and shard routed experts with EP.
5-
max_batch_size: 16
5+
cuda_graph_config:
6+
max_batch_size: 16
67
kv_cache_config:
78
free_gpu_memory_fraction: 0.5
89
transforms:

examples/auto_deploy/model_registry/configs/qwen3Next.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ attn_backend: torch
77
max_seq_len: 4096
88
max_num_tokens: 4096
99
max_batch_size: 64
10+
cuda_graph_config:
11+
max_batch_size: 64
1012
enable_chunked_prefill: true
1113
model_factory: AutoModelForCausalLM
1214
# disable_overlap_scheduler: false

0 commit comments

Comments
 (0)