[None][feat] Add AD custom model for GLM-4 MoE family (#231)

* [None][feat] Add AD custom model for GLM-4 MoE family (glm4_moe)

Add AutoDeploy custom model implementation for the GLM-4 MoE architecture
(model_type: glm4_moe), covering zai-org/GLM-4.6 and zai-org/GLM-4.7.

Key architectural features:
- GQA attention (96 Q heads, 8 KV heads, head_dim=128)
- Partial rotary embeddings (partial_rotary_factor=0.5)
- Per-head QK normalization (RMSNorm)
- MoE with sigmoid gating, group top-k routing (160 experts, top-8)
- First 3 layers dense, rest MoE with shared experts

Uses AD canonical ops: torch_attention, torch_rope_with_explicit_cos_sin,
torch_moe, torch_rmsnorm.

Includes hierarchical unit tests (19 tests): block, layer, full model
equivalence against HF reference, plus torch.export test.

Signed-off-by: Lucas Liebenwein <lliebenwein@nvidia.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

* [None][feat] Add glm4_moe registry config with reduced layers and expert export limit

Add glm4_moe.yaml config for GLM-4 MoE family models (GLM-4.6, GLM-4.7):
- num_hidden_layers: 5 (reduce from 92 for CI/testing)
- num_moe_experts_for_export: 2 (reduce from 160 for export)

Update registry entries for both GLM-4.6 and GLM-4.7 to use the new config.

Signed-off-by: Lucas Liebenwein <lliebenwein@nvidia.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

* [None][fix] Fix device mismatch in GLM4 MoE gate buffer for export

Fix e_score_correction_bias buffer to not hardcode dtype, and cast
to match scores tensor device/dtype in forward. This prevents a
device mismatch (cpu vs meta) during the MoE expert reduction
pre-trace in export_to_gm.

Signed-off-by: Lucas Liebenwein <lliebenwein@nvidia.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

* [None][feat] Use noaux_tc_op for GLM4 MoE gate and split registry configs

Address reviewer feedback:
- Use torch.ops.trtllm.noaux_tc_op for fused sigmoid + bias + group
  top-k routing instead of vanilla PyTorch
- Move num_hidden_layers out of glm4_moe.yaml, use num_hidden_layers_5.yaml
  separately in models.yaml entries
- Ensure e_score_correction_bias is passed as float32 to noaux_tc_op

Signed-off-by: Lucas Liebenwein <lliebenwein@nvidia.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

---------

Signed-off-by: Lucas Liebenwein <lliebenwein@nvidia.com>
Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>

Author: lucaslie

examples/auto_deploy/model_registry/configs/glm4_moe.yaml

@@ -0,0 +1,5 @@
+# Configuration for GLM-4 MoE family (GLM-4.6, GLM-4.7)
+# 160 experts - reduce for export
+transforms:
+  export_to_gm:
+    num_moe_experts_for_export: 2

examples/auto_deploy/model_registry/models.yaml

@@ -238,7 +238,7 @@ models:
   yaml_extra: ['kimi_k2.yaml']
 # --- GLM-4.7 (Dec 2025) ---
 - name: zai-org/GLM-4.7
-  yaml_extra: ['dashboard_default.yaml', 'world_size_8.yaml']
+  yaml_extra: ['dashboard_default.yaml', 'world_size_8.yaml', 'num_hidden_layers_5.yaml', 'glm4_moe.yaml']
 # --- DeepSeek V3.2 (Dec 2025) ---
 - name: deepseek-ai/DeepSeek-V3.2
   yaml_extra: ['dashboard_default.yaml', 'world_size_8.yaml', 'num_hidden_layers_5.yaml']
@@ -248,7 +248,7 @@ models:
   yaml_extra: ['dashboard_default.yaml', 'world_size_8.yaml', 'num_hidden_layers_5.yaml']
 # --- GLM-4.6 (Sep 2025) ---
 - name: zai-org/GLM-4.6
-  yaml_extra: ['dashboard_default.yaml', 'world_size_8.yaml']
+  yaml_extra: ['dashboard_default.yaml', 'world_size_8.yaml', 'num_hidden_layers_5.yaml', 'glm4_moe.yaml']
 # --- Qwen3-Next (Sep 2025) ---
 - name: Qwen/Qwen3-Next-80B-A3B-Instruct
   yaml_extra: ['dashboard_default.yaml', 'world_size_4.yaml', 'qwen3Next.yaml']

tensorrt_llm/_torch/auto_deploy/models/custom/__init__.py

@@ -5,6 +5,7 @@
 from .modeling_exaone import ExaoneForCausalLM
 from .modeling_gemma import GemmaADForCausalLM
 from .modeling_gemma2 import Gemma2ForCausalLM
+from .modeling_glm4_moe import Glm4MoeForCausalLM
 from .modeling_glm4_moe_lite import Glm4MoeLiteForCausalLM
 from .modeling_granite import GraniteForCausalLM
 from .modeling_granite_moe_hybrid import GraniteMoeHybridForCausalLM
@@ -35,6 +36,7 @@
     "ExaoneForCausalLM",
     "GemmaADForCausalLM",
     "Gemma2ForCausalLM",
+    "Glm4MoeForCausalLM",
     "Glm4MoeLiteForCausalLM",
     "GraniteForCausalLM",
     "GraniteMoeHybridForCausalLM",