[Feat] Support GLM-4.7 MTP in vLLM-ATOM plugin

[kliuae]

Motivation

This PR builds on top of the MTP framework in #557, adds MTP support to GLM-4.7 model for vLLM-ATOM.
Currently this PR contains changes from #557, and will be more concise once it gets upstreamed.

Technical Details

• Register Glm4MoeMTPModel
• Add glm4_moe_mtp modeling
• Fix RoPE double apply in mha when ATOM_ENABLE_QK_NORM_ROPE_CACHE_QUANT_FUSION=0

Test Plan

Accuracy test with lm_eval

Model: zai-org/GLM-4.7-FP8

Server command:

ATOM_DISABLE_VLLM_PLUGIN=0 \
ATOM_DISABLE_VLLM_PLUGIN_ATTENTION=0 \
VLLM_USE_V1=1 VLLM_ROCM_USE_AITER=1 \
  vllm serve zai-org/GLM-4.7-FP8 \
  -tp 8 \
  --max-num-seqs 1024 \
  --gpu-memory-utilization 0.9 \
  --no-enable-prefix-caching \
  --disable-uvicorn-access-log \
  --trust-remote-code \
  --load-format fastsafetensors \
  --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE"}' \
  --kv-cache-dtype fp8 \
  --tool-call-parser glm47 \
  --reasoning-parser glm45 \
  --enable-auto-tool-choice \
  --speculative-config.method mtp \
  --speculative-config.num_speculative_tokens 1

lm_eval command

lm_eval --model local-completions   --model_args model=zai-org/GLM-4.7-FP8,base_url=http://localhost:8000/v1/completions,num_concurrent=64,tokenized_requests=False  --tasks gsm8k --num_fewshot 5

Test Result

gsm8k

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
gsm8k	3	flexible-extract	5	exact_match	_	0.9439	_	0.0063
		strict-match	5	exact_match	_	0.9439	_	0.0063

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

[zejunchen-zejun]

Hi, @kliuae

Could you help resolve the conflicts here? Meanwhile the GLM 4.7 MTP should be added into the atom-vllm nightly and benchmark workflow

Thank you

[whx-sjtu]

You might need to skip this if glm4 mtp layer doesn't need to mask input_embeding according to positions.

[kliuae]

Thanks for pointing this out. Like deepseek mtp, glm4 mtp masks inputs at 0 so it can use this

[whx-sjtu]

You might also need to skip this if glm4 mtp layers doesn't need to share llm_head weights with main model.

[kliuae]

lm_head is not shared between mtp and the main model in glm4, but as its inner predictor doesn't carry lm_head, the syncing of lm_head won't be triggered. I think the current logic here doesn't affect glm4 mtp.