-
Notifications
You must be signed in to change notification settings - Fork 234
Experiment with MiniMax-M2.7 EAGLE on B200 / 在 B200 上实验 MiniMax-M2.7 EAGLE #2289
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
cquil11
wants to merge
10
commits into
main
Choose a base branch
from
agent/experimental-minimax-m2.7-eagle
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
6a59ae8
feat: add MiniMax-M2.7 EAGLE B200 experiment [skip-sweep]
cquil11 ade2752
chore: link MiniMax-M2.7 experiment PR
cquil11 931d9e8
perf: prefetch MiniMax checkpoints from Lustre
cquil11 1401c61
fix: leave B200 headroom for EAGLE graphs
cquil11 6110522
ci: route MiniMax experiment to idle B200 partition
cquil11 9bbb9d9
fix: retain authorized B200 Slurm partition
cquil11 e8e80c8
ci: allow MiniMax jobs to backfill B200 nodes
cquil11 ed24358
fix: cap MiniMax AgentX replay context
cquil11 4c43c23
fix: drain slow MiniMax AgentX responses
cquil11 5209dd9
chore: merge main into MiniMax EAGLE experiment
cquil11 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
5 changes: 5 additions & 0 deletions
5
benchmarks/single_node/agentic/minimaxm2.7_fp4_b200_sglang_mtp.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| #!/usr/bin/env bash | ||
| set -Eeuo pipefail | ||
|
|
||
| export SCENARIO_TYPE=agentic-coding | ||
| source "$(dirname "${BASH_SOURCE[0]}")/../fixed_seq_len/minimaxm2.7_fp4_b200_sglang_mtp.sh" |
218 changes: 218 additions & 0 deletions
218
benchmarks/single_node/fixed_seq_len/minimaxm2.7_fp4_b200_sglang_mtp.sh
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,218 @@ | ||
| #!/usr/bin/env bash | ||
| set -Eeuo pipefail | ||
|
|
||
| # MiniMax-M2.7 NVFP4 on B200 with the public full-vocabulary EAGLE3 draft. | ||
| # This entrypoint serves both fixed-sequence and AgentX wrappers so the target, | ||
| # draft, speculative settings, and chat behavior cannot drift between them. | ||
|
|
||
| SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" | ||
| INFERENCEX_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)" | ||
| source "$INFERENCEX_ROOT/benchmarks/benchmark_lib.sh" | ||
|
|
||
| check_env_vars \ | ||
| MODEL \ | ||
| TP \ | ||
| EP_SIZE \ | ||
| CONC \ | ||
| PORT \ | ||
| PRECISION \ | ||
| RANDOM_RANGE_RATIO \ | ||
| RESULT_FILENAME | ||
|
|
||
| if [[ "$PRECISION" != "fp4" || "$EP_SIZE" != "1" ]]; then | ||
| echo "MiniMax-M2.7 EAGLE3 expects NVFP4 with pure tensor parallelism, got precision=$PRECISION EP=$EP_SIZE" >&2 | ||
| exit 1 | ||
| fi | ||
| if [[ "$TP" != "4" && "$TP" != "8" ]]; then | ||
| echo "MiniMax-M2.7 EAGLE3 supports the configured TP4/TP8 search space, got TP=$TP" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| readonly TARGET_MODEL="nvidia/MiniMax-M2.7-NVFP4" | ||
| readonly TARGET_REVISION="e79701cb1f9dce8fe5395b9ed2b20170beebecde" | ||
| readonly DRAFT_MODEL="asherszhang/MiniMax-M2.7-EAGLE3-draft-vocab200k" | ||
| readonly DRAFT_REVISION="252b54f7d05a5d0db7734563ae10e2e6a81d6a7d" | ||
|
|
||
| if [[ "$MODEL" != "$TARGET_MODEL" && "$MODEL" != /* ]]; then | ||
| echo "Unexpected target model: $MODEL (expected $TARGET_MODEL)" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [[ -n "${MODEL_PATH:-}" ]]; then | ||
| TARGET_MODEL_PATH="$MODEL_PATH" | ||
| elif [[ "$MODEL" == /* ]]; then | ||
| TARGET_MODEL_PATH="$MODEL" | ||
| else | ||
| TARGET_MODEL_PATH="$TARGET_MODEL" | ||
| fi | ||
|
|
||
| if [[ "$TARGET_MODEL_PATH" == /* ]]; then | ||
| MODEL_ROOT=$(dirname "$TARGET_MODEL_PATH") | ||
| DRAFT_MODEL_PATH="${DRAFT_MODEL_PATH:-$MODEL_ROOT/MiniMax-M2.7-EAGLE3-draft-vocab200k}" | ||
| mkdir -p "$TARGET_MODEL_PATH" "$DRAFT_MODEL_PATH" | ||
| exec 8>"$MODEL_ROOT/.minimax-m2.7-eagle3-stage.lock" | ||
| flock -w 7200 8 | ||
| if [[ ! -f "$TARGET_MODEL_PATH/config.json" || \ | ||
| ! -f "$TARGET_MODEL_PATH/model.safetensors.index.json" || \ | ||
| $(find "$TARGET_MODEL_PATH" -maxdepth 1 -name 'model-*-of-00015.safetensors' | wc -l) -ne 15 ]]; then | ||
| hf download "$TARGET_MODEL" --revision "$TARGET_REVISION" --local-dir "$TARGET_MODEL_PATH" | ||
| fi | ||
| if [[ ! -f "$DRAFT_MODEL_PATH/config.json" || ! -f "$DRAFT_MODEL_PATH/model.safetensors" ]]; then | ||
| hf download "$DRAFT_MODEL" --revision "$DRAFT_REVISION" --local-dir "$DRAFT_MODEL_PATH" | ||
| fi | ||
| flock -u 8 | ||
| else | ||
| hf download "$TARGET_MODEL" --revision "$TARGET_REVISION" | ||
| hf download "$DRAFT_MODEL" --revision "$DRAFT_REVISION" | ||
| DRAFT_MODEL_PATH="$DRAFT_MODEL" | ||
| fi | ||
|
|
||
| python3 - "$TARGET_MODEL_PATH" "$DRAFT_MODEL_PATH" <<'PY' | ||
| import json | ||
| import sys | ||
| from pathlib import Path | ||
|
|
||
| target_path, draft_path = map(Path, sys.argv[1:]) | ||
| target = json.loads((target_path / "config.json").read_text()) | ||
| draft = json.loads((draft_path / "config.json").read_text()) | ||
| quant = target.get("quantization_config", {}) | ||
| if quant.get("quant_algo") != "NVFP4" or quant.get("quant_method") != "modelopt": | ||
| raise SystemExit(f"Unexpected NVFP4 target configuration: {quant}") | ||
| if target.get("vocab_size") != 200064: | ||
| raise SystemExit(f"Unexpected target vocabulary: {target.get('vocab_size')}") | ||
| if draft.get("architectures") != ["LlamaForCausalLMEagle3"]: | ||
| raise SystemExit(f"Unexpected EAGLE3 architecture: {draft.get('architectures')}") | ||
| if draft.get("vocab_size") != 200064 or draft.get("num_hidden_layers") != 1: | ||
| raise SystemExit( | ||
| f"Unexpected EAGLE3 draft contract: vocab={draft.get('vocab_size')} " | ||
| f"layers={draft.get('num_hidden_layers')}" | ||
| ) | ||
| print(f"Validated target={target_path} draft={draft_path}") | ||
| PY | ||
|
|
||
| if [[ -n "${SLURM_JOB_ID:-}" ]]; then | ||
| echo "JOB $SLURM_JOB_ID running on ${SLURMD_NODENAME:-unknown}" | ||
| fi | ||
| nvidia-smi | ||
|
|
||
| IS_AGENTIC=false | ||
| if [[ "${SCENARIO_TYPE:-fixed-seq-len}" == "agentic-coding" ]]; then | ||
| IS_AGENTIC=true | ||
| check_env_vars DURATION RESULT_DIR KV_OFFLOADING TOTAL_CPU_DRAM_GB | ||
| if [[ "$KV_OFFLOADING" != "none" ]]; then | ||
| echo "This low-concurrency EAGLE3 experiment requires GPU-resident KV cache" >&2 | ||
| exit 1 | ||
| fi | ||
| export INFMAX_CONTAINER_WORKSPACE="${INFMAX_CONTAINER_WORKSPACE:-/workspace}" | ||
| resolve_trace_source | ||
| install_agentic_deps | ||
| OUTPUT_DIR="$RESULT_DIR" | ||
| CONTEXT_LENGTH=196608 | ||
| export MAX_MODEL_LEN="$CONTEXT_LENGTH" | ||
| MAX_RUNNING_REQUESTS=$((CONC * 2)) | ||
| (( MAX_RUNNING_REQUESTS < 8 )) && MAX_RUNNING_REQUESTS=8 | ||
| else | ||
| check_env_vars ISL OSL MAX_MODEL_LEN | ||
| OUTPUT_DIR="$PWD" | ||
| CONTEXT_LENGTH="$MAX_MODEL_LEN" | ||
| MAX_RUNNING_REQUESTS=$((CONC * 2)) | ||
| fi | ||
| mkdir -p "$OUTPUT_DIR" | ||
|
|
||
| SERVER_LOG="$OUTPUT_DIR/server.log" | ||
| SERVER_PID="" | ||
| GPU_MONITOR_STARTED=false | ||
|
|
||
| cleanup() { | ||
| local exit_code=$? | ||
| trap - EXIT INT TERM | ||
| set +e | ||
| if [[ "$GPU_MONITOR_STARTED" == "true" ]]; then | ||
| stop_gpu_monitor | ||
| fi | ||
| stop_background_process_tree "$SERVER_PID" "MiniMax-M2.7 EAGLE3 server" 60 | ||
| exit "$exit_code" | ||
| } | ||
| trap cleanup EXIT INT TERM | ||
|
|
||
| start_gpu_monitor --output "$OUTPUT_DIR/gpu_metrics.csv" | ||
| GPU_MONITOR_STARTED=true | ||
|
|
||
| SGLANG_CMD=( | ||
| python3 -m sglang.launch_server | ||
| --model-path "$TARGET_MODEL_PATH" | ||
| --served-model-name "$TARGET_MODEL" | ||
| --host 0.0.0.0 | ||
| --port "$PORT" | ||
| --trust-remote-code | ||
| --tp "$TP" | ||
| --ep-size "$EP_SIZE" | ||
| --quantization modelopt_fp4 | ||
| --dtype bfloat16 | ||
| --attention-backend triton | ||
| --moe-runner-backend flashinfer_cutlass | ||
| --speculative-algorithm EAGLE3 | ||
| --speculative-draft-model-path "$DRAFT_MODEL_PATH" | ||
| --speculative-num-steps 3 | ||
| --speculative-eagle-topk 1 | ||
| --speculative-num-draft-tokens 4 | ||
| --speculative-draft-model-quantization unquant | ||
| --speculative-draft-attention-backend triton | ||
| --reasoning-parser minimax | ||
| --tool-call-parser minimax-m2 | ||
| --context-length "$CONTEXT_LENGTH" | ||
| --chunked-prefill-size 16384 | ||
| --weight-loader-prefetch-checkpoints | ||
| --max-running-requests "$MAX_RUNNING_REQUESTS" | ||
| --mem-fraction-static 0.85 | ||
| --enable-metrics | ||
| --watchdog-timeout 1800 | ||
| ) | ||
|
|
||
| printf '%q ' "${SGLANG_CMD[@]}" | tee "$OUTPUT_DIR/sglang_command.txt" | ||
| printf '\n' | tee -a "$OUTPUT_DIR/sglang_command.txt" | ||
| "${SGLANG_CMD[@]}" > "$SERVER_LOG" 2>&1 & | ||
| SERVER_PID=$! | ||
|
|
||
| wait_for_server_ready --port "$PORT" --server-log "$SERVER_LOG" --server-pid "$SERVER_PID" | ||
|
|
||
| mtp_metrics=$(curl -fsS "http://127.0.0.1:$PORT/metrics") | ||
| if ! grep -m1 '^sglang:spec_accept_length' <<< "$mtp_metrics"; then | ||
| echo "EAGLE3 acceptance metrics are absent; refusing to publish a non-speculative result" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| if [[ "$IS_AGENTIC" == "true" ]]; then | ||
| # AgentX sends structured chat messages to /v1/chat/completions, so the | ||
| # target checkpoint's native chat template is applied by the server. | ||
| build_replay_cmd "$RESULT_DIR" | ||
| # TP4 can take several minutes to finish a long response that was already | ||
| # admitted near the end of the measurement window. The AIPerf default is | ||
| # only 30 seconds, which cancelled the sole profiling request in the TP4 | ||
| # smoke while the healthy server was still decoding it. Bound the drain at | ||
| # 30 minutes without extending the measured request-admission window. | ||
| REPLAY_CMD+=" --benchmark-grace-period 1800" | ||
| REPLAY_CMD+=" --server-metrics http://localhost:$PORT/metrics" | ||
| run_agentic_replay_and_write_outputs "$RESULT_DIR" | ||
| else | ||
| # EAGLE3 was trained on chat-formatted prompts; keep the benchmark client | ||
| # on the target checkpoint's native chat template. | ||
| run_benchmark_serving \ | ||
| --model "$TARGET_MODEL_PATH" \ | ||
| --port "$PORT" \ | ||
| --backend vllm \ | ||
| --input-len "$ISL" \ | ||
| --output-len "$OSL" \ | ||
| --random-range-ratio "$RANDOM_RANGE_RATIO" \ | ||
| --num-prompts "$((CONC * 10))" \ | ||
| --max-concurrency "$CONC" \ | ||
| --result-filename "$RESULT_FILENAME" \ | ||
| --result-dir "$PWD" \ | ||
| --trust-remote-code \ | ||
| --use-chat-template | ||
|
|
||
| if [[ "${RUN_EVAL:-false}" == "true" ]]; then | ||
| run_eval --framework lm-eval --port "$PORT" | ||
| append_lm_eval_summary | ||
| fi | ||
| fi | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 In the hub-cache fallback branch (no
--local-dir),TARGET_MODEL_PATH/DRAFT_MODEL_PATHare left as bare HF repo IDs (e.g.nvidia/MiniMax-M2.7-NVFP4) instead of resolved filesystem paths, so the Python preflight's(target_path / "config.json").read_text()raisesFileNotFoundErrorbefore the server starts. This is a real defect in the documented hub-cache/local-invocation mode, but it's not exercised in CI sincelaunch_b200-dgxc.shalways exports an absoluteMODEL_PATHfor this recipe (routing through the--local-dirbranch instead); fix by capturing the resolved snapshot dir fromhf download(or passing--local-dir) in that branch too.Extended reasoning...
The new benchmark entrypoint
minimaxm2.7_fp4_b200_sglang_mtp.shsupports two model-resolution modes: an absolute-path branch (used whenMODEL_PATHis set orMODELis already an absolute path) that passes--local-dirtohf downloadand correctly setsTARGET_MODEL_PATH/DRAFT_MODEL_PATHto real directories, and anelsebranch (lines 63-67) intended for hub-cache/local invocation whereMODELis the bare HF id andMODEL_PATHis unset.In that
elsebranch,hf download "$TARGET_MODEL" --revision "$TARGET_REVISION"is called without--local-dir, which places the downloaded files under the HF hub cache (e.g.~/.cache/huggingface/hub/models--nvidia--MiniMax-M2.7-NVFP4/snapshots/<rev>/...), not in a directory literally namednvidia/MiniMax-M2.7-NVFP4relative to$PWD. Despite this, the script setsTARGET_MODEL_PATH="$TARGET_MODEL"(i.e. the bare repo id string) andDRAFT_MODEL_PATH="$DRAFT_MODEL"(the bare draft repo id) rather than capturing the resolved snapshot path.Those two variables are then fed straight into the inline Python preflight:
With
target_path = Path("nvidia/MiniMax-M2.7-NVFP4"), this resolves to the nonexistent relative path./nvidia/MiniMax-M2.7-NVFP4/config.json, andread_text()raisesFileNotFoundError. Because the script runs underset -Eeuo pipefail, this immediately aborts the run —sglang.launch_serveris never even invoked, even thoughsglangitself would have tolerated the bare repo id and resolved it via the HF cache fine. So the preflight validation step, not the actual model loading, is what breaks this mode.Step-by-step proof:
MODEL=nvidia/MiniMax-M2.7-NVFP4andMODEL_PATHunset (exactly the "local validation" scenario the PR description references).TARGET_MODEL_PATHis set to the string"nvidia/MiniMax-M2.7-NVFP4"(line ~65-67), not a filesystem path.hf download nvidia/MiniMax-M2.7-NVFP4 --revision ...(no--local-dir) populates~/.cache/huggingface/hub/...; no./nvidia/MiniMax-M2.7-NVFP4directory is created in$PWD.Path("nvidia/MiniMax-M2.7-NVFP4") / "config.json", which does not exist relative to$PWD.read_text()raisesFileNotFoundError, the script exits nonzero underset -e, and the server never starts.Why this doesn't block the current CI sweep:
runners/launch_b200-dgxc.shhardcodesMODEL_PATH="/lustre/fsw/gharunners/models/MiniMax-M2.7-NVFP4"forMODEL_PREFIX=minimaxm2.7/fp4and then doesexport MODEL="$MODEL_PATH", so in the actual GitHub Actions sweepTARGET_MODEL_PATHis always an absolute path and the--local-dirbranch is taken instead — thiselsebranch is dead code from CI's perspective. The PR's own validation (bash -nplus matrix generation) doesn't execute the script, so it wouldn't have caught this either. All three independent verifiers reached the same conclusion and rated it nit severity for this reason.Suggested fix: capture the resolved snapshot directory that
hf downloadprints/returns (or, simpler, always pass--local-dir "$TARGET_MODEL_PATH"/"$DRAFT_MODEL_PATH"pointing at a real staging directory even in the hub-cache branch, mirroring the absolute-path branch's behavior) so the preflight always receives real filesystem paths.