Add SPEED-Bench reference synthetic AL values for DeepSeek-V4-Pro MTP 1-8

[qiching]

Measured with SPEED-Bench coding dataset, temperature=1.0, thinking=true. Values used for synthetic acceptance rate configuration in MTP benchmarks.

---

Note

Low Risk
Documentation-only benchmark reference data with no application or infrastructure code changes.

Overview
Adds benchmarks/speedbench-reference-al.yaml, a new SPEED-Bench reference for acceptance length (AL) on DeepSeek-V4-Pro with vLLM MTP for num_speculative_tokens 1–8 (coding dataset, temperature 1.0, output_len 4096).

The file documents two AL curves: thinking_on (marked as the production golden reference for synthetic-acceptance modeling) and thinking_off (comparison only). The numeric tables differ from the PR description’s single table, which aligns with the thinking_off values rather than production thinking_on.

^{Reviewed by Cursor Bugbot for commit 1f0d9dd. Bugbot is set up for automated code reviews on this repo. Configure here.}

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[xinli-sw]

@qiching , a few recommendations

1. title [1/N] Synthetic MTP - Add SPEED-Bench reference synthetic AL values for DeepSeek-V4-Pro MTP 1-8

2. In the PR description, mention that we will also have speedbench as part of github workflows, however, as the first iteration, we'd like to get alignment to make sure partners all feel confident and equally about these AL values

3. attach full repro for the numbers (serve command, installation, speedbench command, etc)

4. attach full results of the runs you had (the jsonl file) for audibility

Great work so far, cc @benchislett @functionstackx

[benchislett]

These numbers seem a bit high to me. Let's double check that thinking is on and temperature is being set properly.

[functionstackx]

thanks for this PR, what is the scripts that u used to generate this? can u create an PR that has the github actions for running this AL distribtuion collection script?

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1f0d9dd. Configure here.}

[cursor]

Golden reference wrong thinking block

Medium Severity

Comments designate thinking_on as the production golden reference for synthetic-acceptance modeling, but the PR’s thinking=true AL table (MTP 1–8) aligns with the thinking_off entries, not thinking_on. Anything reading the golden block gets different AL than the values partners are asked to align on.

 

^{Reviewed by Cursor Bugbot for commit 1f0d9dd. Configure here.}

[qiching]

These numbers seem a bit high to me. Let's double check that thinking is on and temperature is being set properly.

updated. Please check the latest number of AL when thinking mode on/off.

[functionstackx]

thanks for the contribution! shouldnt the github action generating the yaml to check into the codebase instead of having it human generated?

i.e. close this PR and then have the other PR generate this yaml? #1650