Motivation
On Apple Silicon (M2 Ultra 64GB), speculative decoding performance varies dramatically by content type. On high-entropy prompts, both model-draft and MTP specdec actively hurt throughput — but spec type is a server startup flag with no way to disable it per-request.
Benchmarks (Qwen3.6-27B verifier, Metal backend, ctx=4096)
| Config |
BST code (temp=0) |
Technical prose (temp=1) |
Creative writing (temp=1) |
| Baseline (no spec) |
22.5 tok/s |
22.3 tok/s |
22.3 tok/s |
| MTP head (n-max=5) |
30.3 (1.35x, 96% accept, 0.89 dpt) |
24.1 (1.08x) |
20.4 (0.91x) |
| 0.8B drafter (n-max=20) |
32.1 (1.43x, 81% accept, 0.98 dpt) |
22.9 (1.03x) |
19.6 (0.88x) |
MTP at n-max=20 drops to 15.2 tok/s (0.68x) — the overhead of generating drafts that get rejected dominates. draft-p-min at 0.999 gets code back to baseline (22.4 tok/s) but creative still suffers at 19.5 tok/s because the draft model sits in VRAM competing for unified memory bandwidth even when idle.
Desired behavior
A per-request flag to disable speculative decoding:
# Option A: extra_body field (like enable_thinking)
client.chat.completions.create(
model="qwopus",
messages=[...],
extra_body={"speculative": False},
)Or a runtime admin endpoint: POST /admin/spec {"enabled": false}
Precedent
enable_thinking is already toggleable per-request via the same extra_body mechanism (discussion #21929). Speculative decoding fits the same pattern — a backend feature that helps on some prompts and hurts on others, where the client (or a downstream proxy/router) is best positioned to decide.
Why this should be easy
The MTP/draft contexts are loaded at startup and stay resident. Toggling spec on/off is a boolean on the slot — skip the draft forward pass in the decode loop. No model reloading, no reallocation. This is the same complexity class as enable_thinking.
Related
Motivation
On Apple Silicon (M2 Ultra 64GB), speculative decoding performance varies dramatically by content type. On high-entropy prompts, both model-draft and MTP specdec actively hurt throughput — but spec type is a server startup flag with no way to disable it per-request.
Benchmarks (Qwen3.6-27B verifier, Metal backend, ctx=4096)
MTP at n-max=20 drops to 15.2 tok/s (0.68x) — the overhead of generating drafts that get rejected dominates.
draft-p-minat 0.999 gets code back to baseline (22.4 tok/s) but creative still suffers at 19.5 tok/s because the draft model sits in VRAM competing for unified memory bandwidth even when idle.Desired behavior
A per-request flag to disable speculative decoding:
Or a runtime admin endpoint:
POST /admin/spec {"enabled": false}Precedent
enable_thinkingis already toggleable per-request via the sameextra_bodymechanism (discussion #21929). Speculative decoding fits the same pattern — a backend feature that helps on some prompts and hurts on others, where the client (or a downstream proxy/router) is best positioned to decide.Why this should be easy
The MTP/draft contexts are loaded at startup and stay resident. Toggling spec on/off is a boolean on the slot — skip the draft forward pass in the decode loop. No model reloading, no reallocation. This is the same complexity class as
enable_thinking.Related