Skip to content

Commit 8542737

Browse files
committed
docs: add dflash section
1 parent 64a4744 commit 8542737

1 file changed

Lines changed: 26 additions & 0 deletions

File tree

docs/speculative.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,32 @@ Supported EAGLE-3 draft models include:
5252

5353
For the full and up-to-date list of supported models, see #18039.
5454

55+
### DFlash (`draft-dflash`)
56+
57+
DFlash produces an entire block of draft tokens in a single forward pass (block diffusion) and
58+
injects the target model's hidden states into the draft model's attention, instead of drafting one
59+
token at a time. This keeps the draft model small while making drafting GPU-friendly. Unlike EAGLE-3
60+
(a single-layer autoregressive draft), the DFlash draft uses several transformer layers but emits a
61+
whole block per draft step.
62+
63+
The draft is a small block-diffusion model trained for a specific target (for example
64+
`z-lab/Qwen3-4B-DFlash` for `Qwen/Qwen3-4B`). Convert it with `--target-model-dir` so it inherits the
65+
target's tokenizer and token embeddings:
66+
67+
```bash
68+
python convert_hf_to_gguf.py z-lab/Qwen3-4B-DFlash \
69+
--target-model-dir Qwen/Qwen3-4B --outtype bf16 --outfile Qwen3-4B-DFlash.gguf
70+
71+
llama-server -m Qwen3-4B.gguf -md Qwen3-4B-DFlash.gguf \
72+
--spec-type draft-dflash --spec-draft-n-max 15 -fa on --jinja
73+
```
74+
75+
`--spec-draft-n-max` is clamped to the draft model's trained block size.
76+
77+
See:
78+
79+
- #22105
80+
5581
### n-gram Cache (`ngram-cache`)
5682

5783
An n-gram is a sequence of n tokens. The n-gram cache implementation maintains statistics about short n-gram sequences.

0 commit comments

Comments
 (0)