File tree Expand file tree Collapse file tree
Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -52,6 +52,32 @@ Supported EAGLE-3 draft models include:
5252
5353For the full and up-to-date list of supported models, see #18039 .
5454
55+ ### DFlash (` draft-dflash ` )
56+
57+ DFlash produces an entire block of draft tokens in a single forward pass (block diffusion) and
58+ injects the target model's hidden states into the draft model's attention, instead of drafting one
59+ token at a time. This keeps the draft model small while making drafting GPU-friendly. Unlike EAGLE-3
60+ (a single-layer autoregressive draft), the DFlash draft uses several transformer layers but emits a
61+ whole block per draft step.
62+
63+ The draft is a small block-diffusion model trained for a specific target (for example
64+ ` z-lab/Qwen3-4B-DFlash ` for ` Qwen/Qwen3-4B ` ). Convert it with ` --target-model-dir ` so it inherits the
65+ target's tokenizer and token embeddings:
66+
67+ ``` bash
68+ python convert_hf_to_gguf.py z-lab/Qwen3-4B-DFlash \
69+ --target-model-dir Qwen/Qwen3-4B --outtype bf16 --outfile Qwen3-4B-DFlash.gguf
70+
71+ llama-server -m Qwen3-4B.gguf -md Qwen3-4B-DFlash.gguf \
72+ --spec-type draft-dflash --spec-draft-n-max 15 -fa on --jinja
73+ ```
74+
75+ ` --spec-draft-n-max ` is clamped to the draft model's trained block size.
76+
77+ See:
78+
79+ - #22105
80+
5581### n-gram Cache (` ngram-cache ` )
5682
5783An n-gram is a sequence of n tokens. The n-gram cache implementation maintains statistics about short n-gram sequences.
You can’t perform that action at this time.
0 commit comments