Skip to content

Commit da04f6f

Browse files
committed
Merge remote-tracking branch 'origin/main' into xabicasa/dflash-gguf-draft-loader-target-layers
2 parents ce5e8dc + 332f7dc commit da04f6f

53 files changed

Lines changed: 6275 additions & 910 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.gitattributes

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
*.gif filter=lfs diff=lfs merge=lfs -text
22
*.png filter=lfs diff=lfs merge=lfs -text
3+
assets/banner.png -filter -diff -merge -text
34
*.jpg filter=lfs diff=lfs merge=lfs -text
45
*.jpeg filter=lfs diff=lfs merge=lfs -text
56
*.mp4 filter=lfs diff=lfs merge=lfs -text

.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@ MANIFEST
3737
# virtualenvs
3838
venv/
3939
.venv/
40+
.conda/
4041
env/
4142
.env
4243

@@ -64,6 +65,7 @@ checkpoints/
6465
*.safetensors
6566
*.bin
6667
.cache/
68+
fix-plan.md
6769

6870
# Secrets
6971
.env.local

README.md

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -40,14 +40,15 @@ Three projects today, more coming. Each one is a self-contained release with its
4040

4141
## Supported models
4242

43-
All speedups measured on **RTX 3090 24 GB** vs vendored llama.cpp (`-fa 1`, matching KV quant).
43+
All speedups measured vs vendored llama.cpp (`-fa 1`, matching KV quant).
4444

45-
| Model | TTFT speedup | Decode speedup |
46-
|-------|:------------:|:--------------:|
47-
| Qwen 3.5-0.8B (Megakernel) || **~** vs F16 |
48-
| Qwen 3.5-27B Q4_K_M (DFlash + DDTree) || **3.43×** vs AR |
49-
| Qwen 3.6-27B Q4_K_M (DFlash + PFlash) | **10.4×** @ 128K | **~** vs AR |
50-
| Laguna-XS.2 33B-A3B Q4_K_M (DFlash + PFlash) | **5.4×** @ 128K | AR (draft pending) |
45+
| GPU | Model | TTFT speedup | Decode speedup |
46+
|-----|-------|:------------:|:--------------:|
47+
| RTX 3090 | Qwen 3.5-0.8B (Megakernel) || **~** vs F16 |
48+
| RTX 3090 | Qwen 3.5-27B Q4_K_M (DFlash + DDTree) || **3.43×** vs AR |
49+
| RTX 3090 | Qwen 3.6-27B Q4_K_M (DFlash + PFlash) | **10.4×** @ 128K | **~** vs AR |
50+
| RTX 3090 | Laguna-XS.2 33B-A3B Q4_K_M (DFlash + PFlash) | **5.4×** @ 128K | AR (draft pending) |
51+
| RTX 5090 | Qwen 3.6-27B Q4_K_M (DFlash + DDTree) || **4.84×** vs AR (205 tok/s) |
5152

5253
## 01 · Megakernel Qwen3.5 0.8B on RTX 3090
5354

@@ -139,7 +140,7 @@ Supported out of the box; the build just needs the right CUDA toolkit. `dflash/C
139140
| RTX 3090 Ampere | `sm_86` | 12.0 | **reference, all numbers above** |
140141
| RTX 2080 Ti Turing | `sm_75` | 12.0 | supported, 53 tok/s DFlash verified (FP16 draft) |
141142
| RTX 4090 Ada | `sm_89` | 12.0 | should work, unverified, pass `-DCMAKE_CUDA_ARCHITECTURES=89` |
142-
| RTX 5090 Blackwell consumer | `sm_120` | 12.8 | supported, auto-added by CMake |
143+
| RTX 5090 Blackwell consumer | `sm_120` | 12.8 | **205 tok/s DFlash, 4.84× vs AR** (Q4_K_M, budget=40) |
143144
| DGX Spark / GB10 | `sm_121` (compute capability 12.1) | 12.9 | supported, auto-added by CMake |
144145
| Jetson AGX Thor | `sm_110` | 13.0 | supported, auto-added by CMake |
145146

@@ -168,9 +169,9 @@ cmake --build build --target test_dflash -j
168169
```
169170

170171
**What will NOT auto-port:**
171-
- **DDTree `budget=22`** tuned for 3090 + Q4_K_M + 24 GB. On cards with more VRAM (5090 32 GB, GB10 128 GB unified), re-sweep, larger tree = more verify throughput until memory bandwidth saturates. `scripts/bench_llm.py` has the sweep hooks.
172+
- **DDTree `budget=22`** tuned for 3090 + Q4_K_M + 24 GB. On the RTX 5090, budget=40 is optimal (swept). On GB10 (128 GB unified), re-sweeplarger tree = more verify throughput until memory bandwidth saturates. `scripts/bench_llm.py --budget N` has the sweep hooks.
172173
- **TQ3_0 KV cache + sliding `target_feat` ring** was shaped by 24 GB (fits up to 256K context on a 3090). On GB10 (128 GB unified) / 5090 (32 GB) you can push context further or skip quantization entirely and keep F16 KV.
173-
- **Perf numbers** (207 tok/s demo, 129.5 HumanEval, 2.8× vs SGLang AWQ) are RTX 3090 @ stock. Blackwell/Ada not yet swept, PRs with `RESULTS.md` entries welcome.
174+
- **Perf numbers** (207 tok/s demo, 129.5 HumanEval, 2.8× vs SGLang AWQ) are RTX 3090 @ stock. RTX 5090 numbers (205 tok/s HumanEval, 4.84×) are in [RESULTS.md](dflash/RESULTS.md). Ada/GB10/Thor not yet swept, PRs with `RESULTS.md` entries welcome.
174175

175176
[Full writeup →](dflash/README.md) · [Benchmarks →](dflash/RESULTS.md) · [Blog post →](https://lucebox.com/blog/dflash27b)
176177

dflash/.gitignore

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,6 @@
11
build/
2+
build-*/
3+
reports/
24
*.o
35
*.a
46
*.so

dflash/CMakeLists.txt

Lines changed: 206 additions & 102 deletions
Large diffs are not rendered by default.

dflash/CODEX.md

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
# Using DFlash with OpenAI Codex CLI
2+
3+
This guide covers installing Codex CLI, configuring it to use the local
4+
DFlash server, and common usage patterns.
5+
6+
## Prerequisites
7+
8+
- **DFlash server** built and model files downloaded (see [DEVELOPER.md](DEVELOPER.md))
9+
- **Node.js ≥ 18**
10+
11+
## 1. Install Codex CLI
12+
13+
```bash
14+
npm install -g @openai/codex
15+
codex --version # should print 0.1xx.x
16+
```
17+
18+
> If you get a permission error, use a user-level prefix:
19+
> ```bash
20+
> mkdir -p ~/.npm-global && npm config set prefix ~/.npm-global
21+
> export PATH=~/.npm-global/bin:$PATH # add to ~/.bashrc
22+
> npm install -g @openai/codex
23+
> ```
24+
25+
## 2. Start the DFlash Server
26+
27+
```bash
28+
cd dflash
29+
python scripts/server.py \
30+
--draft models/draft/draft-Qwen3.6-27B.gguf \
31+
--port 8080
32+
```
33+
34+
Wait until you see:
35+
36+
```
37+
INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
38+
```
39+
40+
### Quick smoke test
41+
42+
```bash
43+
curl -s http://localhost:8080/v1/responses \
44+
-H "Content-Type: application/json" \
45+
-d '{"model":"luce-dflash","input":"Say hi","max_output_tokens":32}' \
46+
| python3 -m json.tool
47+
```
48+
49+
## 3. Configure Codex
50+
51+
Create or edit `~/.codex/config.toml`:
52+
53+
```toml
54+
model = "luce-dflash"
55+
model_provider = "dflash"
56+
57+
[model_providers.dflash]
58+
name = "DFlash"
59+
base_url = "http://localhost:8080/v1"
60+
wire_api = "responses"
61+
supports_websockets = false
62+
```
63+
64+
Key points:
65+
66+
| Field | Why |
67+
|---|---|
68+
| `name` | **Required** — Codex rejects providers with an empty name |
69+
| `wire_api = "responses"` | Codex only supports the Responses API |
70+
| `supports_websockets = false` | DFlash serves HTTP only |
71+
| No `env_key` | Local server needs no auth token |
72+
73+
## 4. Using Codex
74+
75+
### Interactive mode
76+
77+
```bash
78+
codex # opens the TUI
79+
```
80+
81+
### Non-interactive (exec)
82+
83+
```bash
84+
# Simple question
85+
codex exec "What is 2+2?"
86+
87+
# Coding task in a repo
88+
cd /path/to/your/project
89+
codex exec "Add input validation to the login handler"
90+
91+
# Code review
92+
codex exec review
93+
```
94+
95+
### Approval policies
96+
97+
```bash
98+
codex --approval-policy suggest ... # suggest commands, ask before running
99+
codex --approval-policy auto-edit ... # auto-approve file edits, ask for shell
100+
codex --approval-policy full-auto ... # auto-approve everything
101+
```
102+
103+
### Reasoning effort
104+
105+
The server maps reasoning effort to Qwen's thinking mode:
106+
107+
- `low` (default) — thinking disabled, fastest responses
108+
- `medium` / `high` — thinking enabled (`<think>` tags)
109+
110+
```bash
111+
codex -c 'model_reasoning_effort="medium"' exec "Refactor this module"
112+
```
113+
114+
## 5. Troubleshooting
115+
116+
### "provider name must not be empty"
117+
118+
Add `name = "DFlash"` (or any non-empty string) to `[model_providers.dflash]`.
119+
120+
### "Reconnecting… 1/5" loop
121+
122+
The server is returning an error. Check:
123+
124+
1. **Server is running**`curl http://localhost:8080/health`
125+
2. **Port matches config**`base_url` must match `--port`
126+
3. **Server logs** — look at the terminal where `server.py` is running
127+
128+
Common causes:
129+
130+
- Server not started yet (daemon still loading model)
131+
- Port mismatch between config and server
132+
- Proxy intercepting localhost (set `no_proxy=localhost`)
133+
134+
### "Unknown model luce-dflash"
135+
136+
This warning is normal — Codex doesn't recognize custom model names but
137+
uses fallback metadata. It does not affect functionality.
138+
139+
### Slow first response
140+
141+
The first request triggers model loading into GPU memory. Subsequent
142+
requests reuse the loaded model and are much faster.
143+
144+
## 6. Server CLI Reference
145+
146+
```
147+
python scripts/server.py [OPTIONS]
148+
149+
Options:
150+
--host HOST Bind address (default: 0.0.0.0)
151+
--port PORT Bind port (default: 8080)
152+
--target PATH Target model GGUF (default: models/Qwen3.6-27B-Q4_K_M.gguf)
153+
--draft PATH Draft model for speculative decoding
154+
--budget N Speculation budget (default: 22)
155+
--max-ctx N Max context length (default: model-dependent)
156+
--tokenizer NAME HuggingFace tokenizer (auto-detected from model)
157+
```

0 commit comments

Comments
 (0)