Skip to content

Commit 3849949

Browse files
FKKimuraFKKimura
committed
Expand model_validation suite with gptq load/vllm phases, qep_gptq, autobit, jointq
- Add `model_validation/README.md`: cross-recipe overview with At-a-Glance status tables and per-recipe results - Extend `model_validation/gptq/` with Phase 2 (`validate_load.py`) and Phase 3 (`validate_vllm.py`) for load + transformers / vLLM inference - Add `model_validation/qep_gptq/`: GPTQ + QEP variant - Add `model_validation/autobit/`: AutoBit (target_bit=4) without QEP - Add `model_validation/jointq/`: JointQ (4-bit, gs=128, symmetric) sanity-checked on dequantized model - Update `model_validation/autobit_qep/` with full 5-model results and gemma-4-E2B bit-assignment analysis - Move `model_validation/<dir>/{submit*.sh,logs/,outputs/}` patterns from root `.gitignore` to `model_validation/.gitignore` using wildcards (`*/submit*.sh`, `*/logs/`, `*/outputs/`) so new subdirectories are covered automatically - Update CHANGELOG.md Model Validation / Packaging sections accordingly
1 parent 1f0de8d commit 3849949

20 files changed

Lines changed: 1200 additions & 41 deletions

‎.gitignore‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -16,12 +16,6 @@ models
1616
*.err
1717
md/*.md
1818
example/run_example.sh
19-
model_validation/autobit_qep/submit.sh
20-
model_validation/autobit_qep/logs/
21-
model_validation/autobit_qep/outputs/
22-
model_validation/gptq/submit.sh
23-
model_validation/gptq/logs/
24-
model_validation/gptq/outputs/
2519
work/
2620
site/
2721
debug_code/
@@ -31,3 +25,6 @@ debug_code/
3125
tests/onecomp/quantizer/jointq/data/model_layers_0_self_attn_k_proj.pth
3226
.uv-sync.lock
3327
.uv-sync.done
28+
.uv-sync-vllm.lock
29+
.uv-sync-vllm.done
30+
.venv-vllm/

‎CHANGELOG.md‎

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -181,11 +181,20 @@
181181

182182
### Model Validation
183183

184-
- Added `model_validation/autobit_qep/`: Hydra-driven AutoBit (`target_bit=4`, `qep=True`) end-to-end validation script (`validate_autobit.py`, `conf/validate.yaml`, `README.md`) that quantizes a single model selected via either `model_id` (Hugging Face Hub) or `model_path` (local directory), saves the quantized model via `runner.save_quantized_model("./quantized")`, and reports original / quantized perplexity. Default validation set covers TinyLlama-1.1B, Llama-2-7B, Llama-3-8B, Qwen3-8B, and gemma-4-E2B.
184+
- Added `model_validation/README.md`: parent overview of the operational validation suite that exercises OneComp's end-to-end quantize → save → load → inference workflow across multiple architectures and sizes. Provides cross-recipe At-a-Glance status tables (per-model × per-recipe quantization status, and per-model × per-recipe `(save, transformers inference, vllm inference)` status), per-recipe result tables, and a Summary section that explicitly cautions against cross-recipe PPL comparison (PPL is reported only as a per-recipe sanity check; the compact calibration in use sits below typical research settings, partly because the calibration size has to fit the DGX Spark 128 GB UMA budget for 7–8B models with QEP on).
185+
- Added `model_validation/gptq/`: Hydra-driven GPTQ (`wbits=4`, `groupsize=128`, `qep=False`) end-to-end validation across three phases:
186+
- Phase 1 — quantize + save (`validate_gptq.py`, `conf/validate.yaml`): `CalibrationConfig(max_length=512, num_calibration_samples=128)`, saved via `runner.save_quantized_model(...)`, reports original / quantized PPL on `wikitext-2-raw-v1`.
187+
- Phase 2 — load + greedy generation (`validate_load.py`, `conf/validate_load.yaml`): reloads each saved directory via `load_quantized_model` and runs `"Fujitsu is"` with `max_new_tokens=32`. `torch_dtype` is overridable (`float16` / `bfloat16` / `float32` / `null`); gemma-4-E2B requires `bfloat16` because the loader's default `float16` triggers a `Half` / `BFloat16` mismatch at `lm_head`.
188+
- Phase 3 — vLLM offline inference (`validate_vllm.py`, `conf/validate_vllm.yaml`): reloads each saved directory via vLLM's offline `LLM` interface (OneComp's vLLM plugin is auto-registered, so no explicit `quantization=` argument is required) and runs `"Fujitsu is"` with `temperature=0.0`, `max_tokens=32`, `enforce_eager=True`, `max_model_len=512`. `LLM(...)` and `llm.generate(...)` are kept inside `main()` behind `if __name__ == "__main__":` so vLLM worker subprocess re-imports do not recursively spawn new engines. Currently `pending` for all five models.
189+
- Added `model_validation/qep_gptq/`: Hydra-driven GPTQ (`wbits=4`, `groupsize=128`, `qep=True`) end-to-end validation script (`validate_gptq.py`, `conf/validate.yaml`, `README.md`). Calibration: `max_length=1024`, `num_calibration_samples=128` (reduced from defaults to keep 7–8B models within the DGX Spark 128 GB UMA budget with QEP on). Quantize + save only; load / inference is not exercised in this subdirectory yet.
190+
- Added `model_validation/autobit/`: Hydra-driven AutoBit (`target_bit=4`, `qep=False`) end-to-end validation script (`validate_autobit.py`, `conf/validate.yaml`, `README.md`). Candidates `GPTQ(wbits=b, groupsize=128) for b in (2, 3, 4, 8)`, `assignment_strategy="activation_aware"`, `CalibrationConfig(max_length=512, num_calibration_samples=128)`. Quantize + save only.
191+
- Updated `model_validation/autobit_qep/`: AutoBit (`target_bit=4`, `qep=True`) validation. Reduced calibration to `max_length=1024`, `num_calibration_samples=128` to keep 7–8B models within the DGX Spark 128 GB UMA budget. README expanded with per-model bit-assignment counts (`GPTQ_<b>_gs128: <count> layers`) for TinyLlama-1.1B, Llama-2-7B, Llama-3-8B, Qwen3-8B, and gemma-4-E2B; documents the bimodal 8-bit / 2-bit ILP collapse on gemma-4-E2B (every module in the first 15 transformer blocks → 8-bit, remaining 20 blocks → 2-bit; quantized PPL diverges to ~10^14, reproduced after reducing calibration from `max_length=2048, num_calibration_samples=512`) and lists candidate follow-ups (restrict candidate set, disable QEP, switch `assignment_strategy`).
192+
- Added `model_validation/jointq/`: Hydra-driven JointQ (`bits=4`, `group_size=128`, `symmetric=True`, `qep=False`) end-to-end validation script (`validate_jointq.py`, `conf/validate.yaml`, `README.md`). Calibration: `CalibrationConfig(max_length=512, num_calibration_samples=128)`. JointQ does not currently expose a quantized-inference layer (no `save_quantized_model` / `create_quantized_model` path), so quality is sanity-checked on the dequantized model (weights reconstructed from JointQ's quantization parameters); save / inference are reported as `n/a` in the parent At-a-Glance table.
193+
- All five recipes share the same model selection contract: a single model selected via either `model_id` (Hugging Face Hub) or `model_path` (local directory), with any field in `conf/validate*.yaml` overridable on the command line. Default validation set across all recipes is TinyLlama-1.1B, gemma-4-E2B (base), Llama-2-7B, Llama-3-8B, and Qwen3-8B.
185194

186195
### Packaging
187196

188-
- Added `hydra` extra to `pyproject.toml` so `hydra-core` (used by `example/example_autobit.py` and `model_validation/autobit_qep/validate_autobit.py`) installs in one step via `uv sync --extra <cuXXX> --extra hydra` or `pip install "onecomp[hydra]"`, instead of a separate `pip install hydra-core` after sync. Documented the new extra in `README.md` and `model_validation/autobit_qep/README.md`.
197+
- Added `hydra` extra to `pyproject.toml` so `hydra-core` (used by `example/example_autobit.py` and the `model_validation/{gptq,qep_gptq,autobit,autobit_qep,jointq}/validate_*.py` scripts) installs in one step via `uv sync --extra <cuXXX> --extra hydra` or `pip install "onecomp[hydra]"`, instead of a separate `pip install hydra-core` after sync. Documented the new extra in `README.md` and the `model_validation/*/README.md` files. The `model_validation/gptq/` Phase 3 (vLLM inference) additionally requires the `vllm` extra (`uv sync --extra <cuXXX> --extra hydra --extra vllm` or `pip install "onecomp[hydra]" vllm`).
189198

190199
## [v1.0.2] 2026-03-31
191200

‎model_validation/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
*/submit*.sh
2+
*/logs/
3+
*/outputs/

‎model_validation/README.md‎

Lines changed: 250 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,250 @@
1+
# Model Validation
2+
3+
An operational validation suite for OneComp's end-to-end workflow
4+
(quantize → save → load → inference) across a variety of model
5+
architectures and sizes. Each subdirectory exercises one workflow
6+
configuration; what is exercised varies per subdirectory and is listed
7+
below.
8+
9+
Sanity checks reported here:
10+
11+
- Perplexity on `wikitext-2-raw-v1`, only as a check that the quantized
12+
model is not broken — **not** an accuracy benchmark.
13+
- Greedy generation from a saved + reloaded model, only as a check
14+
that the load + inference path runs without errors.
15+
16+
## At a Glance
17+
18+
### Quantization
19+
20+
Status legend (quantization step only):
21+
22+
- **OK** — quantize runs end-to-end and the resulting (quantized, or
23+
dequantized for recipes that don't have a quantized-inference path
24+
yet) model passes the PPL sanity check.
25+
- **OK\*** — runs, but PPL is somewhat worse than expected for the
26+
recipe; worth a closer look.
27+
- **NG** — runs, but PPL is clearly broken.
28+
- **pending** — not yet executed.
29+
30+
| Model | GPTQ | QEP+GPTQ | AutoBit | AutoBit+QEP | JointQ |
31+
|---|:---:|:---:|:---:|:---:|:---:|
32+
| TinyLlama-1.1B | OK | OK | OK | OK | OK |
33+
| gemma-4-E2B (base) | OK\* | OK\* | NG | NG | OK |
34+
| Llama-2-7B | OK | OK | OK | OK | OK |
35+
| Llama-3-8B | OK\* | OK | OK | OK | OK |
36+
| Qwen3-8B | OK | OK | OK | OK | OK |
37+
38+
### Save / Inference
39+
40+
Each cell is `(save, transformers inference, vllm inference)`. Each
41+
entry uses OK / OK\* / NG / pending against that step's own success
42+
criterion:
43+
44+
- **save** — save step completes and produces a saved quantized model directory.
45+
- **transformers inference** — the saved model can be loaded and run inference via HF `transformers`.
46+
- **vllm inference** — the saved model can be loaded and run inference via vLLM.
47+
- **n/a** — the recipe has no implementation for that step yet.
48+
49+
| Model | GPTQ | QEP+GPTQ | AutoBit | AutoBit+QEP | JointQ |
50+
|---|:---:|:---:|:---:|:---:|:---:|
51+
| TinyLlama-1.1B | (OK, OK, pending) | (OK, pending, pending) | (OK, pending, pending) | (OK, pending, pending) | (n/a, n/a, n/a) |
52+
| gemma-4-E2B (base) | (OK, OK\*, pending) | (OK, pending, pending) | (OK, pending, pending) | (OK, pending, pending) | (n/a, n/a, n/a) |
53+
| Llama-2-7B | (OK, OK, pending) | (OK, pending, pending) | (OK, pending, pending) | (OK, pending, pending) | (n/a, n/a, n/a) |
54+
| Llama-3-8B | (OK, OK, pending) | (OK, pending, pending) | (OK, pending, pending) | (OK, pending, pending) | (n/a, n/a, n/a) |
55+
| Qwen3-8B | (OK, OK, pending) | (OK, pending, pending) | (OK, pending, pending) | (OK, pending, pending) | (n/a, n/a, n/a) |
56+
57+
JointQ does not yet provide a quantized-inference layer (no
58+
`save_quantized_model` / `create_quantized_model` path), so save and
59+
inference are listed as `n/a` until those are implemented; the
60+
quantization step for JointQ is sanity-checked on the dequantized
61+
model instead.
62+
63+
See per-recipe details in the [Results](#results) section below.
64+
65+
## Subdirectories
66+
67+
| Directory | Quantization recipe | Steps exercised |
68+
|---|---|---|
69+
| [`gptq/`](gptq/) | `GPTQ(wbits=4, groupsize=128)`, `qep=False` | quantize, save, load + greedy generation, PPL sanity check |
70+
| [`qep_gptq/`](qep_gptq/) | `GPTQ(wbits=4, groupsize=128)`, `qep=True` | quantize, save, PPL sanity check |
71+
| [`autobit/`](autobit/) | `AutoBitQuantizer(target_bit=4)`, candidates `GPTQ(wbits=b, groupsize=128) for b in (2,3,4,8)`, `assignment_strategy="activation_aware"`, `qep=False` | quantize, save, PPL sanity check |
72+
| [`autobit_qep/`](autobit_qep/) | `AutoBitQuantizer(target_bit=4)`, candidates `GPTQ(wbits=b, groupsize=128) for b in (2,3,4,8)`, `assignment_strategy="activation_aware"`, `qep=True` | quantize, save, PPL sanity check |
73+
| [`jointq/`](jointq/) | `JointQ(bits=4, group_size=128, symmetric=True)`, `qep=False` | quantize, PPL sanity check (on dequantized model). Save / inference: not yet implemented. |
74+
75+
## Results
76+
77+
### [`gptq/`](gptq/)
78+
79+
Two phases:
80+
81+
1. Quantize + save (`validate_gptq.py`). Calibration: `max_length=512`,
82+
`num_calibration_samples=128`.
83+
2. Load + greedy generation (`validate_load.py`). Prompt
84+
`"Fujitsu is"`, `max_new_tokens=32`.
85+
86+
Phase 1 — quantization PPL on `wikitext-2-raw-v1`:
87+
88+
| Model | Original PPL | Quantized PPL | Status |
89+
|---|---:|---:|---|
90+
| TinyLlama-1.1B | 7.77 | 8.69 | OK |
91+
| gemma-4-E2B (base) | 25.99 | 35.03 | OK\* |
92+
| Llama-2-7B | 5.47 | 6.59 | OK |
93+
| Llama-3-8B | 6.14 | 27.74 | OK\* |
94+
| Qwen3-8B | 9.72 | 10.72 | OK |
95+
96+
Phase 2 — load + greedy generation:
97+
98+
| Model | torch_dtype | Status |
99+
|---|---|---|
100+
| TinyLlama-1.1B | (default) | OK |
101+
| gemma-4-E2B | `bfloat16` | OK\* |
102+
| Llama-2-7B | (default) | OK |
103+
| Llama-3-8B | (default) | OK |
104+
| Qwen3-8B | (default) | OK |
105+
106+
Notes:
107+
108+
- gemma-4-E2B must be loaded as `bfloat16`; the loader's default
109+
`float16` triggers a `Half`/`BFloat16` mismatch at `lm_head`. With
110+
`bfloat16`, load + generation runs without errors but the output is
111+
degenerate (random tokens / non-Latin scripts), consistent with the
112+
warn-level PPL in phase 1.
113+
- All other models load and generate sensibly under the default dtype.
114+
115+
See [`gptq/README.md`](gptq/README.md) for full details, generated
116+
samples, and discussion.
117+
118+
### [`qep_gptq/`](qep_gptq/)
119+
120+
Quantize + save (`validate_gptq.py`) with QEP on. Calibration:
121+
`max_length=1024`, `num_calibration_samples=128`. Load + inference is
122+
not exercised in this subdirectory yet.
123+
124+
| Model | Original PPL | Quantized PPL | Status |
125+
|---|---:|---:|---|
126+
| TinyLlama-1.1B | 7.77 | 8.63 | OK |
127+
| gemma-4-E2B (base) | 25.99 | 37.82 | OK\* |
128+
| Llama-2-7B | 5.47 | 6.10 | OK |
129+
| Llama-3-8B | 6.14 | 7.14 | OK |
130+
| Qwen3-8B | 9.72 | 10.83 | OK |
131+
132+
See [`qep_gptq/README.md`](qep_gptq/README.md) for details.
133+
134+
### [`autobit/`](autobit/)
135+
136+
Quantize + save (`validate_autobit.py`) with QEP off. Calibration:
137+
`max_length=512`, `num_calibration_samples=128`. Load + inference is
138+
not exercised in this subdirectory yet.
139+
140+
| Model | Original PPL | Quantized PPL | Status |
141+
|---|---:|---:|---|
142+
| TinyLlama-1.1B | 7.77 | 8.75 | OK |
143+
| gemma-4-E2B (base) | 25.99 | 2.40e13 | NG |
144+
| Llama-2-7B | 5.47 | 5.94 | OK |
145+
| Llama-3-8B | 6.14 | 7.27 | OK |
146+
| Qwen3-8B | 9.72 | 10.74 | OK |
147+
148+
See [`autobit/README.md`](autobit/README.md) for details.
149+
150+
### [`autobit_qep/`](autobit_qep/)
151+
152+
Quantize + save (`validate_autobit.py`) with QEP on. Calibration:
153+
`max_length=1024`, `num_calibration_samples=128` (reduced from defaults
154+
to keep 7-8B models within the DGX Spark 128 GB UMA budget). Load +
155+
inference is not exercised in this subdirectory yet.
156+
157+
| Model | Original PPL | Quantized PPL | Status |
158+
|---|---:|---:|---|
159+
| TinyLlama-1.1B | 7.77 | 8.67 | OK |
160+
| gemma-4-E2B (base) | 25.99 | 1.64e14 | NG |
161+
| Llama-2-7B | 5.47 | 5.90 | OK |
162+
| Llama-3-8B | 6.14 | 7.24 | OK |
163+
| Qwen3-8B | 9.72 | 10.82 | OK |
164+
165+
See [`autobit_qep/README.md`](autobit_qep/README.md) for details and
166+
discussion.
167+
168+
### [`jointq/`](jointq/)
169+
170+
Quantize only (`validate_jointq.py`) with `qep=False`. Calibration:
171+
`max_length=512`, `num_calibration_samples=128`. Save and inference
172+
are **not yet implemented** for JointQ (no `save_quantized_model` /
173+
`create_quantized_model` path), so quality is sanity-checked on the
174+
dequantized model.
175+
176+
| Model | Original PPL | Dequantized PPL | Status |
177+
|---|---:|---:|---|
178+
| TinyLlama-1.1B | 7.77 | 8.25 | OK |
179+
| gemma-4-E2B (base) | 25.99 | 27.88 | OK |
180+
| Llama-2-7B | 5.47 | 5.64 | OK |
181+
| Llama-3-8B | 6.14 | 6.67 | OK |
182+
| Qwen3-8B | 9.72 | 10.21 | OK |
183+
184+
See [`jointq/README.md`](jointq/README.md) for details.
185+
186+
## Summary
187+
188+
End-to-end execution succeeded on every model attempted under all
189+
five recipes (`gptq/`, `qep_gptq/`, `autobit/`, `autobit_qep/`,
190+
`jointq/`); no crashes or runtime failures observed. In `gptq/`, save
191+
+ load + greedy generation also ran cleanly on every model. For
192+
`jointq/`, save and inference are not yet implemented, so the
193+
quantization step is sanity-checked on the dequantized model.
194+
195+
**Caveat on cross-recipe PPL comparison.** All five recipes use a
196+
compact calibration (`num_calibration_samples=128` and `max_length`
197+
512–1024) that sits well below typical research settings, partly
198+
because the calibration size has to fit the DGX Spark 128 GB UMA
199+
budget for 7–8B models with QEP on. PPL differences between recipes
200+
under this calibration are not large or stable enough to interpret as
201+
which recipe is "better", or to attribute recovery / regression to a
202+
specific mechanism (QEP vs AutoBit vs JointQ, etc.). PPL is reported
203+
here only to confirm that each recipe produces a model that is not
204+
obviously broken on a per-recipe basis. The bullets below describe
205+
per-recipe outcomes only; cross-recipe ordering should not be read
206+
into them.
207+
208+
Per-recipe sanity-check observations (not accuracy claims):
209+
210+
- TinyLlama-1.1B passes the PPL sanity check under every recipe
211+
attempted; under `gptq/`, load + greedy generation produces
212+
sensible text.
213+
- gemma-4-E2B (base) is borderline under both GPTQ-only recipes
214+
(`gptq/` 35.03, `qep_gptq/` 37.82 vs original 25.99; both OK\*).
215+
Under `gptq/`, the saved model must be loaded as `bfloat16` (the
216+
default `float16` triggers a `Half`/`BFloat16` mismatch at
217+
`lm_head`) and greedy generation produces degenerate output
218+
(random tokens / non-Latin scripts), consistent with the
219+
warn-level PPL. Under both AutoBit recipes the PPL diverges to
220+
~10^13 (`autobit/`) and ~10^14 (`autobit_qep/`); under
221+
`autobit_qep/` the bit assignment polarized into 8-bit / 2-bit
222+
halves (observed directly in the run log). The order-of-magnitude
223+
divergence and the polarized bit assignment are not artifacts of
224+
the limited calibration: under `autobit_qep/` the same bimodal
225+
assignment was reproduced when calibration was reduced from
226+
`max_length=2048, num_calibration_samples=512` to the current
227+
`max_length=1024, num_calibration_samples=128`. This model is
228+
therefore NG under AutoBit on this hardware, regardless of QEP.
229+
Under `jointq/` it passes the PPL sanity check (27.88 vs 25.99,
230+
OK).
231+
- Llama-2-7B passes the PPL sanity check under every recipe; under
232+
`gptq/`, load + greedy generation produces sensible text.
233+
- Llama-3-8B is weak under `gptq/` (PPL 27.74 vs original 6.14, OK\*)
234+
with the compact calibration (`max_length=512`,
235+
`num_calibration_samples=128`); within range under each of the
236+
other four recipes (`qep_gptq/` 7.14, `autobit/` 7.27,
237+
`autobit_qep/` 7.24, `jointq/` 6.67). Load + greedy generation
238+
under `gptq/` runs without errors and produces sensible text.
239+
- Qwen3-8B passes the PPL sanity check under every recipe; under
240+
`gptq/`, load + greedy generation produces sensible text.
241+
- Memory: `qep_gptq/` and `autobit_qep/` at the reduced calibration
242+
(`max_length=1024`, `num_calibration_samples=128`) fit within the
243+
DGX Spark 128 GB UMA budget for all 5 models (no OOM at
244+
`mlp.down_proj` for 7–8B); larger calibration will not fit on this
245+
hardware. The QEP-off recipes (`gptq/`, `autobit/`, `jointq/`) at
246+
`max_length=512` fit comfortably within the same budget.
247+
- Save / inference: load + inference for the three QEP/AutoBit
248+
recipes is pending. For `jointq/`, save and inference are not yet
249+
implemented in OneComp; quality there is reported on the
250+
dequantized model only.

0 commit comments

Comments
 (0)