Skip to content

Commit 508a557

Browse files
aki916faki916f
committed
Merge branch 'lab/model-validation' into lab/gemma
- Add model_validation suite (autobit, gptq, qep_gptq, autobit_qep, jointq) - Add gptq load and vllm validation phases - Add hydra extra and submit-script ergonomics See merge request onecomp/onecomp-lab!64
2 parents 85b2103 + 3849949 commit 508a557

26 files changed

Lines changed: 1763 additions & 174 deletions

‎.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@ models
1010
.cursor/rules/merge-request-review.mdc
1111
.cursor/rules/test-workflow.mdc
1212
.cursor/rules/gitlab-integration.mdc
13+
.cursor/rules/slurm-submit.mdc
1314
.hydra/
1415
*.out
1516
*.err
@@ -22,3 +23,8 @@ debug_code/
2223
*.pt
2324
# Large test data (download separately if needed for regression tests)
2425
tests/onecomp/quantizer/jointq/data/model_layers_0_self_attn_k_proj.pth
26+
.uv-sync.lock
27+
.uv-sync.done
28+
.uv-sync-vllm.lock
29+
.uv-sync-vllm.done
30+
.venv-vllm/

‎CHANGELOG.md‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,23 @@
179179
- `test_lpcd_metrics.py`: `make_lpcd_metrics()` dispatch on synthetic Llama / Qwen3 blocks for every `enable_*` flag combination, `NotImplementedError` for unsupported architectures, `LpcdMetricGroup.mark_as_ready` / `is_refineable` state transitions (CPU only, no weight download)
180180
- `test_lpcd_runner.py`: end-to-end GPTQ + QEP + LPCD on the first TinyLlama decoder block — smoke (`Runner.run()` completes, all linear layers quantized, dequantized weights finite), QEP + LPCD combination with explicit `QEPConfig`, behavioural checks (residual-only LPCD modifies `o_proj` / `down_proj` beyond the QEP-only baseline while pre-attention `q/k/v_proj` match the baseline bit-for-bit); auto-skipped on non-CUDA hosts via `pytest.mark.skipif`
181181

182+
### Model Validation
183+
184+
- Added `model_validation/README.md`: parent overview of the operational validation suite that exercises OneComp's end-to-end quantize → save → load → inference workflow across multiple architectures and sizes. Provides cross-recipe At-a-Glance status tables (per-model × per-recipe quantization status, and per-model × per-recipe `(save, transformers inference, vllm inference)` status), per-recipe result tables, and a Summary section that explicitly cautions against cross-recipe PPL comparison (PPL is reported only as a per-recipe sanity check; the compact calibration in use sits below typical research settings, partly because the calibration size has to fit the DGX Spark 128 GB UMA budget for 7–8B models with QEP on).
185+
- Added `model_validation/gptq/`: Hydra-driven GPTQ (`wbits=4`, `groupsize=128`, `qep=False`) end-to-end validation across three phases:
186+
- Phase 1 — quantize + save (`validate_gptq.py`, `conf/validate.yaml`): `CalibrationConfig(max_length=512, num_calibration_samples=128)`, saved via `runner.save_quantized_model(...)`, reports original / quantized PPL on `wikitext-2-raw-v1`.
187+
- Phase 2 — load + greedy generation (`validate_load.py`, `conf/validate_load.yaml`): reloads each saved directory via `load_quantized_model` and runs `"Fujitsu is"` with `max_new_tokens=32`. `torch_dtype` is overridable (`float16` / `bfloat16` / `float32` / `null`); gemma-4-E2B requires `bfloat16` because the loader's default `float16` triggers a `Half` / `BFloat16` mismatch at `lm_head`.
188+
- Phase 3 — vLLM offline inference (`validate_vllm.py`, `conf/validate_vllm.yaml`): reloads each saved directory via vLLM's offline `LLM` interface (OneComp's vLLM plugin is auto-registered, so no explicit `quantization=` argument is required) and runs `"Fujitsu is"` with `temperature=0.0`, `max_tokens=32`, `enforce_eager=True`, `max_model_len=512`. `LLM(...)` and `llm.generate(...)` are kept inside `main()` behind `if __name__ == "__main__":` so vLLM worker subprocess re-imports do not recursively spawn new engines. Currently `pending` for all five models.
189+
- Added `model_validation/qep_gptq/`: Hydra-driven GPTQ (`wbits=4`, `groupsize=128`, `qep=True`) end-to-end validation script (`validate_gptq.py`, `conf/validate.yaml`, `README.md`). Calibration: `max_length=1024`, `num_calibration_samples=128` (reduced from defaults to keep 7–8B models within the DGX Spark 128 GB UMA budget with QEP on). Quantize + save only; load / inference is not exercised in this subdirectory yet.
190+
- Added `model_validation/autobit/`: Hydra-driven AutoBit (`target_bit=4`, `qep=False`) end-to-end validation script (`validate_autobit.py`, `conf/validate.yaml`, `README.md`). Candidates `GPTQ(wbits=b, groupsize=128) for b in (2, 3, 4, 8)`, `assignment_strategy="activation_aware"`, `CalibrationConfig(max_length=512, num_calibration_samples=128)`. Quantize + save only.
191+
- Updated `model_validation/autobit_qep/`: AutoBit (`target_bit=4`, `qep=True`) validation. Reduced calibration to `max_length=1024`, `num_calibration_samples=128` to keep 7–8B models within the DGX Spark 128 GB UMA budget. README expanded with per-model bit-assignment counts (`GPTQ_<b>_gs128: <count> layers`) for TinyLlama-1.1B, Llama-2-7B, Llama-3-8B, Qwen3-8B, and gemma-4-E2B; documents the bimodal 8-bit / 2-bit ILP collapse on gemma-4-E2B (every module in the first 15 transformer blocks → 8-bit, remaining 20 blocks → 2-bit; quantized PPL diverges to ~10^14, reproduced after reducing calibration from `max_length=2048, num_calibration_samples=512`) and lists candidate follow-ups (restrict candidate set, disable QEP, switch `assignment_strategy`).
192+
- Added `model_validation/jointq/`: Hydra-driven JointQ (`bits=4`, `group_size=128`, `symmetric=True`, `qep=False`) end-to-end validation script (`validate_jointq.py`, `conf/validate.yaml`, `README.md`). Calibration: `CalibrationConfig(max_length=512, num_calibration_samples=128)`. JointQ does not currently expose a quantized-inference layer (no `save_quantized_model` / `create_quantized_model` path), so quality is sanity-checked on the dequantized model (weights reconstructed from JointQ's quantization parameters); save / inference are reported as `n/a` in the parent At-a-Glance table.
193+
- All five recipes share the same model selection contract: a single model selected via either `model_id` (Hugging Face Hub) or `model_path` (local directory), with any field in `conf/validate*.yaml` overridable on the command line. Default validation set across all recipes is TinyLlama-1.1B, gemma-4-E2B (base), Llama-2-7B, Llama-3-8B, and Qwen3-8B.
194+
195+
### Packaging
196+
197+
- Added `hydra` extra to `pyproject.toml` so `hydra-core` (used by `example/example_autobit.py` and the `model_validation/{gptq,qep_gptq,autobit,autobit_qep,jointq}/validate_*.py` scripts) installs in one step via `uv sync --extra <cuXXX> --extra hydra` or `pip install "onecomp[hydra]"`, instead of a separate `pip install hydra-core` after sync. Documented the new extra in `README.md` and the `model_validation/*/README.md` files. The `model_validation/gptq/` Phase 3 (vLLM inference) additionally requires the `vllm` extra (`uv sync --extra <cuXXX> --extra hydra --extra vllm` or `pip install "onecomp[hydra]" vllm`).
198+
182199
## [v1.0.2] 2026-03-31
183200

184201
### Bug Fix

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -104,6 +104,7 @@ PyTorch will be automatically downloaded by `uv`, so you do not need to install
104104

105105
Adding `--extra dev` installs development tools (black, pytest, pylint).
106106
Adding `--extra visualize` installs matplotlib for visualization features.
107+
Adding `--extra hydra` installs `hydra-core` for the example scripts and `model_validation/` runners that use Hydra-based configuration.
107108

108109
To use vLLM for serving quantized models, add `--extra vllm`:
109110

‎model_validation/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
*/submit*.sh
2+
*/logs/
3+
*/outputs/

‎model_validation/README.md‎

Lines changed: 250 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,250 @@
1+
# Model Validation
2+
3+
An operational validation suite for OneComp's end-to-end workflow
4+
(quantize → save → load → inference) across a variety of model
5+
architectures and sizes. Each subdirectory exercises one workflow
6+
configuration; what is exercised varies per subdirectory and is listed
7+
below.
8+
9+
Sanity checks reported here:
10+
11+
- Perplexity on `wikitext-2-raw-v1`, only as a check that the quantized
12+
model is not broken — **not** an accuracy benchmark.
13+
- Greedy generation from a saved + reloaded model, only as a check
14+
that the load + inference path runs without errors.
15+
16+
## At a Glance
17+
18+
### Quantization
19+
20+
Status legend (quantization step only):
21+
22+
- **OK** — quantize runs end-to-end and the resulting (quantized, or
23+
dequantized for recipes that don't have a quantized-inference path
24+
yet) model passes the PPL sanity check.
25+
- **OK\*** — runs, but PPL is somewhat worse than expected for the
26+
recipe; worth a closer look.
27+
- **NG** — runs, but PPL is clearly broken.
28+
- **pending** — not yet executed.
29+
30+
| Model | GPTQ | QEP+GPTQ | AutoBit | AutoBit+QEP | JointQ |
31+
|---|:---:|:---:|:---:|:---:|:---:|
32+
| TinyLlama-1.1B | OK | OK | OK | OK | OK |
33+
| gemma-4-E2B (base) | OK\* | OK\* | NG | NG | OK |
34+
| Llama-2-7B | OK | OK | OK | OK | OK |
35+
| Llama-3-8B | OK\* | OK | OK | OK | OK |
36+
| Qwen3-8B | OK | OK | OK | OK | OK |
37+
38+
### Save / Inference
39+
40+
Each cell is `(save, transformers inference, vllm inference)`. Each
41+
entry uses OK / OK\* / NG / pending against that step's own success
42+
criterion:
43+
44+
- **save** — save step completes and produces a saved quantized model directory.
45+
- **transformers inference** — the saved model can be loaded and run inference via HF `transformers`.
46+
- **vllm inference** — the saved model can be loaded and run inference via vLLM.
47+
- **n/a** — the recipe has no implementation for that step yet.
48+
49+
| Model | GPTQ | QEP+GPTQ | AutoBit | AutoBit+QEP | JointQ |
50+
|---|:---:|:---:|:---:|:---:|:---:|
51+
| TinyLlama-1.1B | (OK, OK, pending) | (OK, pending, pending) | (OK, pending, pending) | (OK, pending, pending) | (n/a, n/a, n/a) |
52+
| gemma-4-E2B (base) | (OK, OK\*, pending) | (OK, pending, pending) | (OK, pending, pending) | (OK, pending, pending) | (n/a, n/a, n/a) |
53+
| Llama-2-7B | (OK, OK, pending) | (OK, pending, pending) | (OK, pending, pending) | (OK, pending, pending) | (n/a, n/a, n/a) |
54+
| Llama-3-8B | (OK, OK, pending) | (OK, pending, pending) | (OK, pending, pending) | (OK, pending, pending) | (n/a, n/a, n/a) |
55+
| Qwen3-8B | (OK, OK, pending) | (OK, pending, pending) | (OK, pending, pending) | (OK, pending, pending) | (n/a, n/a, n/a) |
56+
57+
JointQ does not yet provide a quantized-inference layer (no
58+
`save_quantized_model` / `create_quantized_model` path), so save and
59+
inference are listed as `n/a` until those are implemented; the
60+
quantization step for JointQ is sanity-checked on the dequantized
61+
model instead.
62+
63+
See per-recipe details in the [Results](#results) section below.
64+
65+
## Subdirectories
66+
67+
| Directory | Quantization recipe | Steps exercised |
68+
|---|---|---|
69+
| [`gptq/`](gptq/) | `GPTQ(wbits=4, groupsize=128)`, `qep=False` | quantize, save, load + greedy generation, PPL sanity check |
70+
| [`qep_gptq/`](qep_gptq/) | `GPTQ(wbits=4, groupsize=128)`, `qep=True` | quantize, save, PPL sanity check |
71+
| [`autobit/`](autobit/) | `AutoBitQuantizer(target_bit=4)`, candidates `GPTQ(wbits=b, groupsize=128) for b in (2,3,4,8)`, `assignment_strategy="activation_aware"`, `qep=False` | quantize, save, PPL sanity check |
72+
| [`autobit_qep/`](autobit_qep/) | `AutoBitQuantizer(target_bit=4)`, candidates `GPTQ(wbits=b, groupsize=128) for b in (2,3,4,8)`, `assignment_strategy="activation_aware"`, `qep=True` | quantize, save, PPL sanity check |
73+
| [`jointq/`](jointq/) | `JointQ(bits=4, group_size=128, symmetric=True)`, `qep=False` | quantize, PPL sanity check (on dequantized model). Save / inference: not yet implemented. |
74+
75+
## Results
76+
77+
### [`gptq/`](gptq/)
78+
79+
Two phases:
80+
81+
1. Quantize + save (`validate_gptq.py`). Calibration: `max_length=512`,
82+
`num_calibration_samples=128`.
83+
2. Load + greedy generation (`validate_load.py`). Prompt
84+
`"Fujitsu is"`, `max_new_tokens=32`.
85+
86+
Phase 1 — quantization PPL on `wikitext-2-raw-v1`:
87+
88+
| Model | Original PPL | Quantized PPL | Status |
89+
|---|---:|---:|---|
90+
| TinyLlama-1.1B | 7.77 | 8.69 | OK |
91+
| gemma-4-E2B (base) | 25.99 | 35.03 | OK\* |
92+
| Llama-2-7B | 5.47 | 6.59 | OK |
93+
| Llama-3-8B | 6.14 | 27.74 | OK\* |
94+
| Qwen3-8B | 9.72 | 10.72 | OK |
95+
96+
Phase 2 — load + greedy generation:
97+
98+
| Model | torch_dtype | Status |
99+
|---|---|---|
100+
| TinyLlama-1.1B | (default) | OK |
101+
| gemma-4-E2B | `bfloat16` | OK\* |
102+
| Llama-2-7B | (default) | OK |
103+
| Llama-3-8B | (default) | OK |
104+
| Qwen3-8B | (default) | OK |
105+
106+
Notes:
107+
108+
- gemma-4-E2B must be loaded as `bfloat16`; the loader's default
109+
`float16` triggers a `Half`/`BFloat16` mismatch at `lm_head`. With
110+
`bfloat16`, load + generation runs without errors but the output is
111+
degenerate (random tokens / non-Latin scripts), consistent with the
112+
warn-level PPL in phase 1.
113+
- All other models load and generate sensibly under the default dtype.
114+
115+
See [`gptq/README.md`](gptq/README.md) for full details, generated
116+
samples, and discussion.
117+
118+
### [`qep_gptq/`](qep_gptq/)
119+
120+
Quantize + save (`validate_gptq.py`) with QEP on. Calibration:
121+
`max_length=1024`, `num_calibration_samples=128`. Load + inference is
122+
not exercised in this subdirectory yet.
123+
124+
| Model | Original PPL | Quantized PPL | Status |
125+
|---|---:|---:|---|
126+
| TinyLlama-1.1B | 7.77 | 8.63 | OK |
127+
| gemma-4-E2B (base) | 25.99 | 37.82 | OK\* |
128+
| Llama-2-7B | 5.47 | 6.10 | OK |
129+
| Llama-3-8B | 6.14 | 7.14 | OK |
130+
| Qwen3-8B | 9.72 | 10.83 | OK |
131+
132+
See [`qep_gptq/README.md`](qep_gptq/README.md) for details.
133+
134+
### [`autobit/`](autobit/)
135+
136+
Quantize + save (`validate_autobit.py`) with QEP off. Calibration:
137+
`max_length=512`, `num_calibration_samples=128`. Load + inference is
138+
not exercised in this subdirectory yet.
139+
140+
| Model | Original PPL | Quantized PPL | Status |
141+
|---|---:|---:|---|
142+
| TinyLlama-1.1B | 7.77 | 8.75 | OK |
143+
| gemma-4-E2B (base) | 25.99 | 2.40e13 | NG |
144+
| Llama-2-7B | 5.47 | 5.94 | OK |
145+
| Llama-3-8B | 6.14 | 7.27 | OK |
146+
| Qwen3-8B | 9.72 | 10.74 | OK |
147+
148+
See [`autobit/README.md`](autobit/README.md) for details.
149+
150+
### [`autobit_qep/`](autobit_qep/)
151+
152+
Quantize + save (`validate_autobit.py`) with QEP on. Calibration:
153+
`max_length=1024`, `num_calibration_samples=128` (reduced from defaults
154+
to keep 7-8B models within the DGX Spark 128 GB UMA budget). Load +
155+
inference is not exercised in this subdirectory yet.
156+
157+
| Model | Original PPL | Quantized PPL | Status |
158+
|---|---:|---:|---|
159+
| TinyLlama-1.1B | 7.77 | 8.67 | OK |
160+
| gemma-4-E2B (base) | 25.99 | 1.64e14 | NG |
161+
| Llama-2-7B | 5.47 | 5.90 | OK |
162+
| Llama-3-8B | 6.14 | 7.24 | OK |
163+
| Qwen3-8B | 9.72 | 10.82 | OK |
164+
165+
See [`autobit_qep/README.md`](autobit_qep/README.md) for details and
166+
discussion.
167+
168+
### [`jointq/`](jointq/)
169+
170+
Quantize only (`validate_jointq.py`) with `qep=False`. Calibration:
171+
`max_length=512`, `num_calibration_samples=128`. Save and inference
172+
are **not yet implemented** for JointQ (no `save_quantized_model` /
173+
`create_quantized_model` path), so quality is sanity-checked on the
174+
dequantized model.
175+
176+
| Model | Original PPL | Dequantized PPL | Status |
177+
|---|---:|---:|---|
178+
| TinyLlama-1.1B | 7.77 | 8.25 | OK |
179+
| gemma-4-E2B (base) | 25.99 | 27.88 | OK |
180+
| Llama-2-7B | 5.47 | 5.64 | OK |
181+
| Llama-3-8B | 6.14 | 6.67 | OK |
182+
| Qwen3-8B | 9.72 | 10.21 | OK |
183+
184+
See [`jointq/README.md`](jointq/README.md) for details.
185+
186+
## Summary
187+
188+
End-to-end execution succeeded on every model attempted under all
189+
five recipes (`gptq/`, `qep_gptq/`, `autobit/`, `autobit_qep/`,
190+
`jointq/`); no crashes or runtime failures observed. In `gptq/`, save
191+
+ load + greedy generation also ran cleanly on every model. For
192+
`jointq/`, save and inference are not yet implemented, so the
193+
quantization step is sanity-checked on the dequantized model.
194+
195+
**Caveat on cross-recipe PPL comparison.** All five recipes use a
196+
compact calibration (`num_calibration_samples=128` and `max_length`
197+
512–1024) that sits well below typical research settings, partly
198+
because the calibration size has to fit the DGX Spark 128 GB UMA
199+
budget for 7–8B models with QEP on. PPL differences between recipes
200+
under this calibration are not large or stable enough to interpret as
201+
which recipe is "better", or to attribute recovery / regression to a
202+
specific mechanism (QEP vs AutoBit vs JointQ, etc.). PPL is reported
203+
here only to confirm that each recipe produces a model that is not
204+
obviously broken on a per-recipe basis. The bullets below describe
205+
per-recipe outcomes only; cross-recipe ordering should not be read
206+
into them.
207+
208+
Per-recipe sanity-check observations (not accuracy claims):
209+
210+
- TinyLlama-1.1B passes the PPL sanity check under every recipe
211+
attempted; under `gptq/`, load + greedy generation produces
212+
sensible text.
213+
- gemma-4-E2B (base) is borderline under both GPTQ-only recipes
214+
(`gptq/` 35.03, `qep_gptq/` 37.82 vs original 25.99; both OK\*).
215+
Under `gptq/`, the saved model must be loaded as `bfloat16` (the
216+
default `float16` triggers a `Half`/`BFloat16` mismatch at
217+
`lm_head`) and greedy generation produces degenerate output
218+
(random tokens / non-Latin scripts), consistent with the
219+
warn-level PPL. Under both AutoBit recipes the PPL diverges to
220+
~10^13 (`autobit/`) and ~10^14 (`autobit_qep/`); under
221+
`autobit_qep/` the bit assignment polarized into 8-bit / 2-bit
222+
halves (observed directly in the run log). The order-of-magnitude
223+
divergence and the polarized bit assignment are not artifacts of
224+
the limited calibration: under `autobit_qep/` the same bimodal
225+
assignment was reproduced when calibration was reduced from
226+
`max_length=2048, num_calibration_samples=512` to the current
227+
`max_length=1024, num_calibration_samples=128`. This model is
228+
therefore NG under AutoBit on this hardware, regardless of QEP.
229+
Under `jointq/` it passes the PPL sanity check (27.88 vs 25.99,
230+
OK).
231+
- Llama-2-7B passes the PPL sanity check under every recipe; under
232+
`gptq/`, load + greedy generation produces sensible text.
233+
- Llama-3-8B is weak under `gptq/` (PPL 27.74 vs original 6.14, OK\*)
234+
with the compact calibration (`max_length=512`,
235+
`num_calibration_samples=128`); within range under each of the
236+
other four recipes (`qep_gptq/` 7.14, `autobit/` 7.27,
237+
`autobit_qep/` 7.24, `jointq/` 6.67). Load + greedy generation
238+
under `gptq/` runs without errors and produces sensible text.
239+
- Qwen3-8B passes the PPL sanity check under every recipe; under
240+
`gptq/`, load + greedy generation produces sensible text.
241+
- Memory: `qep_gptq/` and `autobit_qep/` at the reduced calibration
242+
(`max_length=1024`, `num_calibration_samples=128`) fit within the
243+
DGX Spark 128 GB UMA budget for all 5 models (no OOM at
244+
`mlp.down_proj` for 7–8B); larger calibration will not fit on this
245+
hardware. The QEP-off recipes (`gptq/`, `autobit/`, `jointq/`) at
246+
`max_length=512` fit comfortably within the same budget.
247+
- Save / inference: load + inference for the three QEP/AutoBit
248+
recipes is pending. For `jointq/`, save and inference are not yet
249+
implemented in OneComp; quality there is reported on the
250+
dequantized model only.

0 commit comments

Comments
 (0)