Skip to content

Commit 6372882

Browse files
FKKimuraFKKimura
committed
Update benchmark/
1 parent 555b787 commit 6372882

29 files changed

Lines changed: 2599 additions & 678 deletions

‎CHANGELOG.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,8 @@
127127
- Added initialization strategy control: `enable_clip_optimize`, `enable_clip_optimize_ep`, `enable_gptq` parameters to `JointQ` class
128128
- Added `gptq` attribute (`GPTQ` instance) to `JointQ` class for customizing GPTQ parameters (`blocksize`, `percdamp`, `mse`, `q_grid`, `q_norm`). Default GPTQ is auto-created from `bits`/`group_size`/`symmetric`
129129
- Replaced JointQ internal GPTQ module (`jointq/core/gptq.py`) with OneComp GPTQ (`onecomp.quantizer.gptq.GPTQ`); GPTQ initial solution is now generated via the shared GPTQ implementation
130+
- Improved numerical stability of scale optimization for ill-conditioned matrices
131+
- Fixed potential division-by-zero in scale computation
130132

131133
### Breaking Changes
132134

@@ -202,6 +204,15 @@
202204
- Expanded `test_rtn.py`: MSE boundary/abnormal parameters
203205
- Added vLLM mixed group-size tests (`tests/vllm_plugins/gptq/test_mixed_gptq.py`, `tests/vllm_plugins/gptq/test_mixed_gptq_e2e.py`)
204206
- Updated `regression_quantize_helper.py`: updated `EXPECTED_MSE` baseline for OneComp GPTQ integration
207+
208+
### Benchmarking
209+
210+
- Updated all benchmark directories for v1.1.0
211+
- **Results will be added after benchmark runs complete.**
212+
213+
### Dependencies
214+
215+
- Updated `pyproject.toml` and `uv.lock`: added `hydra-core` to `dev` optional dependencies
205216
- Added LPCD tests (`tests/onecomp/lpcd/`, 25 cases)
206217
- `test_lpcd_config.py`: `LPCDConfig` default / custom values, dataclass field set, top-level `from onecomp import LPCDConfig` (CPU only)
207218
- `test_lpcd_metrics.py`: `make_lpcd_metrics()` dispatch on synthetic Llama / Qwen3 blocks for every `enable_*` flag combination, `NotImplementedError` for unsupported architectures, `LpcdMetricGroup.mark_as_ready` / `is_refineable` state transitions (CPU only, no weight download)

‎benchmark/README.md‎

Lines changed: 124 additions & 89 deletions
Original file line numberDiff line numberDiff line change
@@ -32,92 +32,127 @@ python -c "import hydra; print(hydra.__version__)"
3232

3333
## Results Summary
3434

35-
Benchmark results using OneComp v0.3.7 on NVIDIA B200 × 1.
36-
37-
PPL = perplexity on WikiText-2 (↓ lower is better). Accuracy = 0-shot `acc_norm` where available, `acc` otherwise (winogrande) (↑ higher is better).
38-
39-
### Llama-3-8B: GPTQ vs JointQ
40-
41-
| Method | bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
42-
|---|---|---|---|---|---|---|---|---|
43-
| Original | — | — | 6.14 | 0.5401 | 0.7761 | 0.8063 | 0.7380 | — |
44-
| | | | | | | | | |
45-
| GPTQ | 4 | 128 | 12.66 | 0.5026 | 0.7710 | 0.7922 | 0.7206 | **276.8** |
46-
| JointQ | 4 | 128 | **6.67** | **0.5196** | **0.7837** | **0.7954** | **0.7230** | 1196.8 |
47-
| | | | | | | | | |
48-
| GPTQ | 4 | per-channel | 665.94 | 0.3089 | 0.5076 | 0.6861 | 0.6298 | **268.9** |
49-
| JointQ | 4 | per-channel | **8.46** | **0.4753** | **0.7277** | **0.7726** | **0.7269** | 3110.7 |
50-
| | | | | | | | | |
51-
| GPTQ | 3 | 128 | 45.22 | 0.3097 | 0.4886 | 0.6610 | 0.6259 | **273.8** |
52-
| JointQ | 3 | 128 | **9.26** | **0.4454** | **0.6831** | **0.7644** | **0.7064** | 1895.3 |
53-
| | | | | | | | | |
54-
| GPTQ | 3 | per-channel | 1721.06 | 0.2167 | 0.2862 | 0.5419 | 0.5004 | **268.4** |
55-
| JointQ | 3 | per-channel | **21.15** | **0.3567** | **0.5707** | **0.7089** | **0.6946** | 2946.2 |
56-
57-
See [llama3-8b-gptq/](llama3-8b-gptq/) and [llama3-8b-jointq/](llama3-8b-jointq/) for full details.
58-
59-
### Llama-3-8B: GPTQ vs QEP+GPTQ
60-
61-
PPL = perplexity on WikiText-2 (↓ lower is better). Accuracy = 0-shot `acc_norm` where available, `acc` otherwise (winogrande) (↑ higher is better).
62-
63-
| Method | bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
64-
|---|---|---|---|---|---|---|---|---|
65-
| Original | — | — | 6.14 | 0.5401 | 0.7761 | 0.8063 | 0.7380 | — |
66-
| | | | | | | | | |
67-
| GPTQ | 4 | 128 | 12.66 | 0.5026 | 0.7710 | 0.7922 | 0.7206 | **276.8** |
68-
| QEP+GPTQ | 4 | 128 | **6.66** | **0.5265** | **0.7942** | **0.7916** | **0.7293** | 300.7 |
69-
| | | | | | | | | |
70-
| GPTQ | 4 | per-channel | 665.94 | 0.3089 | 0.5076 | 0.6861 | 0.6298 | **268.9** |
71-
| QEP+GPTQ | 4 | per-channel | **7.67** | **0.4957** | **0.7542** | **0.7758** | **0.7269** | 297.7 |
72-
| | | | | | | | | |
73-
| GPTQ | 3 | 128 | 45.22 | 0.3097 | 0.4886 | 0.6610 | 0.6259 | **273.8** |
74-
| QEP+GPTQ | 3 | 128 | **8.95** | **0.4352** | **0.6498** | **0.7546** | **0.6946** | 272.7 |
75-
| | | | | | | | | |
76-
| GPTQ | 3 | per-channel | 1721.06 | 0.2167 | 0.2862 | 0.5419 | 0.5004 | **268.4** |
77-
| QEP+GPTQ | 3 | per-channel | **17.93** | **0.2688** | **0.4184** | **0.6806** | **0.6156** | 292.7 |
78-
79-
See [llama3-8b-gptq/](llama3-8b-gptq/) and [llama3-8b-qep-gptq/](llama3-8b-qep-gptq/) for full details.
80-
81-
### Qwen3-8B: GPTQ vs JointQ
82-
83-
PPL = perplexity on WikiText-2 (↓ lower is better). Accuracy = 0-shot `acc_norm` where available, `acc` otherwise (winogrande) (↑ higher is better).
84-
85-
| Method | bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
86-
|---|---|---|---|---|---|---|---|---|
87-
| Original | — | — | 9.72 | 0.5657 | 0.8093 | 0.7775 | 0.6756 | — |
88-
| | | | | | | | | |
89-
| GPTQ | 4 | 128 | 10.29 | **0.5538** | **0.7946** | **0.7677** | 0.6709 | **275.8** |
90-
| JointQ | 4 | 128 | 10.29 | 0.5469 | 0.7849 | 0.7650 | **0.6819** | 1183.8 |
91-
| | | | | | | | | |
92-
| GPTQ | 4 | per-channel | **10.97** | **0.5085** | 0.7412 | **0.7688** | **0.6693** | **269.0** |
93-
| JointQ | 4 | per-channel | 11.33 | 0.4974 | **0.7483** | 0.7617 | 0.6330 | 2334.3 |
94-
| | | | | | | | | |
95-
| GPTQ | 3 | 128 | **11.71** | 0.4966 | 0.7273 | 0.7486 | 0.6440 | **275.2** |
96-
| JointQ | 3 | 128 | 12.38 | 0.4966 | **0.7395** | **0.7568** | **0.6772** | 1895.9 |
97-
| | | | | | | | | |
98-
| GPTQ | 3 | per-channel | **20.21** | **0.3234** | 0.4293 | **0.6806** | **0.5501** | **269.1** |
99-
| JointQ | 3 | per-channel | 42.10 | 0.2986 | **0.4987** | 0.6779 | 0.5478 | 2443.9 |
100-
101-
See [qwen3-8b-gptq/](qwen3-8b-gptq/) and [qwen3-8b-jointq/](qwen3-8b-jointq/) for full details.
102-
103-
### Qwen3-14B: GPTQ vs JointQ
104-
105-
PPL = perplexity on WikiText-2 (↓ lower is better). Accuracy = 0-shot `acc_norm` where available, `acc` otherwise (winogrande) (↑ higher is better).
106-
107-
| Method | bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
108-
|---|---|---|---|---|---|---|---|---|
109-
| Original | — | — | 8.64 | 0.6032 | 0.8283 | 0.7971 | 0.7293 | — |
110-
| | | | | | | | | |
111-
| GPTQ | 4 | 128 | **8.85** | 0.5947 | 0.8182 | 0.7982 | 0.7316 | **481.9** |
112-
| JointQ | 4 | 128 | 8.88 | **0.6007** | **0.8215** | **0.7992** | **0.7332** | 3024.7 |
113-
| | | | | | | | | |
114-
| GPTQ | 4 | per-channel | **9.15** | 0.5802 | 0.8056 | 0.7873 | 0.7056 | **470.1** |
115-
| JointQ | 4 | per-channel | 9.96 | **0.5845** | **0.8110** | 0.7873 | **0.7080** | 5545.2 |
116-
| | | | | | | | | |
117-
| GPTQ | 3 | 128 | 10.10 | 0.5307 | 0.7727 | **0.7873** | 0.7001 | **480.1** |
118-
| JointQ | 3 | 128 | **9.99** | **0.5401** | **0.7761** | 0.7791 | **0.7088** | 4936.1 |
119-
| | | | | | | | | |
120-
| GPTQ | 3 | per-channel | **13.72** | 0.3976 | 0.5745 | 0.7356 | 0.6172 | **468.4** |
121-
| JointQ | 3 | per-channel | 23.55 | **0.4727** | **0.7247** | **0.7546** | **0.6504** | 5632.7 |
122-
123-
See [qwen3-14b-gptq/](qwen3-14b-gptq/) and [qwen3-14b-jointq/](qwen3-14b-jointq/) for full details.
35+
### GPTQ vs. JointQ
36+
37+
- Referenced GPTQ directories:
38+
- [`llama3-8b-gptq/`](llama3-8b-gptq/)
39+
- [`qwen3-8b-gptq/`](qwen3-8b-gptq/)
40+
- [`qwen3-14b-gptq/`](qwen3-14b-gptq/)
41+
- Referenced JointQ directories:
42+
- [`llama3-8b-jointq/`](llama3-8b-jointq/)
43+
- [`qwen3-8b-jointq/`](qwen3-8b-jointq/)
44+
- [`qwen3-14b-jointq/`](qwen3-14b-jointq/)
45+
- GPTQ rows use the `num_calibration_samples=1024`, `max_length=2048` results from each GPTQ benchmark README.
46+
- JointQ diagonal rows use `λ=0.05` (4-bit) / `λ=0.1` (3-bit) for gs128, and `λ=0.01` (4-bit) / `λ=0.1` (3-bit) for per-channel.
47+
- JointQ diagonal+mse+actorder rows use the same `λ` values.
48+
- Values are judged separately within the `gs128` and `per-channel` groups in each table.
49+
- `PPL` / `Time` mark the best value in each group in bold.
50+
- Accuracy columns mark the best value in each group in bold, and values that match or exceed `Original` are marked with `*`.
51+
52+
#### Llama-3-8B (4-bit)
53+
54+
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
55+
|---|---|---|---|---|---|---|
56+
| Original | 6.14 | 0.5410 | 0.7757 | 0.8058 | 0.7372 | — |
57+
| GPTQ (gs128) | 12.55 | 0.4974 | 0.7731 | 0.7938 | 0.7174 | **261.0** |
58+
| GPTQ (gs128, mse+actorder) | **6.55** | **0.5427*** | 0.7891* | 0.7971 | 0.7348 | 1334.8 |
59+
| JointQ (gs128, diagonal λ=0.05) | 6.66 | 0.5401 | 0.7870* | **0.7992** | 0.7293 | 908.4 |
60+
| JointQ (gs128, diagonal λ=0.05, mse+actorder) | 6.59 | 0.5410* | **0.8001*** | 0.7933 | **0.7364** | 2088.5 |
61+
62+
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
63+
|---|---|---|---|---|---|---|
64+
| Original | 6.14 | 0.5410 | 0.7757 | 0.8058 | 0.7372 | — |
65+
| GPTQ (per-channel) | 581.81 | 0.3174 | 0.5101 | 0.6828 | 0.6290 | **254.1** |
66+
| GPTQ (per-channel, mse+actorder) | 8.19 | 0.4727 | 0.7391 | **0.7894** | **0.7435*** | 351.2 |
67+
| JointQ (per-channel, diagonal λ=0.01) | 7.86 | **0.5051** | **0.7694** | 0.7889 | 0.7285 | 2119.6 |
68+
| JointQ (per-channel, diagonal λ=0.01, mse+actorder) | **7.27** | 0.4761 | 0.7428 | 0.7797 | 0.7380* | 2348.7 |
69+
70+
#### Llama-3-8B (3-bit)
71+
72+
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
73+
|---|---|---|---|---|---|---|
74+
| Original | 6.14 | 0.5410 | 0.7757 | 0.8058 | 0.7372 | — |
75+
| GPTQ (gs128) | 47.74 | 0.3029 | 0.4886 | 0.6665 | 0.6369 | **259.0** |
76+
| GPTQ (gs128, mse+actorder) | **8.06** | **0.4753** | 0.7176 | 0.7661 | **0.7340** | 1571.1 |
77+
| JointQ (gs128, diagonal λ=0.1) | 8.82 | 0.4428 | 0.6974 | 0.7693 | 0.7222 | 1398.7 |
78+
| JointQ (gs128, diagonal λ=0.1, mse+actorder) | 8.32 | 0.4565 | **0.7302** | **0.7709** | 0.7119 | 2774.2 |
79+
80+
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
81+
|---|---|---|---|---|---|---|
82+
| Original | 6.14 | 0.5410 | 0.7757 | 0.8058 | 0.7372 | — |
83+
| GPTQ (per-channel) | 1640.28 | 0.2312 | 0.2942 | 0.5365 | 0.5099 | **253.4** |
84+
| GPTQ (per-channel, mse+actorder) | 22.53 | 0.3106 | 0.4819 | 0.6855 | 0.6780 | 356.1 |
85+
| JointQ (per-channel, diagonal λ=0.1) | 14.84 | **0.3584** | **0.6351** | **0.7388** | **0.7111** | 2669.9 |
86+
| JointQ (per-channel, diagonal λ=0.1, mse+actorder) | **14.48** | 0.3515 | 0.5715 | 0.7182 | 0.6969 | 3016.9 |
87+
88+
#### Qwen3-8B (4-bit)
89+
90+
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
91+
|---|---|---|---|---|---|---|
92+
| Original | 9.72 | 0.5648 | 0.8093 | 0.7769 | 0.6756 | — |
93+
| GPTQ (gs128) | 10.26 | **0.5580** | 0.7934 | 0.7671 | 0.6669 | **259.4** |
94+
| GPTQ (gs128, mse+actorder) | **9.91** | 0.5384 | **0.8056** | **0.7791*** | 0.6835* | 1517.6 |
95+
| JointQ (gs128, diagonal λ=0.05) | 10.17 | 0.5367 | 0.7795 | 0.7682 | **0.6985*** | 982.5 |
96+
| JointQ (gs128, diagonal λ=0.05, mse+actorder) | 9.96 | 0.5461 | 0.7997 | 0.7742 | 0.6867* | 1960.1 |
97+
98+
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
99+
|---|---|---|---|---|---|---|
100+
| Original | 9.72 | 0.5648 | 0.8093 | 0.7769 | 0.6756 | — |
101+
| GPTQ (per-channel) | 10.88 | 0.5119 | 0.7466 | 0.7622 | 0.6740 | **252.3** |
102+
| GPTQ (per-channel, mse+actorder) | 11.22 | 0.5401 | **0.7828** | **0.7715** | 0.6693 | 395.0 |
103+
| JointQ (per-channel, diagonal λ=0.01) | **10.62** | 0.5324 | 0.7614 | 0.7704 | **0.6843*** | 1262.4 |
104+
| JointQ (per-channel, diagonal λ=0.01, mse+actorder) | 11.05 | **0.5478** | 0.7799 | 0.7688 | 0.6764* | 1672.5 |
105+
106+
#### Qwen3-8B (3-bit)
107+
108+
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
109+
|---|---|---|---|---|---|---|
110+
| Original | 9.72 | 0.5648 | 0.8093 | 0.7769 | 0.6756 | — |
111+
| GPTQ (gs128) | 11.75 | 0.4846 | 0.7222 | 0.7481 | 0.6488 | **257.2** |
112+
| GPTQ (gs128, mse+actorder) | **11.24** | **0.5307** | 0.7597 | **0.7601** | **0.6867*** | 1794.6 |
113+
| JointQ (gs128, diagonal λ=0.1) | 11.73 | 0.5154 | **0.7601** | 0.7563 | **0.6867*** | 1546.6 |
114+
| JointQ (gs128, diagonal λ=0.1, mse+actorder) | 12.08 | 0.5196 | 0.7546 | 0.7579 | 0.6756* | 2455.2 |
115+
116+
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
117+
|---|---|---|---|---|---|---|
118+
| Original | 9.72 | 0.5648 | 0.8093 | 0.7769 | 0.6756 | — |
119+
| GPTQ (per-channel) | **20.02** | 0.3200 | 0.4217 | 0.6703 | 0.5391 | **251.9** |
120+
| GPTQ (per-channel, mse+actorder) | 41.77 | 0.3259 | 0.4524 | 0.6801 | 0.5643 | 401.0 |
121+
| JointQ (per-channel, diagonal λ=0.1) | 23.97 | **0.4420** | **0.6709** | **0.7383** | **0.6425** | 2227.4 |
122+
| JointQ (per-channel, diagonal λ=0.1, mse+actorder) | 28.91 | 0.3942 | 0.5875 | 0.7301 | 0.6014 | 2406.3 |
123+
124+
#### Qwen3-14B (4-bit)
125+
126+
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
127+
|---|---|---|---|---|---|---|
128+
| Original | 8.64 | 0.6024 | 0.8283 | 0.7982 | 0.7285 | — |
129+
| GPTQ (gs128) | 8.84 | 0.5947 | 0.8228 | **0.8003*** | 0.7261 | **430.2** |
130+
| GPTQ (gs128, mse+actorder) | 8.87 | **0.6195*** | 0.8237 | 0.7949 | 0.7285* | 2264.1 |
131+
| JointQ (gs128, diagonal λ=0.05) | **8.83** | 0.6101* | 0.8178 | 0.7954 | 0.7182 | 2784.6 |
132+
| JointQ (gs128, diagonal λ=0.05, mse+actorder) | 8.90 | 0.6067* | **0.8258** | 0.7933 | **0.7348*** | 3779.8 |
133+
134+
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
135+
|---|---|---|---|---|---|---|
136+
| Original | 8.64 | 0.6024 | 0.8283 | 0.7982 | 0.7285 | — |
137+
| GPTQ (per-channel) | **9.11** | 0.5862 | 0.8098 | 0.7862 | 0.6953 | **419.9** |
138+
| GPTQ (per-channel, mse+actorder) | 9.26 | 0.5990 | 0.8215 | 0.7960 | 0.7253 | 583.8 |
139+
| JointQ (per-channel, diagonal λ=0.01) | 9.45 | 0.6075* | **0.8363*** | 0.7911 | 0.7214 | 3116.7 |
140+
| JointQ (per-channel, diagonal λ=0.01, mse+actorder) | 9.14 | **0.6092*** | 0.8220 | **0.8047*** | **0.7324*** | 3769.9 |
141+
142+
#### Qwen3-14B (3-bit)
143+
144+
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
145+
|---|---|---|---|---|---|---|
146+
| Original | 8.64 | 0.6024 | 0.8283 | 0.7982 | 0.7285 | — |
147+
| GPTQ (gs128) | 10.11 | 0.5384 | 0.7774 | 0.7878 | 0.6890 | **427.2** |
148+
| GPTQ (gs128, mse+actorder) | **9.47** | **0.5853** | **0.8241** | **0.7927** | 0.7190 | 2672.7 |
149+
| JointQ (gs128, diagonal λ=0.1) | 9.93 | 0.5538 | 0.7959 | 0.7889 | 0.6993 | 4045.5 |
150+
| JointQ (gs128, diagonal λ=0.1, mse+actorder) | 9.67 | 0.5768 | 0.8161 | 0.7873 | **0.7253** | 4933.7 |
151+
152+
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
153+
|---|---|---|---|---|---|---|
154+
| Original | 8.64 | 0.6024 | 0.8283 | 0.7982 | 0.7285 | — |
155+
| GPTQ (per-channel) | **13.61** | 0.4138 | 0.5690 | 0.7296 | 0.6069 | **419.5** |
156+
| GPTQ (per-channel, mse+actorder) | 15.73 | **0.4872** | 0.7336 | 0.7650 | 0.6551 | 589.7 |
157+
| JointQ (per-channel, diagonal λ=0.1) | 20.94 | 0.4855 | **0.7471** | **0.7688** | **0.7096** | 5049.6 |
158+
| JointQ (per-channel, diagonal λ=0.1, mse+actorder) | 16.93 | 0.4684 | 0.7092 | 0.7650 | 0.6780 | 5633.6 |

‎benchmark/llama3-8b-gptq/README.md‎

Lines changed: 40 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Llama-3-8B GPTQ Benchmark
22

3-
GPTQ benchmark for [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) using OneComp v0.3.7.
3+
GPTQ benchmark for [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) using OneComp v1.1.0.
44

55
All combinations of `bits × group_size` are run in a single pass, sharing calibration data accumulation across quantizers for efficiency.
66

@@ -36,7 +36,7 @@ This produces **4 quantizers** (2 bits × 2 group sizes) per configuration.
3636
- Perplexity (WikiText-2)
3737
- Accuracy (lm-eval-harness)
3838

39-
Both are computed for the original (unquantized) model and all quantized models.
39+
Both are computed for the original (unquantized) model and all dequantized models.
4040

4141
## Usage
4242

@@ -71,25 +71,51 @@ PPL = perplexity on WikiText-2 (↓ lower is better). Accuracy = 0-shot `acc_nor
7171

7272
| bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
7373
|---|---|---|---|---|---|---|---|
74-
| — (Original) | — | 6.14 | 0.5401 | 0.7761 | 0.8063 | 0.7380 | — |
75-
| 4 | 128 | 12.66 | 0.5026 | 0.7710 | 0.7922 | 0.7206 | 276.8 |
76-
| 4 | per-channel | 665.94 | 0.3089 | 0.5076 | 0.6861 | 0.6298 | 268.9 |
77-
| 3 | 128 | 45.22 | 0.3097 | 0.4886 | 0.6610 | 0.6259 | 273.8 |
78-
| 3 | per-channel | 1721.06 | 0.2167 | 0.2862 | 0.5419 | 0.5004 | 268.4 |
74+
| — (Original) | — | 6.14 | 0.5410 | 0.7757 | 0.8058 | 0.7372 | — |
75+
| 4 | 128 | 12.55 | 0.4974 | 0.7731 | 0.7938 | 0.7174 | 261.0 |
76+
| 4 | per-channel | 581.81 | 0.3174 | 0.5101 | 0.6828 | 0.6290 | 254.1 |
77+
| 3 | 128 | 47.74 | 0.3029 | 0.4886 | 0.6665 | 0.6369 | 259.0 |
78+
| 3 | per-channel | 1640.28 | 0.2312 | 0.2942 | 0.5365 | 0.5099 | 253.4 |
7979

80-
Total elapsed time (including calibration data preparation): 3697.5 s (~62 min).
80+
Total elapsed time (including calibration data preparation): 3638.8 s (~61 min).
8181

8282
### GPTQ (mse+actorder)
8383

8484
| bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
8585
|---|---|---|---|---|---|---|---|
86-
| — (Original) | — | 6.14 | 0.5401 | 0.7761 | 0.8063 | 0.7380 | — |
87-
| 4 | 128 | 6.55 | 0.5452 | 0.7879 | 0.7987 | 0.7332 | 1412.9 |
88-
| 4 | per-channel | 8.25 | 0.4727 | 0.7475 | 0.7905 | 0.7388 | 366.4 |
89-
| 3 | 128 | 8.02 | 0.4667 | 0.7226 | 0.7699 | 0.7261 | 1663.6 |
90-
| 3 | per-channel | 22.58 | 0.3029 | 0.4811 | 0.6834 | 0.6740 | 369.7 |
86+
| — (Original) | — | 6.14 | 0.5410 | 0.7757 | 0.8058 | 0.7372 | — |
87+
| 4 | 128 | 6.55 | 0.5427 | 0.7891 | 0.7971 | 0.7348 | 1334.8 |
88+
| 4 | per-channel | 8.19 | 0.4727 | 0.7391 | 0.7894 | 0.7435 | 351.2 |
89+
| 3 | 128 | 8.06 | 0.4753 | 0.7176 | 0.7661 | 0.7340 | 1571.1 |
90+
| 3 | per-channel | 22.53 | 0.3106 | 0.4819 | 0.6855 | 0.6780 | 356.1 |
9191

92-
Total elapsed time (including calibration data preparation): 6393.1 s (~107 min).
92+
Total elapsed time (including calibration data preparation): 6224.4 s (~104 min).
93+
94+
## Reduced Calibration Benchmark (num_calibration_samples=128)
95+
96+
Same configurations as above with `num_calibration_samples` reduced from 1024 to 128.
97+
98+
### GPTQ (default, calib=128)
99+
100+
| bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
101+
|---|---|---|---|---|---|---|---|
102+
| 4 | 128 | 16.07 | 0.5085 | 0.7626 | 0.7938 | 0.7301 | 252.0 |
103+
| 4 | per-channel | 1031.66 | 0.3788 | 0.6157 | 0.7144 | 0.6693 | 243.8 |
104+
| 3 | 128 | 46.90 | 0.2619 | 0.4613 | 0.6295 | 0.5959 | 251.1 |
105+
| 3 | per-channel | 4130.83 | 0.2398 | 0.2769 | 0.5272 | 0.5059 | 243.6 |
106+
107+
Total elapsed time (including calibration data preparation): 1566.6 s (~26 min).
108+
109+
### GPTQ (mse+actorder, calib=128)
110+
111+
| bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
112+
|---|---|---|---|---|---|---|---|
113+
| 4 | 128 | 6.58 | 0.5239 | 0.7816 | 0.8047 | 0.7348 | 1577.7 |
114+
| 4 | per-channel | 7.92 | 0.4855 | 0.7588 | 0.7835 | 0.7332 | 404.1 |
115+
| 3 | 128 | 8.70 | 0.4488 | 0.7151 | 0.7601 | 0.7064 | 1873.2 |
116+
| 3 | per-channel | 39.73 | 0.3063 | 0.4651 | 0.6730 | 0.6677 | 411.1 |
117+
118+
Total elapsed time (including calibration data preparation): 4859.7 s (~81 min).
93119

94120
## Environment
95121

‎benchmark/llama3-8b-gptq/quant_benchmark.py‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -86,11 +86,19 @@ def main(cfg: DictConfig):
8686

8787
# Perplexity evaluation
8888
if cfg.calc_ppl:
89-
runner.benchmark_perplexity(original_model=cfg.calc_original_ppl)
89+
runner.benchmark_perplexity(
90+
original_model=cfg.calc_original_ppl,
91+
dequantized_model=True,
92+
quantized_model=False,
93+
)
9094

9195
# Accuracy evaluation
9296
if cfg.calc_acc:
93-
runner.benchmark_accuracy(original_model=cfg.calc_original_acc)
97+
runner.benchmark_accuracy(
98+
original_model=cfg.calc_original_acc,
99+
dequantized_model=True,
100+
quantized_model=False,
101+
)
94102

95103

96104
if __name__ == "__main__":

0 commit comments

Comments
 (0)