Skip to content

Commit ef05733

Browse files
FKKimuraFKKimura
committed
Update benchmark (v1.1.1)
1 parent da41b49 commit ef05733

7 files changed

Lines changed: 154 additions & 218 deletions

File tree

‎benchmark/README.md‎

Lines changed: 1 addition & 107 deletions
Original file line numberDiff line numberDiff line change
@@ -49,110 +49,4 @@ python -c "import hydra; print(hydra.__version__)"
4949
- `PPL` / `Time` mark the best value in each group in bold.
5050
- Accuracy columns mark the best value in each group in bold, and values that match or exceed `Original` are marked with `*`.
5151

52-
#### Llama-3-8B (4-bit)
53-
54-
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
55-
|---|---|---|---|---|---|---|
56-
| Original | 6.14 | 0.5410 | 0.7757 | 0.8058 | 0.7372 | — |
57-
| GPTQ (gs128) | 12.55 | 0.4974 | 0.7731 | 0.7938 | 0.7174 | **261.0** |
58-
| GPTQ (gs128, mse+actorder) | **6.55** | **0.5427*** | 0.7891* | 0.7971 | 0.7348 | 1334.8 |
59-
| JointQ (gs128, diagonal λ=0.05) | 6.66 | 0.5401 | 0.7870* | **0.7992** | 0.7293 | 908.4 |
60-
| JointQ (gs128, diagonal λ=0.05, mse+actorder) | 6.59 | 0.5410* | **0.8001*** | 0.7933 | **0.7364** | 2088.5 |
61-
62-
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
63-
|---|---|---|---|---|---|---|
64-
| Original | 6.14 | 0.5410 | 0.7757 | 0.8058 | 0.7372 | — |
65-
| GPTQ (per-channel) | 581.81 | 0.3174 | 0.5101 | 0.6828 | 0.6290 | **254.1** |
66-
| GPTQ (per-channel, mse+actorder) | 8.19 | 0.4727 | 0.7391 | **0.7894** | **0.7435*** | 351.2 |
67-
| JointQ (per-channel, diagonal λ=0.01) | 7.86 | **0.5051** | **0.7694** | 0.7889 | 0.7285 | 2119.6 |
68-
| JointQ (per-channel, diagonal λ=0.01, mse+actorder) | **7.27** | 0.4761 | 0.7428 | 0.7797 | 0.7380* | 2348.7 |
69-
70-
#### Llama-3-8B (3-bit)
71-
72-
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
73-
|---|---|---|---|---|---|---|
74-
| Original | 6.14 | 0.5410 | 0.7757 | 0.8058 | 0.7372 | — |
75-
| GPTQ (gs128) | 47.74 | 0.3029 | 0.4886 | 0.6665 | 0.6369 | **259.0** |
76-
| GPTQ (gs128, mse+actorder) | **8.06** | **0.4753** | 0.7176 | 0.7661 | **0.7340** | 1571.1 |
77-
| JointQ (gs128, diagonal λ=0.1) | 8.82 | 0.4428 | 0.6974 | 0.7693 | 0.7222 | 1398.7 |
78-
| JointQ (gs128, diagonal λ=0.1, mse+actorder) | 8.32 | 0.4565 | **0.7302** | **0.7709** | 0.7119 | 2774.2 |
79-
80-
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
81-
|---|---|---|---|---|---|---|
82-
| Original | 6.14 | 0.5410 | 0.7757 | 0.8058 | 0.7372 | — |
83-
| GPTQ (per-channel) | 1640.28 | 0.2312 | 0.2942 | 0.5365 | 0.5099 | **253.4** |
84-
| GPTQ (per-channel, mse+actorder) | 22.53 | 0.3106 | 0.4819 | 0.6855 | 0.6780 | 356.1 |
85-
| JointQ (per-channel, diagonal λ=0.1) | 14.84 | **0.3584** | **0.6351** | **0.7388** | **0.7111** | 2669.9 |
86-
| JointQ (per-channel, diagonal λ=0.1, mse+actorder) | **14.48** | 0.3515 | 0.5715 | 0.7182 | 0.6969 | 3016.9 |
87-
88-
#### Qwen3-8B (4-bit)
89-
90-
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
91-
|---|---|---|---|---|---|---|
92-
| Original | 9.72 | 0.5648 | 0.8093 | 0.7769 | 0.6756 | — |
93-
| GPTQ (gs128) | 10.26 | **0.5580** | 0.7934 | 0.7671 | 0.6669 | **259.4** |
94-
| GPTQ (gs128, mse+actorder) | **9.91** | 0.5384 | **0.8056** | **0.7791*** | 0.6835* | 1517.6 |
95-
| JointQ (gs128, diagonal λ=0.05) | 10.17 | 0.5367 | 0.7795 | 0.7682 | **0.6985*** | 982.5 |
96-
| JointQ (gs128, diagonal λ=0.05, mse+actorder) | 9.96 | 0.5461 | 0.7997 | 0.7742 | 0.6867* | 1960.1 |
97-
98-
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
99-
|---|---|---|---|---|---|---|
100-
| Original | 9.72 | 0.5648 | 0.8093 | 0.7769 | 0.6756 | — |
101-
| GPTQ (per-channel) | 10.88 | 0.5119 | 0.7466 | 0.7622 | 0.6740 | **252.3** |
102-
| GPTQ (per-channel, mse+actorder) | 11.22 | 0.5401 | **0.7828** | **0.7715** | 0.6693 | 395.0 |
103-
| JointQ (per-channel, diagonal λ=0.01) | **10.62** | 0.5324 | 0.7614 | 0.7704 | **0.6843*** | 1262.4 |
104-
| JointQ (per-channel, diagonal λ=0.01, mse+actorder) | 11.05 | **0.5478** | 0.7799 | 0.7688 | 0.6764* | 1672.5 |
105-
106-
#### Qwen3-8B (3-bit)
107-
108-
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
109-
|---|---|---|---|---|---|---|
110-
| Original | 9.72 | 0.5648 | 0.8093 | 0.7769 | 0.6756 | — |
111-
| GPTQ (gs128) | 11.75 | 0.4846 | 0.7222 | 0.7481 | 0.6488 | **257.2** |
112-
| GPTQ (gs128, mse+actorder) | **11.24** | **0.5307** | 0.7597 | **0.7601** | **0.6867*** | 1794.6 |
113-
| JointQ (gs128, diagonal λ=0.1) | 11.73 | 0.5154 | **0.7601** | 0.7563 | **0.6867*** | 1546.6 |
114-
| JointQ (gs128, diagonal λ=0.1, mse+actorder) | 12.08 | 0.5196 | 0.7546 | 0.7579 | 0.6756* | 2455.2 |
115-
116-
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
117-
|---|---|---|---|---|---|---|
118-
| Original | 9.72 | 0.5648 | 0.8093 | 0.7769 | 0.6756 | — |
119-
| GPTQ (per-channel) | **20.02** | 0.3200 | 0.4217 | 0.6703 | 0.5391 | **251.9** |
120-
| GPTQ (per-channel, mse+actorder) | 41.77 | 0.3259 | 0.4524 | 0.6801 | 0.5643 | 401.0 |
121-
| JointQ (per-channel, diagonal λ=0.1) | 23.97 | **0.4420** | **0.6709** | **0.7383** | **0.6425** | 2227.4 |
122-
| JointQ (per-channel, diagonal λ=0.1, mse+actorder) | 28.91 | 0.3942 | 0.5875 | 0.7301 | 0.6014 | 2406.3 |
123-
124-
#### Qwen3-14B (4-bit)
125-
126-
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
127-
|---|---|---|---|---|---|---|
128-
| Original | 8.64 | 0.6024 | 0.8283 | 0.7982 | 0.7285 | — |
129-
| GPTQ (gs128) | 8.84 | 0.5947 | 0.8228 | **0.8003*** | 0.7261 | **430.2** |
130-
| GPTQ (gs128, mse+actorder) | 8.87 | **0.6195*** | 0.8237 | 0.7949 | 0.7285* | 2264.1 |
131-
| JointQ (gs128, diagonal λ=0.05) | **8.83** | 0.6101* | 0.8178 | 0.7954 | 0.7182 | 2784.6 |
132-
| JointQ (gs128, diagonal λ=0.05, mse+actorder) | 8.90 | 0.6067* | **0.8258** | 0.7933 | **0.7348*** | 3779.8 |
133-
134-
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
135-
|---|---|---|---|---|---|---|
136-
| Original | 8.64 | 0.6024 | 0.8283 | 0.7982 | 0.7285 | — |
137-
| GPTQ (per-channel) | **9.11** | 0.5862 | 0.8098 | 0.7862 | 0.6953 | **419.9** |
138-
| GPTQ (per-channel, mse+actorder) | 9.26 | 0.5990 | 0.8215 | 0.7960 | 0.7253 | 583.8 |
139-
| JointQ (per-channel, diagonal λ=0.01) | 9.45 | 0.6075* | **0.8363*** | 0.7911 | 0.7214 | 3116.7 |
140-
| JointQ (per-channel, diagonal λ=0.01, mse+actorder) | 9.14 | **0.6092*** | 0.8220 | **0.8047*** | **0.7324*** | 3769.9 |
141-
142-
#### Qwen3-14B (3-bit)
143-
144-
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
145-
|---|---|---|---|---|---|---|
146-
| Original | 8.64 | 0.6024 | 0.8283 | 0.7982 | 0.7285 | — |
147-
| GPTQ (gs128) | 10.11 | 0.5384 | 0.7774 | 0.7878 | 0.6890 | **427.2** |
148-
| GPTQ (gs128, mse+actorder) | **9.47** | **0.5853** | **0.8241** | **0.7927** | 0.7190 | 2672.7 |
149-
| JointQ (gs128, diagonal λ=0.1) | 9.93 | 0.5538 | 0.7959 | 0.7889 | 0.6993 | 4045.5 |
150-
| JointQ (gs128, diagonal λ=0.1, mse+actorder) | 9.67 | 0.5768 | 0.8161 | 0.7873 | **0.7253** | 4933.7 |
151-
152-
| Configuration | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
153-
|---|---|---|---|---|---|---|
154-
| Original | 8.64 | 0.6024 | 0.8283 | 0.7982 | 0.7285 | — |
155-
| GPTQ (per-channel) | **13.61** | 0.4138 | 0.5690 | 0.7296 | 0.6069 | **419.5** |
156-
| GPTQ (per-channel, mse+actorder) | 15.73 | **0.4872** | 0.7336 | 0.7650 | 0.6551 | 589.7 |
157-
| JointQ (per-channel, diagonal λ=0.1) | 20.94 | 0.4855 | **0.7471** | **0.7688** | **0.7096** | 5049.6 |
158-
| JointQ (per-channel, diagonal λ=0.1, mse+actorder) | 16.93 | 0.4684 | 0.7092 | 0.7650 | 0.6780 | 5633.6 |
52+
Comparison tables will be populated after re-running both GPTQ and JointQ benchmarks.

‎benchmark/llama3-8b-gptq/README.md‎

Lines changed: 50 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,15 @@
11
# Llama-3-8B GPTQ Benchmark
22

3-
GPTQ benchmark for [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) using OneComp v1.1.0.
3+
GPTQ benchmark for [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) using OneComp v1.1.1.
44

55
All combinations of `bits × group_size` are run in a single pass, sharing calibration data accumulation across quantizers for efficiency.
66

7-
Two configurations are benchmarked:
7+
Four configurations are benchmarked (the 2×2 grid of `actorder × mse`):
88

99
1. **GPTQ (default)** — `actorder=false`, `mse=false`
10-
2. **GPTQ (mse+actorder)** — `actorder=true`, `mse=true` (strongest GPTQ setting)
10+
2. **GPTQ (actorder)** — `actorder=true`, `mse=false`
11+
3. **GPTQ (mse)** — `actorder=false`, `mse=true`
12+
4. **GPTQ (mse+actorder)** — `actorder=true`, `mse=true` (strongest GPTQ setting)
1113

1214
## Benchmark Configuration
1315

@@ -21,15 +23,16 @@ Two configurations are benchmarked:
2123
| num_calibration_samples | 1024 |
2224
| calibration_strategy | drop_rand |
2325
| max_length | 2048 |
26+
| dtype | bfloat16 |
2427

2528
This produces **4 quantizers** (2 bits × 2 group sizes) per configuration.
2629

2730
### Configuration-Specific Parameters
2831

29-
| Parameter | default | mse+actorder |
30-
|---|---|---|
31-
| actorder | false | true |
32-
| mse | false | true |
32+
| Parameter | default | actorder | mse | mse+actorder |
33+
|---|---|---|---|---|
34+
| actorder | false | true | false | true |
35+
| mse | false | false | true | true |
3336

3437
### Evaluation
3538

@@ -46,6 +49,14 @@ Requires [Hydra](https://hydra.cc/) (see [benchmark/README.md](../README.md) for
4649
# default
4750
python quant_benchmark.py model_path=/path/to/Meta-Llama-3-8B
4851

52+
# actorder
53+
python quant_benchmark.py model_path=/path/to/Meta-Llama-3-8B \
54+
gptq.actorder=true output_dir=llama3-8b-actorder
55+
56+
# mse
57+
python quant_benchmark.py model_path=/path/to/Meta-Llama-3-8B \
58+
gptq.mse=true output_dir=llama3-8b-mse
59+
4960
# mse+actorder
5061
python quant_benchmark.py model_path=/path/to/Meta-Llama-3-8B \
5162
gptq.actorder=true gptq.mse=true output_dir=llama3-8b-mse-actorder
@@ -65,57 +76,60 @@ python quant_benchmark.py model_path=/path/to/model num_calibration_samples=512
6576

6677
## Results
6778

79+
PPL = perplexity on WikiText-2 (↓ lower is better). Accuracy = 0-shot `acc_norm` where available, `acc` otherwise (winogrande) (↑ higher is better).
80+
6881
### GPTQ (default)
6982

70-
PPL = perplexity on WikiText-2 (↓ lower is better). Accuracy = 0-shot `acc_norm` where available, `acc` otherwise (winogrande) (↑ higher is better).
83+
`actorder=false`, `mse=false`
7184

7285
| bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
7386
|---|---|---|---|---|---|---|---|
74-
| — (Original) | — | 6.14 | 0.5410 | 0.7757 | 0.8058 | 0.7372 | — |
75-
| 4 | 128 | 12.55 | 0.4974 | 0.7731 | 0.7938 | 0.7174 | 261.0 |
76-
| 4 | per-channel | 581.81 | 0.3174 | 0.5101 | 0.6828 | 0.6290 | 254.1 |
77-
| 3 | 128 | 47.74 | 0.3029 | 0.4886 | 0.6665 | 0.6369 | 259.0 |
78-
| 3 | per-channel | 1640.28 | 0.2312 | 0.2942 | 0.5365 | 0.5099 | 253.4 |
87+
| — (Original) | — | 6.14 | 0.5384 | 0.7757 | 0.8058 | 0.7332 | — |
88+
| 4 | 128 | 14.09 | 0.5060 | 0.7559 | 0.7818 | 0.7174 | 326.9 |
89+
| 4 | per-channel | 399.02 | 0.3268 | 0.5206 | 0.6763 | 0.6464 | 318.7 |
90+
| 3 | 128 | 57.69 | 0.2645 | 0.4482 | 0.6159 | 0.5904 | 323.3 |
91+
| 3 | per-channel | 3381.46 | 0.2312 | 0.2753 | 0.5299 | 0.5114 | 322.7 |
7992

80-
Total elapsed time (including calibration data preparation): 3638.8 s (~61 min).
93+
Total elapsed time (including calibration data preparation): 3898.0 s (~65 min).
8194

82-
### GPTQ (mse+actorder)
95+
### GPTQ (actorder)
96+
97+
`actorder=true`, `mse=false`
8398

8499
| bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
85100
|---|---|---|---|---|---|---|---|
86-
| — (Original) | — | 6.14 | 0.5410 | 0.7757 | 0.8058 | 0.7372 | — |
87-
| 4 | 128 | 6.55 | 0.5427 | 0.7891 | 0.7971 | 0.7348 | 1334.8 |
88-
| 4 | per-channel | 8.19 | 0.4727 | 0.7391 | 0.7894 | 0.7435 | 351.2 |
89-
| 3 | 128 | 8.06 | 0.4753 | 0.7176 | 0.7661 | 0.7340 | 1571.1 |
90-
| 3 | per-channel | 22.53 | 0.3106 | 0.4819 | 0.6855 | 0.6780 | 356.1 |
101+
| 4 | 128 | 6.63 | 0.5162 | 0.7694 | 0.7911 | 0.7230 | 293.9 |
102+
| 4 | per-channel | 7.65 | 0.4778 | 0.7391 | 0.7764 | 0.7048 | 287.2 |
103+
| 3 | 128 | 9.58 | 0.4113 | 0.6507 | 0.7388 | 0.6953 | 294.6 |
104+
| 3 | per-channel | 1115.93 | 0.2662 | 0.3931 | 0.6045 | 0.5501 | 286.7 |
91105

92-
Total elapsed time (including calibration data preparation): 6224.4 s (~104 min).
106+
Total elapsed time (including calibration data preparation): 3754.2 s (~63 min).
93107

94-
## Reduced Calibration Benchmark (num_calibration_samples=128)
108+
### GPTQ (mse)
95109

96-
Same configurations as above with `num_calibration_samples` reduced from 1024 to 128.
97-
98-
### GPTQ (default, calib=128)
110+
`actorder=false`, `mse=true`
99111

100112
| bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
101113
|---|---|---|---|---|---|---|---|
102-
| 4 | 128 | 16.07 | 0.5085 | 0.7626 | 0.7938 | 0.7301 | 252.0 |
103-
| 4 | per-channel | 1031.66 | 0.3788 | 0.6157 | 0.7144 | 0.6693 | 243.8 |
104-
| 3 | 128 | 46.90 | 0.2619 | 0.4613 | 0.6295 | 0.5959 | 251.1 |
105-
| 3 | per-channel | 4130.83 | 0.2398 | 0.2769 | 0.5272 | 0.5059 | 243.6 |
114+
| 4 | 128 | 8.84 | 0.5051 | 0.7652 | 0.7916 | 0.7269 | 1415.8 |
115+
| 4 | per-channel | 7.79 | 0.4795 | 0.7226 | 0.7856 | 0.7238 | 368.5 |
116+
| 3 | 128 | 56.76 | 0.4044 | 0.6414 | 0.7519 | 0.6993 | 1668.9 |
117+
| 3 | per-channel | 58.55 | 0.2355 | 0.3763 | 0.5974 | 0.5580 | 373.1 |
118+
119+
Total elapsed time (including calibration data preparation): 6377.4 s (~106 min).
106120

107-
Total elapsed time (including calibration data preparation): 1566.6 s (~26 min).
121+
### GPTQ (mse+actorder)
108122

109-
### GPTQ (mse+actorder, calib=128)
123+
`actorder=true`, `mse=true`
110124

111125
| bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
112126
|---|---|---|---|---|---|---|---|
113-
| 4 | 128 | 6.58 | 0.5239 | 0.7816 | 0.8047 | 0.7348 | 1577.7 |
114-
| 4 | per-channel | 7.92 | 0.4855 | 0.7588 | 0.7835 | 0.7332 | 404.1 |
115-
| 3 | 128 | 8.70 | 0.4488 | 0.7151 | 0.7601 | 0.7064 | 1873.2 |
116-
| 3 | per-channel | 39.73 | 0.3063 | 0.4651 | 0.6730 | 0.6677 | 411.1 |
127+
| 4 | 128 | 6.55 | 0.5469 | 0.7992 | 0.7971 | 0.7316 | 1367.4 |
128+
| 4 | per-channel | 8.19 | 0.4693 | 0.7542 | 0.7905 | 0.7309 | 360.7 |
129+
| 3 | 128 | 8.37 | 0.4744 | 0.7327 | 0.7655 | 0.7364 | 1613.3 |
130+
| 3 | per-channel | 27.55 | 0.3063 | 0.4802 | 0.6915 | 0.6732 | 366.9 |
117131

118-
Total elapsed time (including calibration data preparation): 4859.7 s (~81 min).
132+
Total elapsed time (including calibration data preparation): 6266.2 s (~104 min).
119133

120134
## Environment
121135

‎benchmark/llama3-8b-gptq/quant_benchmark.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,7 @@ def create_quantizers(cfg: DictConfig):
5454
def main(cfg: DictConfig):
5555
print(OmegaConf.to_yaml(cfg))
5656

57-
model_config = ModelConfig(path=cfg.model_path, device=cfg.model_device)
57+
model_config = ModelConfig(path=cfg.model_path, device=cfg.model_device, dtype="bfloat16")
5858

5959
quantizers = create_quantizers(cfg)
6060

0 commit comments

Comments
 (0)