Skip to content

Commit f278628

Browse files
FKKimuraFKKimura
committed
Merge branch 'lab/issue-9-jointq' into 'export/v0-4-4'
Lab/issue 9 jointq See merge request onecomp/onecomp-lab!35
2 parents 165d608 + 80bbb39 commit f278628

62 files changed

Lines changed: 7409 additions & 106 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 31 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,15 @@
1515
- Provides learning-based post-quantization fine-tuning for GPTQ-quantized models
1616
- Public API is exposed as `PostProcessLoraSFT`
1717

18+
### Added JointQ Quantizer
19+
20+
- **Added new `JointQ` quantizer (`onecomp/quantizer/jointq/`)**
21+
- Local-search-based post-training quantization method that minimizes ||Y - hat{W} X^T||_F^2
22+
- Supports both symmetric and asymmetric quantization (1–4 bits)
23+
- Group-wise quantization with configurable group size
24+
- Tikhonov regularization for over-fitting (X^T X + nλI)
25+
- Three initialization strategies: Clip-Optimize, Clip-Optimize with Error Propagation, and GPTQ
26+
1827
### API changes
1928

2029
- Made `Runner.create_quantized_model()` a public method (renamed from `_create_quantized_model`)
@@ -27,11 +36,25 @@
2736
- Counterpart to `save_quantized_model_pt`; uses `torch.load` to restore models with custom modules
2837
- Also available as `onecomp.load_quantized_model_pt()` convenience alias
2938

39+
### Bug Fix: Onebit Quantizer
40+
41+
- Fixed `Onebit` to declare `flag_calibration=True` and `flag_hessian=True` (`onecomp/quantizer/onebit/_onebit.py`)
42+
- Previously, Onebit computed the Hessian internally from `input` despite declaring all flags as `False`, causing a crash when used through `quantize_without_calibration` or chunked quantization paths
43+
- Now uses the Hessian provided by the Runner, consistent with other calibration-based quantizers (GPTQ, DBF, QUIP)
44+
45+
### Quantizer Signature Consistency
46+
47+
- Added `input=None` default to `quantize_layer` in `RTN`, `CQ`, `QBB` (`onecomp/quantizer/{rtn,cq,qbb}/`)
48+
- Aligns with the base `Quantizer.quantize_layer(self, module, input=None, hessian=None)` signature
49+
- Enables these quantizers to be used in `Runner(quantizers=[...])` via the chunked quantization path
50+
- Added `input=None, hessian=None` defaults to `Onebit.quantize_layer` for the same reason
51+
3052
### Examples
3153

3254
- Added `example/post_process/example_lora_sft.py`: End-to-end demo — GPTQ 4-bit quantization + LoRA SFT (WikiText-2) + PPL evaluation + save/load with `save_quantized_model_pt` / `load_quantized_model_pt`
3355
- Added `example/post_process/example_lora_sft_knowledge.py`: Knowledge injection demo — teaches the quantized model about "OneCompression" via LoRA SFT and compares generation before/after
3456
- Added `example/post_process/onecomp_knowledge.jsonl`: Training data describing OneCompression for the knowledge injection example
57+
- Added `example/example_jointq.py`: JointQ 4-bit (groupsize=128) quantization example with dequantized model PPL evaluation
3558

3659
### Documentation
3760

@@ -53,6 +76,14 @@
5376
- Expanded and updated unit tests for GPTQ quantizer (`tests/onecomp/quantizer/gptq/test_gptq.py`)
5477
- Extended boundary and abnormal parameter cases; aligned with `BaseQuantizeSpec` and current GPTQ API
5578
- Adjusted DBF and GPTQ quantizer implementations for test compatibility and consistency (`onecomp/quantizer/dbf/_dbf.py`, `onecomp/quantizer/gptq/_gptq.py`)
79+
- Fixed and improved JointQ unit tests (`tests/onecomp/quantizer/jointq/test_jointq.py`)
80+
- Use `compute_dequantized_weight()` instead of direct `dequantized_weight` access
81+
- Override boundary test to use CUDA with 128×128 layers for group_size compatibility
82+
- Skip CPU-only tests (JointQ is GPU-based)
83+
- Fix `batch_size` validation: `>= 0` → `>= 1` (`onecomp/quantizer/jointq/_jointq.py`)
84+
- Improved JointQ regression test (`tests/onecomp/quantizer/jointq/test_quantize_regression.py`)
85+
- Replaced exact tensor match with MSE-based quality check for environment portability
86+
- Hardcoded expected MSE in helper; removed `.pth` baseline file
5687

5788
## [v0.4.3] 2026-03-26
5889

@@ -224,19 +255,6 @@
224255
- PyTorch does not yet provide wheels for Python 3.14, causing `uv sync` to fail when uv auto-selects CPython 3.14
225256
- Updated `uv.lock` to reflect the new Python version constraint
226257

227-
### Bug Fix: Onebit Quantizer
228-
229-
- Fixed `Onebit` to declare `flag_calibration=True` and `flag_hessian=True` (`onecomp/quantizer/onebit/_onebit.py`)
230-
- Previously, Onebit computed the Hessian internally from `input` despite declaring all flags as `False`, causing a crash when used through `quantize_without_calibration` or chunked quantization paths
231-
- Now uses the Hessian provided by the Runner, consistent with other calibration-based quantizers (GPTQ, DBF, QUIP)
232-
233-
### Quantizer Signature Consistency
234-
235-
- Added `input=None` default to `quantize_layer` in `RTN`, `CQ`, `QBB` (`onecomp/quantizer/{rtn,cq,qbb}/`)
236-
- Aligns with the base `Quantizer.quantize_layer(self, module, input=None, hessian=None)` signature
237-
- Enables these quantizers to be used in `Runner(quantizers=[...])` via the chunked quantization path
238-
- Added `input=None, hessian=None` defaults to `Onebit.quantize_layer` for the same reason
239-
240258
## [v0.3.7] 2026-03-16
241259

242260
### GPU Memory Optimization for Architecture-aware QEP

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ This package is currently under development (version 0) and may behave unstably.
99
- **Quantization Error Propagation (QEP)**: A post-training quantization method that corrects quantization errors by propagating them to subsequent layers, improving the accuracy of quantized LLMs. See [Arai & Ichikawa, NeurIPS 2025](https://openreview.net/forum?id=a3l3K9khbL) for details.
1010
- **vLLM Plugin Integration**: Serve OneComp-quantized models with [vLLM](https://docs.vllm.ai/) via built-in plugins for DBF and Mixed-GPTQ quantization methods.
1111
- **AutoBit**: Mixed-precision quantization with ILP-based bitwidth assignment. Automatically estimates the target bitwidth from available VRAM and assigns per-layer bitwidths to minimize quantization error under the memory budget.
12+
- **JointQ**: Joint quantization method that optimizes weight assignments and scale parameters simultaneously for improved quantization accuracy. Supports group-wise quantization (e.g., 4-bit, groupsize=128).
1213
- **LoRA SFT Post-Process**: Fine-tune quantized models with LoRA adapters for accuracy recovery or domain-specific knowledge injection. Supports SFT loss, teacher distillation, and intermediate block alignment.
1314
- (TBD)
1415

‎benchmark/README.md‎

Lines changed: 98 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -21,38 +21,103 @@ python -c "import hydra; print(hydra.__version__)"
2121

2222
| Directory | Description |
2323
|---|---|
24-
| [llama3-8b-gptq/](llama3-8b-gptq/) | GPTQ (4bit/3bit × gs128/per-channel) |
25-
| [llama3-8b-qep-gptq/](llama3-8b-qep-gptq/) | QEP+GPTQ (4bit/3bit × gs128/per-channel) |
26-
| [llama3-8b-various/](llama3-8b-various/) | Various quantizers with default parameters (no QEP) |
27-
28-
## Llama-3-8B: GPTQ vs QEP+GPTQ
29-
30-
Comparison of GPTQ with and without QEP (Quantization Error Propagation) on [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) (OneComp v0.3.7).
31-
32-
### Perplexity (WikiText-2, ↓ lower is better)
33-
34-
| bits | group_size | GPTQ | QEP+GPTQ |
35-
|---|---|---|---|
36-
| — | — | 6.14 (original) | 6.14 (original) |
37-
| 4 | 128 | 12.66 | **6.66** |
38-
| 4 | per-channel | 665.94 | **7.67** |
39-
| 3 | 128 | 45.22 | **8.95** |
40-
| 3 | per-channel | 1721.06 | **17.93** |
41-
42-
### Accuracy (0-shot, ↑ higher is better)
43-
44-
Values are `acc_norm` where available, `acc` otherwise (winogrande).
45-
46-
| bits | group_size | Method | ARC-c | ARC-e | PIQA | WinoGrande |
47-
|---|---|---|---|---|---|---|
48-
| — | — | Original | 0.5401 | 0.7761 | 0.8063 | 0.7380 |
49-
| 4 | 128 | GPTQ | 0.5026 | 0.7710 | 0.7922 | 0.7206 |
50-
| 4 | 128 | **QEP+GPTQ** | **0.5265** | **0.7942** | **0.7916** | **0.7293** |
51-
| 4 | per-channel | GPTQ | 0.3089 | 0.5076 | 0.6861 | 0.6298 |
52-
| 4 | per-channel | **QEP+GPTQ** | **0.4957** | **0.7542** | **0.7758** | **0.7269** |
53-
| 3 | 128 | GPTQ | 0.3097 | 0.4886 | 0.6610 | 0.6259 |
54-
| 3 | 128 | **QEP+GPTQ** | **0.4352** | **0.6498** | **0.7546** | **0.6946** |
55-
| 3 | per-channel | GPTQ | 0.2167 | 0.2862 | 0.5419 | 0.5004 |
56-
| 3 | per-channel | **QEP+GPTQ** | **0.2688** | **0.4184** | **0.6806** | **0.6156** |
24+
| [llama3-8b-gptq/](llama3-8b-gptq/) | Llama-3-8B GPTQ (4bit/3bit × gs128/per-channel) |
25+
| [llama3-8b-jointq/](llama3-8b-jointq/) | Llama-3-8B JointQ (4bit/3bit × gs128/per-channel) |
26+
| [llama3-8b-qep-gptq/](llama3-8b-qep-gptq/) | Llama-3-8B QEP+GPTQ (4bit/3bit × gs128/per-channel) |
27+
| [llama3-8b-various/](llama3-8b-various/) | Llama-3-8B Various quantizers with default parameters (no QEP) |
28+
| [qwen3-8b-gptq/](qwen3-8b-gptq/) | Qwen3-8B GPTQ (4bit/3bit × gs128/per-channel) |
29+
| [qwen3-8b-jointq/](qwen3-8b-jointq/) | Qwen3-8B JointQ (4bit/3bit × gs128/per-channel) |
30+
| [qwen3-14b-gptq/](qwen3-14b-gptq/) | Qwen3-14B GPTQ (4bit/3bit × gs128/per-channel) |
31+
| [qwen3-14b-jointq/](qwen3-14b-jointq/) | Qwen3-14B JointQ (4bit/3bit × gs128/per-channel) |
32+
33+
## Results Summary
34+
35+
Benchmark results using OneComp v0.3.7 on NVIDIA B200 × 1.
36+
37+
PPL = perplexity on WikiText-2 (↓ lower is better). Accuracy = 0-shot `acc_norm` where available, `acc` otherwise (winogrande) (↑ higher is better).
38+
39+
### Llama-3-8B: GPTQ vs JointQ
40+
41+
| Method | bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
42+
|---|---|---|---|---|---|---|---|---|
43+
| Original | — | — | 6.14 | 0.5401 | 0.7761 | 0.8063 | 0.7380 | — |
44+
| | | | | | | | | |
45+
| GPTQ | 4 | 128 | 12.66 | 0.5026 | 0.7710 | 0.7922 | 0.7206 | **276.8** |
46+
| JointQ | 4 | 128 | **6.67** | **0.5196** | **0.7837** | **0.7954** | **0.7230** | 1196.8 |
47+
| | | | | | | | | |
48+
| GPTQ | 4 | per-channel | 665.94 | 0.3089 | 0.5076 | 0.6861 | 0.6298 | **268.9** |
49+
| JointQ | 4 | per-channel | **8.46** | **0.4753** | **0.7277** | **0.7726** | **0.7269** | 3110.7 |
50+
| | | | | | | | | |
51+
| GPTQ | 3 | 128 | 45.22 | 0.3097 | 0.4886 | 0.6610 | 0.6259 | **273.8** |
52+
| JointQ | 3 | 128 | **9.26** | **0.4454** | **0.6831** | **0.7644** | **0.7064** | 1895.3 |
53+
| | | | | | | | | |
54+
| GPTQ | 3 | per-channel | 1721.06 | 0.2167 | 0.2862 | 0.5419 | 0.5004 | **268.4** |
55+
| JointQ | 3 | per-channel | **21.15** | **0.3567** | **0.5707** | **0.7089** | **0.6946** | 2946.2 |
56+
57+
See [llama3-8b-gptq/](llama3-8b-gptq/) and [llama3-8b-jointq/](llama3-8b-jointq/) for full details.
58+
59+
### Llama-3-8B: GPTQ vs QEP+GPTQ
60+
61+
PPL = perplexity on WikiText-2 (↓ lower is better). Accuracy = 0-shot `acc_norm` where available, `acc` otherwise (winogrande) (↑ higher is better).
62+
63+
| Method | bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
64+
|---|---|---|---|---|---|---|---|---|
65+
| Original | — | — | 6.14 | 0.5401 | 0.7761 | 0.8063 | 0.7380 | — |
66+
| | | | | | | | | |
67+
| GPTQ | 4 | 128 | 12.66 | 0.5026 | 0.7710 | 0.7922 | 0.7206 | **276.8** |
68+
| QEP+GPTQ | 4 | 128 | **6.66** | **0.5265** | **0.7942** | **0.7916** | **0.7293** | 300.7 |
69+
| | | | | | | | | |
70+
| GPTQ | 4 | per-channel | 665.94 | 0.3089 | 0.5076 | 0.6861 | 0.6298 | **268.9** |
71+
| QEP+GPTQ | 4 | per-channel | **7.67** | **0.4957** | **0.7542** | **0.7758** | **0.7269** | 297.7 |
72+
| | | | | | | | | |
73+
| GPTQ | 3 | 128 | 45.22 | 0.3097 | 0.4886 | 0.6610 | 0.6259 | **273.8** |
74+
| QEP+GPTQ | 3 | 128 | **8.95** | **0.4352** | **0.6498** | **0.7546** | **0.6946** | 272.7 |
75+
| | | | | | | | | |
76+
| GPTQ | 3 | per-channel | 1721.06 | 0.2167 | 0.2862 | 0.5419 | 0.5004 | **268.4** |
77+
| QEP+GPTQ | 3 | per-channel | **17.93** | **0.2688** | **0.4184** | **0.6806** | **0.6156** | 292.7 |
5778

5879
See [llama3-8b-gptq/](llama3-8b-gptq/) and [llama3-8b-qep-gptq/](llama3-8b-qep-gptq/) for full details.
80+
81+
### Qwen3-8B: GPTQ vs JointQ
82+
83+
PPL = perplexity on WikiText-2 (↓ lower is better). Accuracy = 0-shot `acc_norm` where available, `acc` otherwise (winogrande) (↑ higher is better).
84+
85+
| Method | bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
86+
|---|---|---|---|---|---|---|---|---|
87+
| Original | — | — | 9.72 | 0.5657 | 0.8093 | 0.7775 | 0.6756 | — |
88+
| | | | | | | | | |
89+
| GPTQ | 4 | 128 | 10.29 | **0.5538** | **0.7946** | **0.7677** | 0.6709 | **275.8** |
90+
| JointQ | 4 | 128 | 10.29 | 0.5469 | 0.7849 | 0.7650 | **0.6819** | 1183.8 |
91+
| | | | | | | | | |
92+
| GPTQ | 4 | per-channel | **10.97** | **0.5085** | 0.7412 | **0.7688** | **0.6693** | **269.0** |
93+
| JointQ | 4 | per-channel | 11.33 | 0.4974 | **0.7483** | 0.7617 | 0.6330 | 2334.3 |
94+
| | | | | | | | | |
95+
| GPTQ | 3 | 128 | **11.71** | 0.4966 | 0.7273 | 0.7486 | 0.6440 | **275.2** |
96+
| JointQ | 3 | 128 | 12.38 | 0.4966 | **0.7395** | **0.7568** | **0.6772** | 1895.9 |
97+
| | | | | | | | | |
98+
| GPTQ | 3 | per-channel | **20.21** | **0.3234** | 0.4293 | **0.6806** | **0.5501** | **269.1** |
99+
| JointQ | 3 | per-channel | 42.10 | 0.2986 | **0.4987** | 0.6779 | 0.5478 | 2443.9 |
100+
101+
See [qwen3-8b-gptq/](qwen3-8b-gptq/) and [qwen3-8b-jointq/](qwen3-8b-jointq/) for full details.
102+
103+
### Qwen3-14B: GPTQ vs JointQ
104+
105+
PPL = perplexity on WikiText-2 (↓ lower is better). Accuracy = 0-shot `acc_norm` where available, `acc` otherwise (winogrande) (↑ higher is better).
106+
107+
| Method | bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
108+
|---|---|---|---|---|---|---|---|---|
109+
| Original | — | — | 8.64 | 0.6032 | 0.8283 | 0.7971 | 0.7293 | — |
110+
| | | | | | | | | |
111+
| GPTQ | 4 | 128 | **8.85** | 0.5947 | 0.8182 | 0.7982 | 0.7316 | **481.9** |
112+
| JointQ | 4 | 128 | 8.88 | **0.6007** | **0.8215** | **0.7992** | **0.7332** | 3024.7 |
113+
| | | | | | | | | |
114+
| GPTQ | 4 | per-channel | **9.15** | 0.5802 | 0.8056 | 0.7873 | 0.7056 | **470.1** |
115+
| JointQ | 4 | per-channel | 9.96 | **0.5845** | **0.8110** | 0.7873 | **0.7080** | 5545.2 |
116+
| | | | | | | | | |
117+
| GPTQ | 3 | 128 | 10.10 | 0.5307 | 0.7727 | **0.7873** | 0.7001 | **480.1** |
118+
| JointQ | 3 | 128 | **9.99** | **0.5401** | **0.7761** | 0.7791 | **0.7088** | 4936.1 |
119+
| | | | | | | | | |
120+
| GPTQ | 3 | per-channel | **13.72** | 0.3976 | 0.5745 | 0.7356 | 0.6172 | **468.4** |
121+
| JointQ | 3 | per-channel | 23.55 | **0.4727** | **0.7247** | **0.7546** | **0.6504** | 5632.7 |
122+
123+
See [qwen3-14b-gptq/](qwen3-14b-gptq/) and [qwen3-14b-jointq/](qwen3-14b-jointq/) for full details.

‎benchmark/llama3-8b-gptq/README.md‎

Lines changed: 39 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,15 @@ GPTQ benchmark for [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llam
44

55
All combinations of `bits × group_size` are run in a single pass, sharing calibration data accumulation across quantizers for efficiency.
66

7+
Two configurations are benchmarked:
8+
9+
1. **GPTQ (default)** — `actorder=false`, `mse=false`
10+
2. **GPTQ (mse+actorder)** — `actorder=true`, `mse=true` (strongest GPTQ setting)
11+
712
## Benchmark Configuration
813

14+
### Common Parameters
15+
916
| Parameter | Values |
1017
|---|---|
1118
| bits | 4, 3 |
@@ -15,7 +22,14 @@ All combinations of `bits × group_size` are run in a single pass, sharing calib
1522
| calibration_strategy | drop_rand |
1623
| max_length | 2048 |
1724

18-
This produces **4 quantizers** (2 bits × 2 group sizes) per run.
25+
This produces **4 quantizers** (2 bits × 2 group sizes) per configuration.
26+
27+
### Configuration-Specific Parameters
28+
29+
| Parameter | default | mse+actorder |
30+
|---|---|---|
31+
| actorder | false | true |
32+
| mse | false | true |
1933

2034
### Evaluation
2135

@@ -28,10 +42,13 @@ Both are computed for the original (unquantized) model and all quantized models.
2842

2943
Requires [Hydra](https://hydra.cc/) (see [benchmark/README.md](../README.md) for installation).
3044

31-
Specify the path to the model via `model_path`:
32-
3345
```bash
46+
# default
3447
python quant_benchmark.py model_path=/path/to/Meta-Llama-3-8B
48+
49+
# mse+actorder
50+
python quant_benchmark.py model_path=/path/to/Meta-Llama-3-8B \
51+
gptq.actorder=true gptq.mse=true output_dir=llama3-8b-mse-actorder
3552
```
3653

3754
### Hydra Overrides
@@ -48,38 +65,31 @@ python quant_benchmark.py model_path=/path/to/model num_calibration_samples=512
4865

4966
## Results
5067

51-
### Perplexity (WikiText-2, ↓ lower is better)
52-
53-
| Model | bits | group_size | PPL |
54-
|---|---|---|---|
55-
| Original | — | — | 6.14 |
56-
| GPTQ | 4 | 128 | 12.66 |
57-
| GPTQ | 4 | per-channel | 665.94 |
58-
| GPTQ | 3 | 128 | 45.22 |
59-
| GPTQ | 3 | per-channel | 1721.06 |
68+
### GPTQ (default)
6069

61-
### Accuracy (0-shot, ↑ higher is better)
70+
PPL = perplexity on WikiText-2 (↓ lower is better). Accuracy = 0-shot `acc_norm` where available, `acc` otherwise (winogrande) (↑ higher is better).
6271

63-
Values are `acc_norm` where available, `acc` otherwise (winogrande).
72+
| bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
73+
|---|---|---|---|---|---|---|---|
74+
| — (Original) | — | 6.14 | 0.5401 | 0.7761 | 0.8063 | 0.7380 | — |
75+
| 4 | 128 | 12.66 | 0.5026 | 0.7710 | 0.7922 | 0.7206 | 276.8 |
76+
| 4 | per-channel | 665.94 | 0.3089 | 0.5076 | 0.6861 | 0.6298 | 268.9 |
77+
| 3 | 128 | 45.22 | 0.3097 | 0.4886 | 0.6610 | 0.6259 | 273.8 |
78+
| 3 | per-channel | 1721.06 | 0.2167 | 0.2862 | 0.5419 | 0.5004 | 268.4 |
6479

65-
| Model | bits | group_size | ARC-c | ARC-e | PIQA | WinoGrande |
66-
|---|---|---|---|---|---|---|
67-
| Original | — | — | 0.5401 | 0.7761 | 0.8063 | 0.7380 |
68-
| GPTQ | 4 | 128 | 0.5026 | 0.7710 | 0.7922 | 0.7206 |
69-
| GPTQ | 4 | per-channel | 0.3089 | 0.5076 | 0.6861 | 0.6298 |
70-
| GPTQ | 3 | 128 | 0.3097 | 0.4886 | 0.6610 | 0.6259 |
71-
| GPTQ | 3 | per-channel | 0.2167 | 0.2862 | 0.5419 | 0.5004 |
80+
Total elapsed time (including calibration data preparation): 3697.5 s (~62 min).
7281

73-
### Quantization Time
82+
### GPTQ (mse+actorder)
7483

75-
| Model | bits | group_size | Time (s) |
76-
|---|---|---|---|
77-
| GPTQ | 4 | 128 | 276.8 |
78-
| GPTQ | 4 | per-channel | 268.9 |
79-
| GPTQ | 3 | 128 | 273.8 |
80-
| GPTQ | 3 | per-channel | 268.4 |
84+
| bits | group_size | PPL | ARC-c | ARC-e | PIQA | WinoGrande | Time (s) |
85+
|---|---|---|---|---|---|---|---|
86+
| — (Original) | — | 6.14 | 0.5401 | 0.7761 | 0.8063 | 0.7380 | — |
87+
| 4 | 128 | 6.55 | 0.5452 | 0.7879 | 0.7987 | 0.7332 | 1412.9 |
88+
| 4 | per-channel | 8.25 | 0.4727 | 0.7475 | 0.7905 | 0.7388 | 366.4 |
89+
| 3 | 128 | 8.02 | 0.4667 | 0.7226 | 0.7699 | 0.7261 | 1663.6 |
90+
| 3 | per-channel | 22.58 | 0.3029 | 0.4811 | 0.6834 | 0.6740 | 369.7 |
8191

82-
Total elapsed time (including calibration data preparation): 3697.5 s (~62 min).
92+
Total elapsed time (including calibration data preparation): 6393.1 s (~107 min).
8393

8494
## Environment
8595

‎benchmark/llama3-8b-jointq/.gitignore‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
run_llama3-8b.sh
2+
run_llama3-8b_actorder.sh
3+
run_llama3-8b_noreg.sh
4+
run_llama3-8b_noreg_actorder.sh
5+
llama3-8b
6+
llama3-8b-actorder
7+
llama3-8b-noreg
8+
llama3-8b-noreg-actorder

0 commit comments

Comments
 (0)