Skip to content

Commit 50d58ee

Browse files
statmlbenstatmlben
committed
add benchmark
1 parent 8af87e3 commit 50d58ee

7 files changed

Lines changed: 2273 additions & 0 deletions

File tree

‎benchmarks/README.md‎

Lines changed: 292 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,292 @@
1+
# ReHLine Mini Benchmark
2+
3+
This folder is a lightweight GridSearchCV benchmark harness for quick ReHLine
4+
version checks. It focuses on wall-clock time for `GridSearchCV.fit`, not on
5+
paper-grade solver comparisons. Regression rows report the best MSE found by
6+
GridSearchCV; classification rows report the best accuracy.
7+
8+
## One-line usage
9+
10+
Edit `benchmarks/mini_config.json`, then run:
11+
12+
```bash
13+
python -m benchmarks.mini_gridsearch
14+
```
15+
16+
This writes a Markdown result file named `rehline-<version>.md` under
17+
`benchmarks/results/`, for example:
18+
19+
```text
20+
benchmarks/results/rehline-0.1.3.dev33-g8af87e306.md
21+
```
22+
23+
The config controls tasks, datasets, hyperparameter grids, X preprocessing, CV
24+
folds, repeats, and the output directory:
25+
26+
```json
27+
{
28+
"task_datasets": {
29+
"ridge_quantile": [
30+
"california_housing",
31+
"make_regression_100k",
32+
"make_friedman1_5k_100"
33+
],
34+
"ridge_quantile_monotonic": [
35+
"california_housing",
36+
"make_regression_100k",
37+
"make_friedman1_5k_100"
38+
],
39+
"elasticnet_quantile": [
40+
"california_housing",
41+
"make_regression_100k",
42+
"make_friedman1_5k_100"
43+
],
44+
"elasticnet_quantile_monotonic": [
45+
"california_housing",
46+
"make_regression_100k",
47+
"make_friedman1_5k_100"
48+
],
49+
"ridge_svm": [
50+
"digits_low_high",
51+
"make_classification_100k",
52+
"openml_bioresponse"
53+
],
54+
"elasticnet_svm": [
55+
"digits_low_high",
56+
"make_classification_100k",
57+
"openml_bioresponse"
58+
]
59+
},
60+
"cv": 2,
61+
"repeats": 1,
62+
"n_jobs": null,
63+
"max_iter": 5000000,
64+
"tol": 0.0001,
65+
"C_grid": [0.1, 1.0, 10.0],
66+
"l1_ratio_grid": [0.5],
67+
"quantile_grid": [0.25],
68+
"preprocess_X": "standard",
69+
"output_dir": "benchmarks/results"
70+
}
71+
```
72+
73+
`task_datasets` locks each task to its own dataset list, so regression tasks do
74+
not accidentally run on all regression datasets and classification tasks do not
75+
accidentally run on all classification datasets. `preprocess_X` supports
76+
`"standard"` (default), `"minmax"`, and `"none"`.
77+
78+
Use a different config:
79+
80+
```bash
81+
python -m benchmarks.mini_gridsearch --config path/to/config.json
82+
```
83+
84+
Run the larger dataset suite:
85+
86+
```bash
87+
python -m benchmarks.mini_gridsearch --config benchmarks/large_config.json
88+
```
89+
90+
`benchmarks/large_config.json` keeps the same task-specific schema but focuses
91+
on heavier datasets such as `openml_buzz_twitter`, `openml_guillermo`,
92+
`make_regression_300k`, `make_classification_300k`, and `covtype_binary_100k`.
93+
Its output goes to `benchmarks/results/large/`.
94+
95+
## Python usage
96+
97+
```python
98+
from benchmarks import run_default_benchmark
99+
100+
results = run_default_benchmark()
101+
print(results)
102+
```
103+
104+
Markdown table output:
105+
106+
```python
107+
from benchmarks import run_default_benchmark
108+
109+
print(run_default_benchmark(as_markdown=True))
110+
```
111+
112+
## Select tasks and datasets
113+
114+
```python
115+
from benchmarks import available_datasets, available_tasks, run_gridsearch_benchmark
116+
117+
tasks = available_tasks()
118+
datasets = available_datasets()
119+
120+
results = run_gridsearch_benchmark(
121+
tasks=[tasks["ridge_quantile"], tasks["elasticnet_svm"]],
122+
datasets=[datasets["diabetes"], datasets["breast_cancer"]],
123+
cv=3,
124+
repeats=3,
125+
)
126+
print(results)
127+
```
128+
129+
Built-in tasks:
130+
131+
- `ridge_quantile`
132+
- `ridge_quantile_monotonic`
133+
- `ridge_quantile_eps`
134+
- `ridge_mae`
135+
- `ridge_huber`
136+
- `ridge_svr`
137+
- `elasticnet_quantile`
138+
- `elasticnet_quantile_monotonic`
139+
- `elasticnet_quantile_eps`
140+
- `elasticnet_mae`
141+
- `elasticnet_huber`
142+
- `elasticnet_svr`
143+
- `ridge_svm`
144+
- `ridge_smooth_svm`
145+
- `ridge_squared_svm`
146+
- `elasticnet_svm`
147+
- `elasticnet_smooth_svm`
148+
- `elasticnet_squared_svm`
149+
150+
The extra tasks mirror the sklearn-compatible examples under
151+
`doc/source/examples`: `MAE.ipynb`, `Huber.ipynb`, `SVR.ipynb`,
152+
`QR_eps.ipynb`, `CustomQR.ipynb`, `MonotonicSVM.ipynb`,
153+
`Smooth_SVM.ipynb`, `Squared_SVM.ipynb`, `GridSearchCV_reg_losses.ipynb`,
154+
and `GridSearchCV_SVM_losses.ipynb`. `ridge_mse` and `elasticnet_mse` are
155+
intentionally excluded from this mini suite because these cases dominated the
156+
runtime in local tests.
157+
Examples such as `CQR.ipynb`, `Path_solution.ipynb`, `Warm_start.ipynb`,
158+
`RankRegression.ipynb`, and `NMF.ipynb` are better handled by separate
159+
benchmark runners because they are not plain sklearn `GridSearchCV` tasks over
160+
one estimator/loss pair.
161+
162+
Built-in datasets:
163+
164+
- `toy_regression`
165+
- `make_regression_10k`
166+
- `make_regression_100k`
167+
- `make_regression_300k`
168+
- `california_housing`
169+
- `diabetes`
170+
- `friedman1`
171+
- `make_friedman1_5k_100`
172+
- `openml_buzz_twitter`
173+
- `sparse_uncorrelated`
174+
- `linnerud_weight`
175+
- `toy_classification`
176+
- `make_classification_100k`
177+
- `make_classification_300k`
178+
- `openml_guillermo`
179+
- `openml_bioresponse`
180+
- `covtype_binary`
181+
- `covtype_binary_50k`
182+
- `covtype_binary_100k`
183+
- `covtype_binary_full`
184+
- `breast_cancer`
185+
- `iris_binary`
186+
- `wine_binary`
187+
- `digits_0_1`
188+
- `digits_low_high`
189+
190+
The default mini benchmark mixes a small loader dataset, medium regression data,
191+
generated 100k-scale dense data, and one compact OpenML classification dataset.
192+
`fetch_covtype` variants and larger OpenML datasets are available for stress
193+
testing, but they are not part of the default mini config because they can
194+
dominate total runtime. `fetch_*` datasets may
195+
download once to the sklearn cache; `openml_*` datasets may download once to
196+
the OpenML cache; `load_*` and `make_*` datasets do not download data.
197+
Multi-class sklearn datasets are exposed as binary variants for the SVM tasks.
198+
199+
Default dataset mix:
200+
201+
| dataset | sklearn source | task | scale | notes |
202+
| --- | --- | --- | --- | --- |
203+
| `california_housing` | `fetch_california_housing` | regression | medium, 20,640 x 8 | downloads once to sklearn cache |
204+
| `make_regression_100k` | `make_regression` | regression | large, 100,000 x 20 | generated locally |
205+
| `make_friedman1_5k_100` | `make_friedman1` | regression | medium/high-dimensional, 5,000 x 100 | generated locally; suitable for default mini benchmark |
206+
| `digits_low_high` | `load_digits` | classification | small, 1,797 x 64 | digits `0-4` vs `5-9`, no download |
207+
| `make_classification_100k` | `make_classification` | classification | large, 100,000 x 20 | generated locally |
208+
| `openml_bioresponse` | `fetch_openml(data_id=4134)` | classification | 3,751 x 1,776 | downloads once to OpenML cache |
209+
210+
Optional stress datasets:
211+
212+
| dataset | sklearn source | task | scale | notes |
213+
| --- | --- | --- | --- | --- |
214+
| `make_regression_300k` | `make_regression` | regression | large, 300,000 x 20 | generated locally |
215+
| `openml_buzz_twitter` | `fetch_openml(data_id=4549)` | regression | 583,250 x 77 | target is `Annotation`; downloads once to OpenML cache |
216+
| `make_classification_300k` | `make_classification` | classification | large, 300,000 x 20 | generated locally |
217+
| `openml_guillermo` | `fetch_openml(data_id=41159)` | classification | 20,000 x 4,296 | high-dimensional; has an ARFF fallback cache for known OpenML md5 mismatch |
218+
| `covtype_binary_50k` | `fetch_covtype` | classification | 50,000 x 54 | fixed subsample after filtering classes 1/2 |
219+
| `covtype_binary_100k` | `fetch_covtype` | classification | 100,000 x 54 | fixed subsample after filtering classes 1/2 |
220+
| `covtype_binary_full` | `fetch_covtype` | classification | 495k x 54 after filtering classes 1/2 | stress-only; can take hours |
221+
| `covtype_binary` | `fetch_covtype` | classification | alias for full binary covtype | kept for compatibility |
222+
223+
## Mini Config Hyperparameter Grids
224+
225+
| task | grid | candidates |
226+
| --- | --- | --- |
227+
| `ridge_quantile` | `C=[0.1, 1, 10]`, `qt=[0.25]` | 3 |
228+
| `ridge_quantile_monotonic` | `C=[0.1, 1, 10]`, `qt=[0.25]`, `constraint=[monotonic increasing]` | 3 |
229+
| `elasticnet_quantile` | `C=[0.1, 1, 10]`, `l1_ratio=[0.5]`, `qt=[0.25]` | 3 |
230+
| `elasticnet_quantile_monotonic` | `C=[0.1, 1, 10]`, `l1_ratio=[0.5]`, `qt=[0.25]`, `constraint=[monotonic increasing]` | 3 |
231+
| `ridge_svm` | `C=[0.1, 1, 10]` | 3 |
232+
| `elasticnet_svm` | `C=[0.1, 1, 10]`, `l1_ratio=[0.5]` | 3 |
233+
234+
Override grids from Python:
235+
236+
```python
237+
from benchmarks import run_default_benchmark
238+
239+
print(
240+
run_default_benchmark(
241+
C_grid=[0.1, 1, 10],
242+
l1_ratio_grid=[0.2, 0.5, 0.8],
243+
quantile_grid=[0.25, 0.5, 0.75],
244+
preprocess_X="standard",
245+
as_markdown=True,
246+
)
247+
)
248+
```
249+
250+
## Command line
251+
252+
```bash
253+
python -m benchmarks.mini_gridsearch --task ridge_quantile --dataset diabetes --cv 3
254+
```
255+
256+
CLI flags override values from the config file.
257+
258+
Override grids from CLI by repeating flags:
259+
260+
```bash
261+
python -m benchmarks.mini_gridsearch \
262+
--task elasticnet_quantile \
263+
--dataset diabetes \
264+
--C 0.1 --C 1 --C 10 \
265+
--l1-ratio 0.2 --l1-ratio 0.5 --l1-ratio 0.8 \
266+
--quantile 0.25 --quantile 0.5 --quantile 0.75 \
267+
--preprocess-X standard
268+
```
269+
270+
The CLI writes Markdown by default. To choose a path:
271+
272+
```bash
273+
python -m benchmarks.mini_gridsearch --task ridge_quantile --dataset diabetes --output results.md
274+
python -m benchmarks.mini_gridsearch --task ridge_quantile --dataset diabetes --output results.csv
275+
```
276+
277+
## Custom datasets
278+
279+
```python
280+
from benchmarks import DatasetSpec, available_tasks, run_gridsearch_benchmark
281+
282+
def my_data():
283+
return X, y
284+
285+
results = run_gridsearch_benchmark(
286+
tasks=[available_tasks()["ridge_quantile"]],
287+
datasets=[DatasetSpec("my_dataset", "regression", my_data)],
288+
)
289+
```
290+
291+
Use `problem_type="regression"` for quantile-regression tasks and
292+
`problem_type="classification"` for SVM tasks.

‎benchmarks/__init__.py‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
"""Small GridSearchCV benchmarks for ReHLine."""
2+
3+
from importlib import import_module
4+
5+
__all__ = [
6+
"BenchmarkTask",
7+
"DatasetSpec",
8+
"available_datasets",
9+
"available_tasks",
10+
"benchmark_results_to_markdown",
11+
"load_benchmark_config",
12+
"make_dataset",
13+
"run_configured_benchmark",
14+
"run_default_benchmark",
15+
"run_gridsearch_benchmark",
16+
]
17+
18+
19+
def __getattr__(name):
20+
if name in __all__:
21+
module = import_module("benchmarks.mini_gridsearch")
22+
return getattr(module, name)
23+
raise AttributeError(f"module 'benchmarks' has no attribute '{name}'")

‎benchmarks/large_config.json‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
{
2+
"task_datasets": {
3+
"ridge_quantile": [
4+
"make_regression_300k",
5+
"openml_buzz_twitter"
6+
],
7+
"elasticnet_quantile": [
8+
"make_regression_300k",
9+
"openml_buzz_twitter"
10+
],
11+
"ridge_svm": [
12+
"make_classification_300k",
13+
"openml_guillermo",
14+
"covtype_binary_100k"
15+
],
16+
"elasticnet_svm": [
17+
"make_classification_300k",
18+
"openml_guillermo",
19+
"covtype_binary_100k"
20+
]
21+
},
22+
"cv": 2,
23+
"repeats": 1,
24+
"n_jobs": null,
25+
"max_iter": 5000000,
26+
"tol": 1e-4,
27+
"C_grid": [0.1, 1.0, 10.0],
28+
"l1_ratio_grid": [0.5],
29+
"quantile_grid": [0.25],
30+
"preprocess_X": "standard",
31+
"output_dir": "benchmarks/results/large"
32+
}

0 commit comments

Comments
 (0)