Skip to content

Commit 6cbcc5e

Browse files
committed
Added benchmark guidelines to doc
1 parent 87bb04d commit 6cbcc5e

File tree

2 files changed

+35
-0
lines changed

2 files changed

+35
-0
lines changed

docs/guidelines.md

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
2+
3+
## Hyperparameters
4+
You should keep all thunder default hyperparameters for all tasks and datasets, except for the `segpath_epithelial` and `segpath_lymphocytes` datasets (`segmentation` task) for which you must override the number of epochs to `9` and `21` respectively (because of the size of the datasets). See [this tutorial](https://mics-lab.github.io/thunder/custom_config/) for more details about how to override the default configuration.
5+
6+
**CLI**
7+
```bash
8+
# Evaluating hiboub on segpath_epithelial
9+
thunder benchmark hiboub segpath_epithelial segmentation --loading-mode=embedding_pre_loading --adaptation.epochs=9
10+
11+
# Evaluating hiboub on segpath_lymphocytes
12+
thunder benchmark hiboub segpath_lymphocytes segmentation --loading-mode=embedding_pre_loading --adaptation.epochs=21
13+
```
14+
15+
**API**
16+
```python
17+
import thunder
18+
19+
# Evaluating hiboub on segpath_epithelial
20+
thunder.benchmark(
21+
"hiboub",
22+
dataset="segpath_epithelial",
23+
task="segmentation",
24+
loading_mode="embedding_pre_loading",
25+
**{"adaptation.epochs": 9})
26+
27+
# Evaluating hiboub on segpath_lymphocytes
28+
thunder.benchmark(
29+
"hiboub",
30+
dataset="segpath_lymphocytes",
31+
task="segmentation",
32+
loading_mode="embedding_pre_loading",
33+
**{"adaptation.epochs": 21})
34+
```

mkdocs.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ nav:
2323
- Leaderboards: leaderboards.md
2424
- Tutorials:
2525
- Getting Started: getting_started.md
26+
- Guidelines to benchmark an exisiting (reproducing our results) or new model: guidelines.md
2627
- Data Loading (Linear Probing & Segmentation): data_loading.md
2728
- Examples: examples.md
2829
- Benchmarking a Custom Model: custom_model.md

0 commit comments

Comments
 (0)