Skip to content

Commit d9abf0d

Browse files
Encourage GNU parallel as the default for multi-run experiments
1 parent 82159ed commit d9abf0d

File tree

1 file changed

+1
-2
lines changed

1 file changed

+1
-2
lines changed

AAE.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -123,11 +123,10 @@ Run the benchmark and collect results:
123123
- Extract the key metric(s) from the log.
124124
- If the run crashes, diagnose from the log tail.
125125

126-
**Parallel runs**: When the benchmark must be run across multiple inputs, seeds, or configurations, use GNU `parallel` to execute them concurrently. For example:
126+
**Parallel runs**: Whenever the benchmark involves multiple inputs, seeds, or configurations, prefer GNU `parallel` to run them concurrently. This dramatically reduces wall-clock time per iteration and should be the default approach unless runs compete for shared resources (e.g. a single GPU). Example:
127127
```bash
128128
parallel --jobs 4 --results run_results/ './bench.sh --input {} > logs/run_{}.log 2>&1' ::: input1 input2 input3 input4
129129
```
130-
This reduces wall-clock time per iteration significantly. Only parallelize when runs are independent and the system has sufficient resources (CPU cores, memory, GPU slots) to avoid contention that would distort measurements.
131130

132131
**Aggregation**: When an experiment produces multiple measurements (e.g. across inputs or seeds), use the **geometric mean** as the default summary statistic. The geometric mean is appropriate for ratios and multiplicative quantities like speedups or normalized scores, and is less sensitive to outliers than the arithmetic mean. Compute it as:
133132
```

0 commit comments

Comments
 (0)