You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Use a simpler pull request checklist, part of the GitHub template.
* Updates to Code Formatting page.
* Improved formatting of benchmarks docs.
* Updates to Contributing a Whats New page.
* Correct PR number curl command.
* What's New entry.
* More clarity about pre-commit further reading.
* Friendlier checklist.
* Better triggering guidance.
* Better line break.
> **The Iris core developers are here to help!** If anything below is unclear, just post a comment asking for help 😊
10
9
11
-
---
12
-
Add any of the below labels to trigger actions on this PR:
10
+
-[ ] Included a [**What's New**](https://scitools-iris.readthedocs.io/en/latest/developers_guide/documenting/whats_new_contributions.html) entry
11
+
-[ ] Incorporated [**type hints**](https://scitools-iris.readthedocs.io/en/latest/developers_guide/contributing_code_formatting.html#type-hinting) in any changed code
12
+
-[ ] Checked if [**tests**](https://scitools-iris.readthedocs.io/en/latest/developers_guide/contributing_tests.html) need updating
13
+
-[ ] Checked if [**benchmarks**](../benchmarks/README.md#writing-benchmarks) need updating
14
+
-[ ] Checked if **documentation** needs updating
15
+
-[Docstrings](https://scitools-iris.readthedocs.io/en/latest/developers_guide/documenting/docstrings.html) (we love code examples!)
-[ ] Checked if [**dependencies**](https://scitools-iris.readthedocs.io/en/latest/developers_guide/contributing_ci_tests.html#gha-test-env) need updating
18
+
-[ ] Confirmed that the GitHub **'checks'** on this PR are passing :white_check_mark:
> - Add this label to trigger benchmarks: https://github.com/SciTools/iris/labels/benchmark_this
26
+
> - Visit this URL - swapping `9999` for this PR's number - to re-trigger the [CLA check](https://github.com/SciTools/.github/wiki/Contributor-Licence-Agreement):
or any time you know you will repeatedly benchmark the same commit. **NOTE:**
81
-
SciTools environments tend to large so this option can consume a lot of disk
82
-
space.
55
+
| Name | Required | Description | Notes |
56
+
|------|----------|-------------|-------|
57
+
|`OVERRIDE_TEST_DATA_REPOSITORY`|**required**| Some benchmarks use `iris-test-data` content, and your local `site.cfg` is not available for benchmark scripts. The benchmark runner defers to any value already set in the shell, but will otherwise download `iris-test-data` and set the variable accordingly. ||
58
+
|`DATA_GEN_PYTHON`|**required**| Path to a Python executable that can be used to generate benchmark test objects/files; see [Data generation](#data-generation). The benchmark runner sets this automatically, but will defer to any value already set in the shell. |[Mule](https://github.com/MetOffice/mule) will be automatically installed into this environment, and sometimes [iris-test-data](https://github.com/SciTools/iris-test-data) (see `OVERRIDE_TEST_DATA_REPOSITORY`). |
59
+
|`BENCHMARK_DATA`| optional | Path to a directory for benchmark synthetic test data, which the benchmark scripts will create if it doesn't already exist. Defaults to `<root>/benchmarks/.data/` if not set. | Some of the generated files, especially in the 'SPerf' suite, are many GB in size so plan accordingly. |
60
+
|`ON_DEMAND_BENCHMARKS`| optional | When set (to any value): benchmarks decorated with `@on_demand_benchmark` are included in the ASV run. Usually coupled with the ASV `--bench` argument to only run the benchmark(s) of interest. Is set during the benchmark runner `cperf` and `sperf` sub-commands. ||
61
+
|`ASV_COMMIT_ENVS`| optional | Instruct the [delegated environment management](#benchmark-environments) to create a dedicated environment for each commit being benchmarked when set (to any value). This means that benchmarking commits with different environment requirements will not be delayed by repeated environment setup - especially relevant given the [benchmark runner](bm_runner.py)'s use of [--interleave-rounds](https://asv.readthedocs.io/en/stable/commands.html?highlight=interleave-rounds#asv-run), or any time you know you will repeatedly benchmark the same commit. |**SciTools environments tend to be large so this option can consume a lot of disk space.**|
83
62
84
63
## Writing benchmarks
85
64
86
-
[See the ASV docs](https://asv.readthedocs.io/) for full detail.
65
+
[See the ASV docs](https://asv.readthedocs.io/en/stable/) for full detail.
87
66
88
67
### What benchmarks to write
89
68
@@ -96,19 +75,21 @@ positive regressions.
96
75
We therefore recommend writing benchmarks representing scripts or single
97
76
operations that are likely to be run at the user level.
98
77
99
-
The drawback of this approach: a reported regression is less likely to reveal
100
-
the root cause (e.g. if a commit caused a regression in coordinate-creation
101
-
time, but the only benchmark covering this was for file-loading). Be prepared
102
-
for manual investigations; and consider committing any useful benchmarks as
103
-
[on-demand benchmarks](#on-demand-benchmarks) for future developers to use.
78
+
> [!CAUTION]
79
+
> The drawback of this approach: a reported regression is less likely to reveal
80
+
> the root cause (e.g. if a commit caused a regression in coordinate-creation
81
+
> time, but the only benchmark covering this was for file-loading). Be prepared
82
+
> for manual investigations; and consider committing any useful benchmarks as
83
+
> [on-demand benchmarks](#on-demand-benchmarks) for future developers to use.
104
84
105
85
### Data generation
106
86
107
-
**Important:** be sure not to use the benchmarking environment to generate any
108
-
test objects/files, as this environment changes with each commit being
109
-
benchmarked, creating inconsistent benchmark 'conditions'. The
110
-
[generate_data](./benchmarks/generate_data/__init__.py) module offers a
111
-
solution; read more detail there.
87
+
> [!WARNING]
88
+
> Be sure not to use the benchmarking environment to generate any
89
+
> test objects/files, as this environment changes with each commit being
90
+
> benchmarked, creating inconsistent benchmark 'conditions'. The
91
+
> [generate_data](./benchmarks/generate_data/__init__.py) module offers a
92
+
> solution; read more detail there.
112
93
113
94
### ASV re-run behaviour
114
95
@@ -136,9 +117,10 @@ detail.
136
117
137
118
### Scaling / non-Scaling Performance Differences
138
119
139
-
**(We no longer advocate the below for benchmarks run during CI, given the
140
-
limited available runtime and risk of false-positives. It remains useful for
141
-
manual investigations).**
120
+
> [!CAUTION]
121
+
> We no longer advocate the below for benchmarks run during CI, given the
122
+
> limited available runtime and risk of false-positives. It remains useful for
123
+
> manual investigations.
142
124
143
125
When comparing performance between commits/file-type/whatever it can be helpful
144
126
to know if the differences exist in scaling or non-scaling parts of the
@@ -160,16 +142,16 @@ suites for the UK Met Office NG-VAT project.
160
142
161
143
## Benchmark environments
162
144
163
-
We have disabled ASV's standard environment management, instead using an
145
+
We have disabled ASV's standard environment management[^1], instead using an
164
146
environment built using the same scripts that set up the package test
165
147
environments.
166
148
This is done using ASV's plugin architecture - see
167
149
[`asv_delegated.py`](asv_delegated.py) and associated
168
150
references in [`asv.conf.json`](asv.conf.json) (`environment_type` and
169
151
`plugins`).
170
152
171
-
(ASV is written to control the environment(s) that benchmarks are run in -
153
+
[^1]: ASV is written to control the environment(s) that benchmarks are run in -
172
154
minimising external factors and also allowing it to compare between a matrix
173
155
of dependencies (each in a separate environment). We have chosen to sacrifice
174
156
these features in favour of testing each commit with its intended dependencies,
175
-
controlled by the test environment setup script(s)).
157
+
controlled by the test environment setup script(s).
0 commit comments