Skip to content

Commit f5a6a30

Browse files
ccarougeccarouge
committed
Responses to reviews
1 parent 3e369ec commit f5a6a30

2 files changed

Lines changed: 46 additions & 37 deletions

File tree

‎docs/includes/payu.md‎

Lines changed: 29 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,9 @@
55

66
## About
77

8-
<!--start:about-->
8+
<!--start:payu-about-->
9+
## What is _payu_ and how are the simulation files organised by _payu_
10+
911
[_payu_](https://github.com/payu-org/payu) is a workflow manager tool for running numerical models in supercomputing environments. It is an open-source software, distributed under an Apache 2.0 Licence.
1012

1113
For in-depth information about _payu_, check its [technical documentation](https://payu.readthedocs.io/en/stable/).
@@ -48,22 +50,30 @@ A representation of the data organisation for _payu_ is given in the following d
4850
As shown in the diagram, the general layout of a _payu_-supported model run consists of two main directories:
4951

5052
- The _control_ directory contains the model configuration and is the directory from which the model run is started.
51-
This directory contains information to manage the simulation and the scientific options that define the algorithms used in the model component or the diagnostics saved by the model component. The simulation is orchestrated from a main `config.yaml` file contained in this directory. The files specific to each model component are either located directly in the _control_ directory if the model has one model component, or in subdirectories if the model has several model components. The `submodels` section of the `config.yaml` file specifies the name of the submodels and of the subdirectories containing the pertinent files. To modify these options please refer to the configurations documentation of the respective model component.
52-
- The _laboratory_ directory contains all data from _payu_ experiments of the same model. By default, it is `/scratch/$PROJECT/$USER/<model_name>`. `$PROJECT` and `$USER` are environment variables on _Gadi_ that points to your [default project](/getting_started/set_up_nci_account/#change-default-project-on-gadi) and your username respectively. This location can be changed.
53+
This directory contains information to manage the simulation and the scientific options that define the algorithms used in the model component or the diagnostics saved by the model component. In the _control_ directory, you will find:
54+
55+
- `config.yaml` file: it is used to orchestrate the simulation.
56+
- model components' configuration files:
57+
- if the model has only one component: these files are located directly in the _control_ directory
58+
- if the model has several components: these files are in subdirectories. The `submodels` section of the `config.yaml` file specifies the name of the submodels and of the subdirectories containing the pertinent files.
59+
60+
To modify the model components' options, please refer to the configurations documentation of the model.
61+
- The _laboratory_ directory contains all data from _payu_ experiments of the same model. By default, it is `/scratch/$PROJECT/$USER/<model_name>`. `$PROJECT` and `$USER` are environment variables on _Gadi_ that points to your [default project](/getting_started/set_up_nci_account/#change-default-project-on-gadi) and your username respectively. This location can be changed. Inside the _laboratory_ directory, there are two subdirectories of particular interest:
62+
- _work_ &rarr; for temporary storage of files needed by the model while it runs. _payu_ creates this directory at the start of each run and removes it upon their successful completion. It is left untouched in case of error to facilitate the identification of the cause of the model failure
63+
- _archive_ &rarr; for storing the output following each successful run. The output, log and restart files are automatically transferred from _work_ to _archive_ upon successful completion of runs.
64+
65+
The _archive_ and _work_ directories for an experiment are most easily accessed through the symbolic links created in the _control_ directory.
66+
5367
<!--See the section on [modifying the PBS resources](#modify-pbs-resources) to learn how to change the _laboratory_ location. (Should the include be cut to allow adding this sentence and link?)-->
5468

5569
!!! tip
5670

57-
On _Gadi_, it is good practice to put experiment _control_ directories in your `$HOME` directory as this is the only filesystem that is actively backed-up. There is a 10GB limit for home directories, but the _control_ directory only contains text files and symlinks, and so uses relatively little space (<1MB). The _laboratory_ directory is on `/scratch` which is optimised for fast reading and writing of large data and where there is adequate space available for large model output.
58-
71+
On _Gadi_, it is good practice to put experiment _control_ directories in your `$HOME` directory as this is the only filesystem that is actively backed-up. There is a 10GB limit for home directories, but the _control_ directory only contains text files and symlinks, and so uses relatively little space (<1MB).
72+
5973
If you decide to locate your _control_ directory under `/g/data`, be aware of [some complications](https://forum.access-hive.org.au/t/changing-project-codes-for-payu-control-directories-under-g-data/6566) linked to that choice.
74+
<!-- for some unknown reason, I haven't been able to add a blank line. <br> does not work. &nbsp; does not work either.-->
6075

61-
Inside the _laboratory_ directory, there are two subdirectories of particular interest:
62-
63-
- _work_ &rarr; for temporary storage of files needed by the model while it runs. _payu_ creates and removes directories and files in this directory upon successful completion of runs. It is left untouched in case of error to facilitate the identification of the cause of the model failure
64-
- _archive_ &rarr; for storing the output following each successful run. The output, log and restart files are automatically transferred from _work_ to _archive_ upon successful completion of runs.
65-
66-
The _archive_ and _work_ directories for an experiment are most easily accessed through the symbolic links created in the _control_ directory.
76+
The _laboratory_ directory is on `/scratch` which is optimised for fast reading and writing of large data and where there is adequate space available for large model output.
6777

6878
!!! warning
6979
Files on the `/scratch` drive, such as the _laboratory_ directory, might be deleted if not accessed for several days. All experiments which are to be kept should be moved to `/g/data/` by enabling the `sync` step in _payu_.
@@ -74,25 +84,26 @@ Within each of the _work_ and _archive_ directories, _payu_ automatically create
7484

7585
##### Error and output log files
7686

77-
**PBS output files**
87+
*PBS output files*
7888

79-
When the model fails or completes a run, PBS writes the standard output and error streams to two files inside the _control_ directory: `<jobname>.o<job-ID>` and `<jobname>.e<job-ID>`, respectively.
89+
When the model fails or completes a run, PBS writes the standard output and error streams to two files inside the _control_ directory: `<jobname>.o<job-ID>` and `<jobname>.e<job-ID>`, respectively. These files usually contain logs about _payu_ tasks, and give an overview of the resources used by the job.<br>
90+
<br/>
8091

81-
These files usually contain logs about _payu_ tasks, and give an overview of the resources used by the job.<br>
8292
To move these files to the _archive_ directory, use the following commmand:
8393

8494
```
8595
payu sweep
8696
```
8797

88-
**Model log files**
98+
<br>
99+
*Model log files*
89100

90101
While the model is running, the standard output and error streams are saved to file in the _control_ directory. You can examine the contents of these log files to check on the status of a run as it progresses (or after a failed run has completed).
91102

92103
!!! warning
93104
At the end of a successful run, the model log files are archived to the _archive_ directory and will no longer be found in the _control_ directory. If they remain in the _control_ directory after the PBS job for a run has completed, it means the run has failed.
94105

95-
<!--end:about-->
106+
<!--end:payu-about-->
96107

97108
## Prerequisites for _payu_
98109

@@ -102,9 +113,9 @@ While the model is running, the standard output and error streams are saved to f
102113
- **Join NCI projects**<br>
103114
Join the following project by requesting membership on its NCI project page:
104115

105-
<!--start:projects-->
116+
<!--start:payu-projects-->
106117
- [vk83](https://my.nci.org.au/mancini/project/vk83/join)
107-
<!--end:projects-->
118+
<!--end:payu-projects-->
108119
For more information on joining specific NCI projects, refer to [How to connect to a project](https://opus.nci.org.au/display/Help/How+to+connect+to+a+project).
109120

110121
!!! warning

‎docs/models/run_a_model/run_access-esm.md‎

Lines changed: 17 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@
1111
[model components]: /models/access_models/access-esm/#model-components
1212
[model configurations]: /models/access_models/access-esm/#access-esm15
1313

14-
1514
<div class="text-card-group" markdown>
1615

1716
[:fontawesome-brands-github:{: class="twemoji icon-before-text"} {{ model }} configurations]({{github_configs}}){: class="text-card"}
@@ -32,7 +31,6 @@ All {{model}} configurations are open source, licensed under [CC BY 4.0](https:/
3231

3332
{{ model }} release notes are [available on the ACCESS-Hive Forum]({{release_notes}}) and are updated when new releases are made available.
3433

35-
3634
## Prerequisites
3735

3836
- **NCI Account**<br>
@@ -50,8 +48,8 @@ All {{model}} configurations are open source, licensed under [CC BY 4.0](https:/
5048
- [ki32_mosrs](https://my.nci.org.au/mancini/project/ki32_mosrs/join)
5149
{%
5250
include-markdown "includes/payu.md"
53-
start="<!--start:projects-->"
54-
end="<!--end:projects-->"
51+
start="<!--start:payu-projects-->"
52+
end="<!--end:payu-projects-->"
5553
%}
5654
!!! tip
5755
To request membership for the _ki32_mosrs_ subproject, you need to:
@@ -68,8 +66,8 @@ All {{model}} configurations are open source, licensed under [CC BY 4.0](https:/
6866
??? info "What is _payu_ and how are the simulation files organised by _payu_"
6967
{%
7068
include-markdown "includes/payu.md"
71-
start="<!--start:about-->"
72-
end="<!--end:about-->"
69+
start="<!--start:payu-about-->"
70+
end="<!--end:payu-about-->"
7371
%}
7472
For {{model}}, the standard output is saved in the file `access.out` and the standard error in `access.err`.
7573

@@ -83,23 +81,28 @@ All {{model}} configurations are open source, licensed under [CC BY 4.0](https:/
8381

8482
## Get {{ model }} configuration
8583

86-
Released configurations are tested and supported by ACCESS-NRI, as an adaptation of those originally developed by [CSIRO](https://www.csiro.au/en/research/environmental-impacts/climate-change/climate-science-centre) and [CLEX CMS](https://github.com/coecms/access-esm).<br>
84+
Released configurations are tested and supported by ACCESS-NRI, and build upon those originally developed by [CSIRO](https://www.csiro.au/en/research/environmental-impacts/climate-change/climate-science-centre) and [CLEX CMS](https://github.com/coecms/access-esm).<br>
85+
8786
All released {{ model }} configurations are available from the [{{ model }} configs]({{github_configs}}) GitHub repository: `{{github_configs}}`.<br>
8887

8988
[Supported configurations:][model configurations]
9089

9190
| Configuration | Branch name |
9291
|---------------|-------------|
9392
| CMIP6 Concentration-driven pre-industrial | release-preindustrial+concentrations |
94-
| CMIP6 Concentration-driven historical | release-historical+concentrations
93+
| CMIP6 Concentration-driven historical | release-historical+concentrations |
94+
95+
!!! note
96+
97+
The released configurations for ESM1.5 do not reproduce the published CMIP6 model outputs for various reasons. However, the configurations are properly using the CMIP6 experiment protocol in the setup of each supported configuration.
9598

9699
{%
97100
include-markdown "includes/payu.md"
98101
start="<!--start:get-config-payu-->"
99102
end="<!--end:get-config-payu-->"
100103
%}
101104

102-
??? abstract "Experiment example"
105+
??? abstract "Example: Cloning a configuration"
103106
{%
104107
include-markdown "includes/payu.md"
105108
start="<!--start:payu-clone-example-->"
@@ -145,11 +148,6 @@ All released {{ model }} configurations are available from the [{{ model }} conf
145148
end="<!--end:payu-monitor-->"
146149
%}
147150

148-
### Model Live Diagnostics
149-
150-
ACCESS-NRI developed the [Model Live Diagnostics](/model_evaluation/evaluation_on_gadi/model_live_diagnostics) framework to check, monitor, visualise, and evaluate model behaviour and progress of ACCESS models currently running on _Gadi_.<br>
151-
For a complete documentation on how to use this framework, check the [Model Diagnostics documentation](https://med-live-diagnostics.readthedocs.io/en/latest/index.html).
152-
153151
----------------------------------------------------------------------------------------
154152

155153
## Edit {{ model }} configuration {: #edit-{{ model.lower() }}-configuration }
@@ -176,20 +174,20 @@ For a complete documentation on how to use this framework, check the [Model Diag
176174
At the end of each run length, each model component saves its state into a _restart file_, allowing the simulation to be continued in subsequent runs.
177175

178176
!!! warning
179-
The _run length_ (controlled by `runtime`) should be left at 1 year for {{model}} experiments in order to avoid errors. Shorter simulations can be useful when setting up and debugging new experiments, however they require additional configuration changes. See the section [Run for less than one year](#shorter-runs) for details.
177+
The _run length_ (controlled by `runtime`) should be left at 1 year for {{model}} experiments in production in order to avoid errors. However, when testing and debugging new experiments, shorter simulations can be useful. It is possible to set _run length_ to less than a year but additional configuration changes are required. See the section [Run for less than one year](#shorter-runs) for details.
180178

181-
To run the model for longer than the default run length, conduct multiple runs, see
182-
[Run an experiment](#run-an-experiment).
179+
To run the model for longer than the default run length, conduct multiple runs as explained in [Run an experiment](#run-an-experiment). _payu_ has options to manage the length of a simulation for each `payu run` command: _runtime_, _runspersub_ and _-n_. They allow you to have complete control on the length of your experiments.
183180
184181
#### Understand _runtime_, _runspersub_, and _-n_ parameters {: id="multiple-runs"}
185182

186-
It is possible to have more than one model run per queue submit. With the correct use of `runtime`, `runspersub`, `-n` and `walltime` parameters, you can have full control of your experiment.<br>
183+
The `runtime`, `runspersub`, `-n` and `walltime` parameters control various aspects of the simulation related to the length of the simulation: <br>
187184

188185
- `runtime` defines the _run length_.
189186
- `runspersub` defines the maximum number of runs for every [PBS job] submission.
190187
- `-n` sets the total number of runs to be performed.
191188
- `walltime` defines the maximum time of every [PBS job] submission.
192189

190+
By using these parameters correctly, you can fully control the length of your simulation. <br>
193191
Now some practical examples:
194192

195193
- **Run 20 years of simulation with resubmission every 5 years**<br>
@@ -218,7 +216,7 @@ For a complete documentation on how to use this framework, check the [Model Diag
218216
days: 0
219217
```
220218

221-
With the default configuration settings, the sea ice component of {{ model }} will produce restart files only at the end of each year. If valid restart files are required when running shorter simulations, the sea ice model configuration should be modified so that restart files are produced at monthly frequencies. To do this, change the `dumpfreq = 'y'` setting to `dumpfreq = 'm'` in the `cice_in.nml` configuration file located in the `ice` subdirectory of the _control_ directory.
219+
With the default configuration settings, the sea ice component of {{ model }} will produce restart files only at the end of each year. If you may want to continue your short simulation over a longer period, you will need valid restart files created at the end of each run. For this, the sea ice model configuration should be modified so that restart files are produced at monthly frequencies, to match _runtime_. To do this, change the `dumpfreq = 'y'` setting to `dumpfreq = 'm'` in the `cice_in.nml` configuration file located in the `ice` subdirectory of the _control_ directory.
222220

223221
??? info "Specify the restart file"
224222

0 commit comments

Comments
 (0)