+
+ +To move these files to the _archive_ directory, use the following commmand: + +``` +payu sweep +``` + +
+*Model log files* + +While the model is running, the standard output and error streams are saved to file in the _control_ directory. You can examine the contents of these log files to check on the status of a run as it progresses (or after a failed run has completed). + +!!! warning + At the end of a successful run, the model log files are archived to the _archive_ directory and will no longer be found in the _control_ directory. If they remain in the _control_ directory after the PBS job for a run has completed, it means the run has failed. + + + +## Prerequisites for _payu_ + +- **NCI account**
+ Before running a _payu_ experiment, you need to [Set Up your NCI Account](/getting_started/set_up_nci_account). + +- **Join NCI projects**
+ Join the following project by requesting membership on its NCI project page: + + + - [vk83](https://my.nci.org.au/mancini/project/vk83/join) + + For more information on joining specific NCI projects, refer to [How to connect to a project](https://opus.nci.org.au/display/Help/How+to+connect+to+a+project). + + !!! warning + Different model configurations will likely require you to **join additional projects**. Please refer to the [Run a Model][Run a Model] page of your chosen model for the list of additional projects. + + +### Accessing _payu_ + +_Payu_ on _Gadi_ is available through a dedicated environment in the _vk83_ project.
+After joining the _vk83_ project, load the _payu_ module: + + module use /g/data/vk83/modules + module load payu + +To check that _payu_ is available, run: + + payu --version + + +## Run an experiment + +### Get the model configuration + + +Before downloading (cloning) a local copy of a configuration, you need to: + +- Know the `
+ The `-f` option to `payu run` lets the model run in all cases and delete any existing data under _work_. + + + +### Run the experiment + +An experiment consists of a series of sequential runs, with each run continuing from where the previous run ended. +_payu_ supports automatically running a fixed number of runs using the `-n` option: + + payu run -n
+ + +## Monitor the experiment + + +Alternatively, you can also use the PBS `job-ID` to monitor the job using the PBS commands available from NCI. + +To print out information on the status of a specific job, you can execute the following command: +``` +qstat
+This can be changed by altering the `jobname` in the [PBS resources section](#modify-pbs-resources) of the `config.yaml` file. + +_S_ indicates the status of your run, where: + +- _Q_ → Job waiting in the queue to start +- _R_ → Job running +- _E_ → Job ending +- _H_ → Job on hold + +If there are no jobs listed with your `jobname` (or if no job is listed), your run either successfully completed or was terminated due to an error.
+For more information, check [NCI documentation](https://opus.nci.org.au/display/Help/FAQ+1%3A+Why+My+Jobs+are+NOT+Running). + + +_payu_ provides the [`payu status`](https://payu.readthedocs.io/en/stable/usage.html#monitoring-payu-jobs) command for monitoring jobs. This command can return the scheduler job ID and the stage the _payu_ run is currently at. When the job is complete, it displays the exit statuses from the model and overall _payu_ run, and points to the PBS log files. + +!!! note + `payu status` is available in _payu_ versions `1.2.0` and above. This command does not yet support monitoring post-processing jobs from the configuration (e.g., `payu collate` and `payu sync`). + +??? exemple "Example outputs from `payu status`" + + Example output from `payu status` for a running simulation: + + ``` + ======================================== + Run: 8 + Job ID: running_example.gadi-pbs + Run ID: xxxx + Stage: model-run + Current Expt Time: 1950-10-01T00:00:00 + Exit Status: 0 (Success) + Model Exit Code: 0 (Success) + Output Log: /home/189/USER/expt.o100 + Error Log: /home/189/USER/expt.3100 + Job File: /scratch/\$PROJECT/USER/archive/expt-branch—6dhash/payu_jobs/8/run/running_example.gadi-pbs.json + ======================================== + ``` + + Example output from `payu status` for an archived simulation: + + + ``` + ======================================== + Run: 8 + Job ID: archive_example.gadi-pbs + Run ID: xxxx + Stage: archive + Total Queue Time: 0h 1m 7s + Model Finish Time: 1950-10-01T00:00:00 + Exit Status: 0 (Success) + Model Exit Code: 0 (Success) + Output Log: /home/189/USER/expt.o100 + Error Log: /home/189/USER/expt.3100 + Job File: /scratch/\$PROJECT/USER/archive/expt-branch—6dhash/payu_jobs/8/run/archive_example.gadi-pbs.json + ======================================== + ``` + +To monitor the current queue time of a queued job, use `payu status --update`. + + +??? info "Stop a run" + + ### Stop a run + + If you want to manually terminate a run, you can do so by executing: + ``` + qdel
+ This will prevent _payu_ from submitting another job after the current one completes. + + + +## Edit a _payu_ configuration + + +The modifications discussed in this section can change how the model and its components are configured, or the way the model is run by _payu_. + +The `config.yaml` file located in the [_control_ directory](#control-directory) is the _payu_ configuration file, which controls the general model configuration. It contains several parts, some of which are more likely to need modification, and others which are rarely changed without having a deep understanding of how the model is configured. + +To find out more about configuration settings for the `config.yaml` file, refer to [how to configure your experiment with payu](https://payu.readthedocs.io/en/stable/config.html). + + +### Change run length + +Adjusting the duration of the model run is one of the most common change to apply. However, models follow different ways to adapt the duration of the run. Please refer to the [Run a Model][Run a Model] page of the model of your choice for information
+ + + +### Start the run from a specific restart file {: id='specific-restart'} + +To configure the experiment to start from specific restart files, add a [`restart:` entry](https://payu.readthedocs.io/en/stable/config.html#miscellaneous) to the `config.yaml` file, specifying the path to a folder containing existing restart files. +Or to do this automatically when setting up an experiment using `payu clone` interactive, give the restart path when prompted: `Do you want to specify a custom restart path?`. + +!!! warning + In some cases, if the supplied restart file is not fully compatible with the model configuration, experiments using a custom restart file may require additional manual adjustments to run correctly. + +!!! warning + The restart option used here will only be applied if there is no restart directory in archive, and so does not have to be removed for subsequent submissions. See [Payu docs](https://payu.readthedocs.io/en/stable/config.html#miscellaneous) for further details. + + + +### Specify the compute project and storage location {: id='compute-storage-choice'} + +If you want to submit an experiment or part of an experiment using a different project for the compute resources or a non-default location for the archive directory, you will need to modify the following entries in `config.yaml`: + +```yaml +# If submitting to a different project to your default, uncomment line below +# and replace PROJECT_CODE with appropriate code. This may require setting shortpath +# project: PROJECT_CODE + +# Force payu to always find, and save, files in this scratch project directory +# shortpath: /scratch/PROJECT_CODE +``` + +For example, to run under the {{WG_project}}, uncomment the line beginning with `# project` by deleting the `#` symbol and replace `PROJECT_CODE` with `{{WG_project_code}}`: + +```yaml +project: {{WG_project_code}} +``` + +For model configurations and output to be saved to a `/scratch` storage location other than `project` (or your default if `project` is not set) then also set `shortpath` to the desired path. + +!!! warning + If changing the project providing the compute resources during an experiment, set the `shortpath` field so that it's the same for all runs of an experiment. + Doing this will make sure the same `/scratch` location is used for the _laboratory_, regardless of which project is used to run the experiment. + + + +### Modify PBS resources + +If the model has been altered and needs more time or memory to complete, or needs to be submitted under a different NCI project, you will need to modify the following options in the `config.yaml`: + +```yaml +queue: normal +walltime: 3:00:00 +mem: 1000GB +jobname: 1deg_jra55_ryf +``` + +These lines can be edited to change the [PBS directives](https://opus.nci.org.au/display/Help/PBS+Directives+Explained) for the [PBS job][PBS job]. + + + +### Syncing output data to long-term storage + +The _laboratory_ directory is typically under the `/scratch` storage on _Gadi_, where [files are regularly deleted once they have been unaccessed for a period of time](https://opus.nci.org.au/pages/viewpage.action?pageId=156434436). For this reason climate model outputs need to be moved to a location with longer term storage.
+On _Gadi_, this is typically in a folder under a project code on `/g/data`. + +_Payu_ has built-in support to sync outputs, restarts and a copy of the _control_ directory git history to another location.
+This feature is controlled by the following section in the `config.yaml` file: +```yaml +# Sync options for automatically copying data from ephemeral scratch space to +# longer term storage +sync: + enable: False # set base_path below and change to true + restart: True + base_path: none # Final sync location will be
+ _payu_ updates the manifest files for every run, and relies on `runlog` to save this information in the _git_ history, so there is a record of all inputs, restarts, and executables used in an experiment. + +#### `userscripts` section {: .no-toc } + +They are used to run scripts or subcommands at various stages of a _payu_ submission: + +- `error` field: script is called if the model does not run correctly and exits with an error. +- `run` field: script is called after each model run successful execution, but prior to archiving the model output. If using `payu -n` for automatic resubmission, it is run for each submission. +- `sync` field: script is called at the start of the sync PBS job. + +For more information about specific `userscripts` fields, check the relevant section of [_payu_ Configuration Settings documentation](https://payu.readthedocs.io/en/stable/config.html#postprocessing). + +#### `postscript` option {: .no-toc } +Postprocessing scripts that run after _payu_ has completed all steps of each run (for example, with `payu run -n 10`, the postscript will run 10 times). Scripts that might alter the output directory, for example, can be run as postscripts. These run in PBS jobs separate from the main model simulation. + +#### Miscellaneous {: .no-toc } + +The following configuration settings should never require changing: + +```yaml +stacksize: unlimited +qsub_flags: -W umask=027 +``` + + + +#### Collate {: .no-toc } + +Rather than outputting a single diagnostic file over the whole model horizontal grid, the ocean component [MOM](/models/model_components/ocean/#modular-ocean-model-mom) typically generates diagnostic outputs as tiles, each of which spans a portion of model grid. + +The `collate` section in the `config.yaml` file controls the process that combines these smaller files into a single outputfile. + +```yaml +# Collation +collate: + exe: mppnccombine.spack + restart: true + mem: 4GB + walltime: 1:00:00 + mpi: false +``` +Restart files are typically tiled in the same way and will also be combined together if the `restart` field is set to`true`. + + + +## Edit a model components' configuration + +To modify the physics used by a model component, the input data or the model variables saved in the output, you will need to modify the model component's configuration files. These are located inside a subfolder of the _control_ directory, named according to the submodel's `name` specified in the `config.yaml` `submodels` section. + \ No newline at end of file diff --git a/docs/includes/terminology.md b/docs/includes/terminology.md new file mode 100644 index 000000000..833720292 --- /dev/null +++ b/docs/includes/terminology.md @@ -0,0 +1,17 @@ + +### _Configuration_ and _experiment_ definitions + +The terms _configuration_ and _experiment_ are not interchangeable although they are closely related. + +- A _configuration_ defines a specific way to run the model it relates to.
+ A configuration is defined by: + + - model version and build (model executable(s)) + - set of input files (ancillaries, forcings, restarts) + - set of physical and modelling options for each model component, such as namelists, configuration files and MPI layout + + Changing any one of these elements creates a new configuration + +- An _experiment_ is a realisation of a configuration: a series of sequential runs that generate model data over a span of model time. + + diff --git a/docs/models/run_a_model/index.md b/docs/models/run_a_model/index.md index 51d84dc0a..944b79ca6 100644 --- a/docs/models/run_a_model/index.md +++ b/docs/models/run_a_model/index.md @@ -22,6 +22,27 @@ If you are unsure which ACCESS model is the best fit for your application, you c Run ACCESS-CM2 + + +
+ 
+
+
+
+ Run ACCESS-rAM3
+
+
+
+
+
+
+
+
diff --git a/docs/models/run_a_model/run_access-esm.md b/docs/models/run_a_model/run_access-esm.md
index 1860f38f8..1018144ec 100644
--- a/docs/models/run_a_model/run_access-esm.md
+++ b/docs/models/run_a_model/run_access-esm.md
@@ -1,6 +1,10 @@
{% set model = "ACCESS-ESM1.5" %}
{% set github_configs = "https://github.com/ACCESS-NRI/access-esm1.5-configs" %}
{% set release_notes = "https://forum.access-hive.org.au/t/access-esm1-5-release-information/2352" %}
+{% set config_example = "release-preindustrial+concentrations" %}
+{% set WG_project = "`lg87` project (ESM Working Group)" %}
+{% set WG_project_code = "lg87" %}
+{% set configs_docs = "https://github.com/ACCESS-NRI/access-esm1.5-configs" %}
[PBS job]: https://opus.nci.org.au/display/Help/4.+PBS+Jobs
[gadi]: https://opus.nci.org.au/display/Help/0.+Welcome+to+Gadi#id-0.WelcometoGadi-Overview
[payu]: https://github.com/payu-org/payu
@@ -27,7 +31,6 @@ All {{model}} configurations are open source, licensed under [CC BY 4.0](https:/
{{ model }} release notes are [available on the ACCESS-Hive Forum]({{release_notes}}) and are updated when new releases are made available.
-
## Prerequisites
- **NCI Account**
+ 
+
+
+
+ Run models using Payu
+
+
@@ -49,13 +70,4 @@ If you are unsure which ACCESS model is the best fit for your application, you c
Run ACCESS-OM3
-
-
-
-
-
-
-
- Run ACCESS-rAM3
-
-
@@ -41,10 +44,13 @@ All {{model}} configurations are open source, licensed under [CC BY 4.0](https:/ - **Join NCI projects**
Join the following projects by requesting membership on their respective NCI project pages: - - [vk83](https://my.nci.org.au/mancini/project/vk83/join) - [ki32](https://my.nci.org.au/mancini/project/ki32/join) - [ki32_mosrs](https://my.nci.org.au/mancini/project/ki32_mosrs/join) - +{% + include-markdown "includes/payu.md" + start="" + end="" +%} !!! tip To request membership for the _ki32_mosrs_ subproject, you need to: @@ -55,654 +61,251 @@ All {{model}} configurations are open source, licensed under [CC BY 4.0](https:/ For more information on joining specific NCI projects, refer to [How to connect to a project](https://opus.nci.org.au/display/Help/How+to+connect+to+a+project). -- **Payu**
- [_Payu_][payu] is a workflow management tool for running numerical models in supercomputing environments, for which there is extensive [documentation](https://payu.readthedocs.io/en/latest/).
- _Payu_ on _Gadi_ is available through a dedicated `conda` environment in the _vk83_ project.
- After joining the _vk83_ project, load the `payu` module: - - module use /g/data/vk83/modules - module load payu +## Terminology - To check that _payu_ is available, run: +??? info "_Configuration_ and _experiment_ definitions" - payu --version - -
-Released configurations are tested and supported by ACCESS-NRI, as an adaptation of those originally developed by [CSIRO](https://www.csiro.au/en/research/environmental-impacts/climate-change/climate-science-centre) and [CLEX CMS](https://github.com/coecms/access-esm). - -For more information on {{ model }} configurations, check [{{model}}][model configurations] page. - -More information about the available experiments and the naming scheme of the branches can also be found in the [{{ model }} configs]({{github_configs}}) GitHub repository. - -The first step is to choose a configuration from those available.
- -For example, if the required configuration is the co2 concentration driven pre-industrial configuration, then the branch to select is [`release-preindustrial+concentrations`](https://forum.access-hive.org.au/t/access-esm1-5-release-information/2352). - -To clone this branch to a location on _Gadi_, run: - - mkdir -p ~/access-esm1.5 - cd ~/access-esm1.5 - payu clone -b expt -B release-preindustrial+concentrations {{ github_configs }} preindustrial+concentrations - -!!! tip - If you want to restart your experiment from a specific restart point, please refer to [Start the run from a specific restart file](#specific-restart). - -In the example above the `payu clone` command clones the concentration driven pre-industrial configuration (`-B release-preindustrial+concentrations`) -to a new experiment branch (`-b expt`) to a directory named `preindustrial+concentrations`. - -!!! admonition tip - Anyone using a configuration is advised to clone only a single branch (as shown in the example above) and not the entire repository. - -
- For this reason, it is recommended to always set the cloned branch name (`expt` in the example above) to something meaningful for the planned experiment.
- For more information refer to this [payu tutorial](https://forum.access-hive.org.au/t/access-om2-payu-tutorial/1750#select-experiment-12). +{% + include-markdown "includes/payu.md" + start="" + end="" +%} ---------------------------------------------------------------------------------------- -## Run {{ model }} configuration -If you want to modify your configuration, refer to [Edit {{ model }} configuration](#edit-{{ model.lower() }}-configuration). - -{{ model }} configurations run on [_Gadi_][gadi] through a [PBS job][PBS job] submission managed by [_payu_][payu]. - -The general layout of a _payu_-supported model run consists of two main directories: - -- The _control_ directory contains the model configuration and serves as the execution directory for running the model (in this example, the cloned directory `~/access-esm1.5/preindustrial+concentrations`). -- The _laboratory_ directory, where all the model components reside. For {{ model }}, it is typically `/scratch/$PROJECT/$USER/access-esm`. - -This separates the small text configuration files from the larger binary outputs and inputs. In this way, the _control_ directory can be in the `$HOME` directory (as it is the only filesystem actively backed-up on _Gadi_). The quotas for `$HOME` are low and strict, which limits what can be stored there, so it is not suitable for larger files. - -The _laboratory_ directory is a shared space for all _payu_ experiments using the same model.
-Inside the _laboratory_ directory there are two subdirectories: - -- `work` → a directory where _payu_ automatically creates a temporary subdirectory while the model is run. The temporary subdirectory gets created as part of a run and then removed after the run succeeds. -- `archive` → the directory where the output is stored following each successful run. - -Within each of the above directories _payu_ automatically creates subdirectories uniquely named according to the experiment being run.
-_Payu_ also creates symbolic links in the _control_ directory pointing to the `archive` and `work` directories. - -This design allows multiple self-resubmitting experiments that share common executables and input data to be run simultaneously. - -!!! admonition warning - Files on the `/scratch` drive, such as the _laboratory_ directory, might get deleted if not accessed for several days and the `/scratch` drive is limited in space. For these reasons, all model runs which are to be kept should be moved to `/g/data/` by enabling the _sync_ step in _payu_. To know more refer to [Syncing output data](#syncing-output-data). +## Get {{ model }} configuration -### Run configuration +Released configurations are tested and supported by ACCESS-NRI, and build upon those originally developed by [CSIRO](https://www.csiro.au/en/research/environmental-impacts/climate-change/climate-science-centre) and [CLEX CMS](https://github.com/coecms/access-esm).
-To run {{ model }} configuration execute the following command from within the *control* directory: +All released {{ model }} configurations are available from the [{{ model }} configs]({{github_configs}}) GitHub repository: `{{github_configs}}`.
- payu run +Supported configurations: -This will submit a single job to the queue with a run length given by [`runtime`](#runtime) in the `config.yaml` file.
-To extend the run for longer than the run length, refer to [Run an experiment](#run-an-experiment) +| Configuration | Branch name | +|---------------|-------------| +| CMIP6 Concentration-driven pre-industrial | release-preindustrial+concentrations | +| CMIP6 Concentration-driven historical | release-historical+concentrations | +!!! note -
-To print out information on the status of a specific job, you can execute the following command: -``` -qstat
-This can be overwritten by altering the `jobname` in the [PBS resources section](#modify-pbs-resources) of the `config.yaml` file. - -_S_ indicates the status of your run, where: - -- _Q_ → Job waiting in the queue to start -- _R_ → Job running -- _E_ → Job ending -- _H_ → Job on hold - -If there are no jobs listed with your `jobname` (or if no job is listed), your run either successfully completed or was terminated due to an error.
-For more information, check [NCI documentation](https://opus.nci.org.au/display/Help/FAQ+1%3A+Why+My+Jobs+are+NOT+Running). - -### Stop a run - -If you want to manually terminate a run, you can do so by executing: -``` -qdel
- This will prevent _payu_ from submitting another job. - -### Error and output log files - -#### PBS output files {: .no-toc } -When the model completes a run, PBS writes the standard output and error streams to two files inside the _control_ directory: `
-To move these files to the `archive` directory, use the following command: -``` -payu sweep -``` - -#### Model log files {: .no-toc } - -While the model is running, _payu_ saves the model standard output and error streams in the `access.out` and `access.err` files inside the _control_ directory, respectively.
-You can examine the contents of these files to check on the status of a run as it progresses (or after a failed run has completed). - -!!! warning - At the end of a successful run these log files are archived to the `archive` directory and will no longer be found in the _control_ directory. If they remain in the _control_ directory after the PBS job for a run has completed it means the run has failed. - -### Model Live Diagnostics - -ACCESS-NRI developed the [Model Live Diagnostics](/model_evaluation/evaluation_on_gadi/model_live_diagnostics) framework to check, monitor, visualise, and evaluate model behaviour and progress of ACCESS models currently running on _Gadi_.
-For a complete documentation on how to use this framework, check the [Model Diagnostics documentation](https://med-live-diagnostics.readthedocs.io/en/latest/index.html). - -### Trouble-shooting - -If _payu_ doesn't run correctly for some reason, a good first step is to run the following command from within the _control_ directory: - - payu setup - -This command will: - - - create the _laboratory_ and `work` directories based on the experiment configuration - - generate manifests - - report useful information to the user, such as the location of the _laboratory_ where the `work` and `archive` directories are located - - -
-Symbolic links to these directories are also provided in the _control_ directory for convenience. +{% include-markdown "includes/payu.md" + start="" + end="" +%} -If a model run is unsuccessful, the `work` directory is left untouched to facilitate the identification of the cause of the model failure. +??? tip "Identifying `run_length` for your experiment" -Outputs and restarts are stored in subfolders within the `archive` directory, subdivided for each run of the model.
-Output and restart folders are called `outputXXX` and `restartXXX`, respectively, where _XXX_ is the run number starting from `000`. + In {{model}}, `run_length` is controlled by the `runtime` setting in the `config.yaml` file in the configuration. For example, a 1-year `run_length` is given by: -Model components are separated into subdirectories within the output and restart directories. - -
-The modifications discussed in this section can change the way {{ model }} is run by _payu_, or how its specific [model components] are configured and coupled together. - -The `config.yaml` file located in the _control_ directory is the _Master Configuration_ file, which controls the general model configuration. It contains several parts, some of which it is more likely will need modification, and others which are rarely changed without having a deep understanding of how the model is configured. - -To find out more about configuration settings for the `config.yaml` file, refer to [how to configure your experiment with payu](https://payu.readthedocs.io/en/latest/config.html). - -### Change run length {: #runtime .no-toc } - -One of the most common changes is to adjust the duration of the model run.
{{model}} simulations are split into smaller _run lengths_, each with the duration specified by the `runtime` settings in the `config.yaml` file: - -The length of an {{model}} run is controlled by the `runtime` settings in the `config.yaml` file: - -```yml - runtime: - years: 1 - months: 0 - days: 0 -``` -At the end of each run length, each model component saves its state into a _restart file_, allowing the simulation to be continued in subsequent runs. +{% include-markdown "includes/payu.md" + start=" + end="" +%} -!!! warning - The _run length_ (controlled by `runtime`) should be left at 1 year for {{model}} experiments in order to avoid errors. Shorter simulations can be useful when setting up and debugging new experiments, however they require additional configuration changes. See the section [Run for less than one year](#shorter-runs) for details. +??? info "Change run length" -To run the model for longer than the default run length, conduct multiple runs, see -[Run an experiment](#run-an-experiment). + ### Change run length -#### Understand _runtime_, _runspersub_, and _-n_ parameters {: id="multiple-runs"} - -It is possible to have more than one model run per queue submit. With the correct use of [`runtime`](#runtime), `runspersub`, `-n` and `walltime` parameters, you can have full control of your experiment.
- -- `runtime` defines the _run length_. -- `runspersub` defines the maximum number of runs for every [PBS job] submission. -- `-n` sets the number of runs to be performed. -- `walltime` defines the maximum time of every [PBS job] submission. - -Now some practical examples: - -- **Run 20 years of simulation with resubmission every 5 years**
- To have a _total experiment length_ of 20 years with a 5-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year`, set `runspersub` to `5` and `walltime` to `10:00:00`. Then, run the configuration with `-n` set to `20`: - ``` - payu run-f -n 20 - ``` - This will submit subsequent jobs for the following years: 1 to 5, 6 to 10, 11 to 15, and 16 to 20, which is a total of 4 PBS jobs. - -- **Run 7 years of simulation with resubmission every 3 years**
- To have a _total experiment length_ of 7 years with a 3-year resubmission cycle, leave [`runtime`](#runtime) as the default value of `1 year`, set `runspersub` to `3` and `walltime` to `6:00:00`. Then, run the configuration with `-n` set to `7`: - ``` - payu run -f -n 7 + One of the most common changes is to adjust the duration of the model run.
{{model}} simulations are split into smaller _run lengths_, each with the duration specified by the `runtime` settings in the `config.yaml` file: + + The length of an {{model}} run is controlled by the `runtime` settings in the `config.yaml` file: + + ```yml + runtime: + years: 1 + months: 0 + days: 0 ``` - This will submit subsequent jobs for the following years: 1 to 3, 4 to 6, and 7, which is a total of 3 PBS jobs. -!!! tip - The `walltime` must be set to be long enough that the PBS job can complete. The model usually runs a single year in 90 minutes or less, but the `walltime` for a single model run is set to `2:30:00` out of an abundance of caution to make sure the model has time to run when there are occasional slower runs for unpredictable reasons. When setting `runspersub > 1` the `walltime` doesn't need to be a simple multiple of `2:30:00` because it is highly unlikely that there will be multiple anomalously slow runs per submit. - -#### Run for less than one year {: id="shorter-runs"} -When debugging changes to a model, it is common to reduce the run length to minimise resource consumption and return faster feedback on changes. In order to run the model for a single month, the `runtime` can be changed to - -```yml - runtime: - years: 0 - months: 1 - days: 0 -``` - -With the default configuration settings, the sea ice component of {{ model }} will produce restart files only at the end of each year. If valid restart files are required when running shorter simulations, the sea ice model configuration should be modified so that restart files are produced at monthly frequencies. To do this, change the `dumpfreq = 'y'` setting to `dumpfreq = 'm'` in the `cice_in.nml` configuration file located in the `ice` subdirectory of the _control_ directory. - -### Start the run from a specific restart file {: id='specific-restart'} - -To start the run with the initial conditions coming from a specific restart file, you can add the `--restart` option when obtaining the model configuration through the `payu clone ...` command. - -For example, to get the `preindustrial+concentrations` configuration and set its initial condition to the `/g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/restart` restart file, run: - -``` -payu clone -b expt -B release-preindustrial+concentrations https://github.com/ACCESS-NRI/access-esm1.5-configs preindustrial+concentrations --restart /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/restart -``` - -!!! warning - In some cases, if the supplied restart file is not fully compatible with the model configuration, experiments using a custom restart file may require additional manual adjustments to run correctly. - -### Modify PBS resources - -If the model has been altered and needs more time to complete, more memory, or needs to be submitted under a different NCI project, you will need to modify the following section in the `config.yaml`: - - -```yaml -# PBS configuration - -# If submitting to a different project to your default, uncomment line below -# and replace PROJECT_CODE with appropriate code. This may require setting shortpath -# project: PROJECT_CODE - -# Force payu to always find, and save, files in this scratch project directory -# shortpath: /scratch/PROJECT_CODE - -# Note: if laboratory is relative path, it is relative to shortpath/$USER -laboratory: access-esm - -jobname: pre-industrial -queue: normal -walltime: 2:30:00 -``` - -These lines can be edited to change the [PBS directives](https://opus.nci.org.au/display/Help/PBS+Directives+Explained) for the [PBS job][PBS job]. - -By default the model will be submitted to the PBS queue using your default project. To run {{ model }} using the resources of a specific project, for example the `lg87` project (ESM Working Group), uncomment the line beginning with `# project` by deleting the `#` symbol and replace `PROJECT_CODE` with `lg87`: - -```yaml -project: lg87 -``` - -!!! warning - If more than one project is used to run an {{ model }} configuration the `shortpath` option also needs to be uncommented and the path to the desired `/scratch/PROJECT_CODE` directory added.
- This ensures the same `/scratch` location is used for the _laboratory_, regardless of which project is used to run the experiment. -
- To run {{ model }}, you need to be a member of a project with allocated _SU_. For more information, check [how to join relevant NCI projects](/getting_started/set_up_nci_account#join-relevant-nci-projects). - -### Syncing output data - -The _laboratory_ directory is typically under the `/scratch` storage on _Gadi_, where [files are regularly deleted once they have been unaccessed for a period of time](https://opus.nci.org.au/pages/viewpage.action?pageId=156434436). For this reason climate model outputs need to be moved to a location with longer term storage.
-On _Gadi_, this is typically in a folder under a project code on `/g/data`. - -_Payu_ has built-in support to sync outputs, restarts and a copy of the _control_ directory git history to another location.
-This feature is controlled by the following section in the `config.yaml` file: -```yaml -# Sync options for automatically copying data from ephemeral scratch space to -# longer term storage -sync: - enable: False # set path below and change to true - path: null # Set to location on /g/data or a remote server and path (rsync syntax) -``` -To enable syncing, change `enable` to `True`, and set `path` to a location on `/g/data`, where _payu_ will copy output and restart folders. - -!!! Warning - The {{model}} configurations include a [postprocessing script](#postscripts) which converts atmospheric outputs to NetCDF format. This script runs in a separate PBS job and prevents the output and restart files of the most recent run from being automatically synced.
- After a series of runs and the final post-processing is completed, manually execute `payu sync` in the _control_ directory to sync the final output and restart files. - -### Saving model restarts - -{{ model }} outputs restart files after every run to allow for subsequent runs to start from a previously saved model state.
-Restart files can occupy a significant amount of disk space, and keeping a lot of them is often not necessary. - -The `restart_freq` field in the `config.yaml` file specifies a strategy for retaining restart files.
-This can either be a number (in which case every _nth_ restart file is retained), or one of the following pandas-style datetime frequencies: - -- `YS` → start of the year -- `MS` → start of the month -- `D` → day -- `H` → hour -- `T` → minute -- `S` → second - -For example, to preserve the ability to restart {{ model }} every 50 model-years, set: -```yaml -restart_freq: '50YS' -``` - -The most recent sequential restarts are retained, and only deleted after a permanently archived restart file has been produced. - -For more information, check [_payu_ Configuration Settings documentation](https://payu.readthedocs.io/en/latest/config.html#model). - -### Other configuration options - -!!! warning - The following sections in the `config.yaml` file control configuration options that are rarely modified, and often require a deeper understanding of how {{ model }} is structured to be safely changed. - -#### Model configuration {: .no-toc } - -This section tells _payu_ which driver to use for the main model (`access` refers to {{ model }}). - -```yaml -model: access -``` - - -#### Submodels {: .no-toc } - -{{ model }} is a coupled model deploying multiple submodels (i.e. [model components]). - -This section specifies the submodels and configuration options required to execute {{ model }} correctly. - -Each submodel contains additional configuration options that are read in when the submodel is running. These options are specified in the subfolder of the _control_ directory, whose name matches the submodel's `name` (e.g., configuration options for the `atmosphere` submodel are in the `~/access-esm/preindustrial+concentrations/atmosphere` directory). - - -??? code "Expand to show the full `submodels` section" - - ```yaml - submodels: - - name: atmosphere - model: um - ncpus: 240 - exe: um_hg3.exe - input: - # Aerosols - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/aerosol/global.N96/2020.05.19/OCFF_1850_ESM1.anc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/aerosol/global.N96/2020.05.19/BC_hi_1850_ESM1.anc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/aerosol/global.N96/2020.05.19/scycl_1850_ESM1_v4.anc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/aerosol/global.N96/2020.05.19/Bio_1850_ESM1.anc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/aerosol/global.N96/2020.05.19/biogenic_351sm.N96L38 - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/aerosol/global.N96/2020.05.19/sulpc_oxidants_N96_L38 - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/aerosol/global.N96/2020.05.19/DMS_conc.N96 - # Forcing - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/forcing/global.N96/2020.05.19/ozone_1850_ESM1.anc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/forcing/resolution_independent/2020.05.19/volcts_18502000ave.dat - # Land - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/pre-industrial/atmosphere/land/biogeochemistry/global.N96/2020.05.19/Ndep_1850_ESM1.anc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/land/soiltype/global.N96/2020.05.19/qrparm.soil_igbp_vg - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/land/vegetation/global.N96/2020.05.19/cable_vegfunc_N96.anc - - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/land/biogeochemistry/resolution_independent/2020.05.19/modis_phenology_csiro.txt - - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/land/biogeochemistry/resolution_independent/2020.05.19/pftlookup_csiro_v16_17tiles_wtlnds.csv - - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/land/biogeophysics/resolution_independent/2020.05.19/def_soil_params.txt - - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/land/biogeophysics/resolution_independent/2020.05.19/def_veg_params.txt - # Spectral - - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/spectral/resolution_independent/2020.05.19/spec3a_sw_hadgem1_6on - - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/spectral/resolution_independent/2020.05.19/spec3a_lw_hadgem1_6on - # Grids - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/atmosphere/grids/global.N96/2020.05.19/qrparm.mask - - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/grids/resolution_independent/2020.05.19/vertlevs_G3 - # STASH - - /g/data/vk83/configurations/inputs/access-esm1p5/share/atmosphere/stash/2024.11.01 - - - name: ocean - model: mom - ncpus: 180 - exe: fms_ACCESS-CM.x - input: - # Biogeochemistry - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/biogeochemistry/global.1deg/2020.05.19/dust.nc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/biogeochemistry/global.1deg/2020.05.19/ocmip2_press_monthly_om1p5_bc.nc - - /g/data/vk83/configurations/inputs/access-esm1p5/share/ocean/biogeochemistry/global.1deg/2024.07.12/bgc_param.nc - # Tides - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/tides/global.1deg/2020.05.19/roughness_amp.nc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/tides/global.1deg/2020.05.19/tideamp.nc - # Shortwave - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/shortwave_penetration/global.1deg/2020.05.19/ssw_atten_depth.nc - # Grids - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/grids/mosaic/global.1deg/2020.05.19/grid_spec.nc - # Basin mask - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ocean/basins/global.1deg/2020.05.19/basin_mask.nc - - - name: ice - model: cice - ncpus: 12 - exe: cice_access_360x300_12x1_12p.exe - input: - # Grids - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ice/grids/global.1deg/2020.05.19/kmt.nc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ice/grids/global.1deg/2020.05.19/grid.nc - # Climatology - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/ice/climatology/global.1deg/2020.05.19/monthly_sstsss.nc - - - name: coupler - model: oasis - ncpus: 0 - input: - # Grids - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/grids/global.oi_1deg.a_N96/2020.05.19/grids.nc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/grids/global.oi_1deg.a_N96/2020.05.19/areas.nc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/grids/global.oi_1deg.a_N96/2020.05.19/masks.nc - # Remapping weights - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_cice_to_um1t_CONSERV_FRACNNEI.nc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_um1u_to_cice_CONSERV_FRACNNEI.nc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_um1t_to_cice_CONSERV_DESTAREA.nc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_cice_to_um1u_CONSERV_FRACNNEI.nc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_um1v_to_cice_CONSERV_FRACNNEI.nc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_um1t_to_cice_CONSERV_FRACNNEI.nc - - /g/data/vk83/configurations/inputs/access-esm1p5/modern/share/coupler/remapping_weights/global.oi_1deg.a_N96/2020.05.19/rmp_cice_to_um1v_CONSERV_FRACNNEI.nc - + At the end of each run length, each model component saves its state into a _restart file_, allowing the simulation to be continued in subsequent runs. + + !!! warning + The _run length_ (controlled by `runtime`) should be left at 1 year for {{model}} experiments in production in order to avoid errors. However, when testing and debugging new experiments, shorter simulations can be useful. It is possible to set _run length_ to less than a year but additional configuration changes are required. See the section [Run for less than one year](#shorter-runs) for details. + + To run the model for longer than the default run length, conduct multiple runs as explained in [Run an experiment](#run-an-experiment). _payu_ has options to manage the length of a simulation for each `payu run` command: _runtime_, _runspersub_ and _-n_. They allow you to have complete control on the length of your experiments. + + #### Understand _runtime_, _runspersub_, and _-n_ parameters {: id="multiple-runs"} + + The `runtime`, `runspersub`, `-n` and `walltime` parameters control various aspects of the simulation related to the length of the simulation:
+ + - `runtime` defines the _run length_. + - `runspersub` defines the maximum number of runs for every [PBS job] submission. + - `-n` sets the total number of runs to be performed. + - `walltime` defines the maximum time of every [PBS job] submission. + + By using these parameters correctly, you can fully control the length of your simulation.
+ Now some practical examples: + + - **Run 20 years of simulation with resubmission every 5 years**
+ To have a _total experiment length_ of 20 years with a 5-year resubmission cycle, leave `runtime` in `config.yaml` as the default value of `1 year`, set `runspersub` to `5` and `walltime` to `10:00:00`. Then, run the configuration with `-n` set to `20`: + ``` + payu run -f -n 20 + ``` + This will submit subsequent jobs for the following years: 1 to 5, 6 to 10, 11 to 15, and 16 to 20, which is a total of 4 PBS jobs. + + - **Run 7 years of simulation with resubmission every 3 years**
+ To have a _total experiment length_ of 7 years with a 3-year resubmission cycle, leave `runtime` as the default value of `1 year`, set `runspersub` to `3` and `walltime` to `6:00:00`. Then, run the configuration with `-n` set to `7`: + ``` + payu run -f -n 7 + ``` + This will submit subsequent jobs for the following years: 1 to 3, 4 to 6, and 7, which is a total of 3 PBS jobs. + !!! tip + The `walltime` must be set to be long enough that the PBS job can complete. The model usually runs a single year in 90 minutes or less, but the `walltime` for a single model run is set to `2:30:00` out of an abundance of caution to make sure the model has time to run when there are occasional slower runs for unpredictable reasons. When setting `runspersub > 1` the `walltime` doesn't need to be a simple multiple of `2:30:00` because it is highly unlikely that there will be multiple anomalously slow runs per submit. + + #### Run for less than one year {: id="shorter-runs"} + When debugging changes to a model, it is common to reduce the run length to minimise resource consumption and return faster feedback on changes. In order to run the model for a single month, the `runtime` can be changed to + + ```yml + runtime: + years: 0 + months: 1 + days: 0 ``` + + With the default configuration settings, the sea ice component of {{ model }} will produce restart files only at the end of each year. If you may want to continue your short simulation over a longer period, you will need valid restart files created at the end of each run. For this, the sea ice model configuration should be modified so that restart files are produced at monthly frequencies, to match _runtime_. To do this, change the `dumpfreq = 'y'` setting to `dumpfreq = 'm'` in the `cice_in.nml` configuration file located in the `ice` subdirectory of the _control_ directory. -#### Collate {: .no-toc } - -Rather than outputting a single diagnostic file over the whole model horizontal grid, the ocean component [MOM](/models/model_components/ocean/#modular-ocean-model-mom) typically generates diagnostic outputs as tiles, each of which spans a portion of model grid. - -The `collate` section in the `config.yaml` file controls the process that combines these smaller files into a single output file. - -```yaml -# Collation -collate: - exe: mppnccombine.spack - restart: true - mem: 4GB - walltime: 1:00:00 - mpi: false -``` -Restart files are typically tiled in the same way and will also be combined together if the `restart` field is set to `true`. - -#### Runlog {: .no-toc } - -```yaml -runlog: true -``` -When running a new configuration, _payu_ automatically commits changes with `git` if `runlog` is set to `true`. - -!!! warning - This should not be changed as it is an essential part of the provenance of an experiment.
- _payu_ updates the manifest files for every run, and relies on `runlog` to save this information in the `git` history, so there is a record of all inputs, restarts, and executables used in an experiment. - - -#### Userscripts {: .no-toc } -```yaml -userscripts: - # Apply land use changes after each run - run: ./scripts/update_landuse_driver.sh -``` - -Run scripts or subcommands at various stages of a _payu_ submission. The above example comes from the `release-historical+concentrations` configuration, where the ```update_landuse_driver.sh``` is used to apply historical land use changes at the end of each run. - -For more information about specific `userscripts` fields, check the relevant section of [_payu_ Configuration Settings documentation](https://payu.readthedocs.io/en/latest/config.html#postprocessing). - +??? info "Specify the restart file" -#### Postscripts {: .no-toc } -Postprocessing scripts that run after _payu_ has completed all steps of each run (for example, with `payu run -n 10`, the postscript will run 10 times). Scripts that might alter the output directory, for example, can be run as postscripts. These run in PBS jobs separate from the main model simulation. + {% include-markdown "includes/payu.md" + start="" + end="" + %} -```yaml -postscript: -v PAYU_CURRENT_OUTPUT_DIR,PROJECT -lstorage=${PBS_NCI_STORAGE} ./scripts/NetCDF-conversion/UM_conversion_job.sh -``` +??? info "Specify the compute project and storage location" -All {{ model }} configurations include the NetCDF conversion postscript mentioned above. This script converts the [UM](/models/model_components/atmosphere#unified-model-um)'s fields file format output to NetCDF in order to facilitate analysis and reduce storage requirements. By default, the conversion script will delete the fields files upon successful completion, leaving only the NetCDF output. This automatic deletion can be disabled by commenting out the `--delete-ff` command line flag from the conversion job submission script located in the _control_ directory under `scripts/NetCDF-conversion/UM_conversion_job.sh`.
-That means changing + {% include-markdown "includes/payu.md" + start="" + end="" + %} -```bash -esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR --delete-ff -``` +??? info "Modify PBS resources" -to + {% include-markdown "includes/payu.md" + start="" + end="" + %} -```bash -esm1p5_convert_nc $PAYU_CURRENT_OUTPUT_DIR # --delete-ff -``` +??? info "Syncing output data" -#### Miscellaneous {: .no-toc } + {% include-markdown "includes/payu.md" + start="" + end="" + %} -The following configuration settings should never require changing: +??? info "Pruning model restarts" -```yaml -stacksize: unlimited -qsub_flags: -W umask=027 -``` + {% include-markdown "includes/payu.md" + start="" + end="" + %} -### Edit a single {{ model }} component configuration +??? info "_payu_ advance options" -Each of [{{ model }} components][model components] contains additional configuration options that are read in when the model component is running.
These options are typically useful to modify the physics used in the model, the input data, or the model variables saved in the output files. + {% include-markdown "includes/payu.md" + start="" + end="" + %} -These configuration options are specified in files located inside a subfolder of the _control_ directory, named according to the submodel's `name` specified in the `config.yaml` `submodels` section (e.g., configuration options for the _ocean_ component are in the `~/access-esm/preindustrial+concentrations/ocean` directory).
-To modify these options please refer to the User Guide of the respective model component. +??? info "Collate of ocean output files" -### Create a custom {{ model }} build -All the executables needed to run {{ model }} are pre-built into independent configurations using _Spack_.
-To customise {{ model }}'s build (for example to run {{ model }} with changes in the source code of one of its component), refer to [Modify and build an ACCESS model's source code](/models/build_a_model/build_source_code#{{model|lower}}). + {% include-markdown "includes/payu.md" + start="" + end="" + %} -### Controlling model output -Selecting the variables to save from a simulation can be a balance between enabling future analysis and minimising storage requirements. The choice and frequency of variables saved by each model can be configured from within each submodel's _control_ directory. +{% include-markdown "includes/payu.md" + start="" + end="" +%} -Each submodel's _control_ directory contains _detailed_ and _standard_ presets for controlling the output, located in the `diagnostic_profiles` subdirectories (e.g. `~/access-esm/preindustrial+concentrations/ice/diagnostic_profiles` for the sea ice submodel). The _detailed_ profiles request a large number of variables at higher frequencies, while the _standard_ profiles restrict the output to variables more regularly used across the community. Details on the variables saved by each preset are available in [this Hive Forum topic](https://forum.access-hive.org.au/t/preset-output-profiles-for-esm1-5/3629). +??? info "Create a custom {{ model }} build" -Selecting a preset output profile to use in a simulation can be done by pointing the following symbolic links to the desired profile: + ### Create a custom {{ model }} build + All the executables needed to run {{ model }} are pre-built into independent configurations using _Spack_.
+ To customise {{ model }}'s build (for example to run {{ model }} with changes in the source code of one of its component), refer to [Modify and build an ACCESS model's source code](/models/build_a_model/build_source_code#{{model|lower}}). - * `STASHC` in the atmosphere _control_ directory. - * `diag_table` in the ocean _control_ directory. - * `ice_history.nml` in the ice _control_ directory. +??? info "Controlling the diagnostics output by the model" -For example, to select the _detailed_ output profile for the atmosphere: -