Skip to content

Commit 3e369ec

Browse files
ccarougeccarouge
committed
Corrections from read through
1 parent c24f202 commit 3e369ec

2 files changed

Lines changed: 123 additions & 109 deletions

File tree

‎docs/includes/payu.md‎

Lines changed: 95 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -48,27 +48,34 @@ A representation of the data organisation for _payu_ is given in the following d
4848
As shown in the diagram, the general layout of a _payu_-supported model run consists of two main directories:
4949

5050
- The _control_ directory contains the model configuration and is the directory from which the model run is started.
51-
This directory contains information to manage the simulation and the scientific options that define the algorithms used in the model component or the diagnostics saved by the model component. If a model has only one model component, the files with the scientific options are located directly in the _control_ directory. If the model has several model components, the files are locate inside subfolders of the _control_ directory named according to the submodel's `name` specified in the `config.yaml` [`submodels` section](#submodels). To modify these options please refer to the configurations documentation of the respective model component, found on the [Run a Model][Run a Model] page for your chosen model.
52-
- The _laboratory_ directory contains all data from _payu_ experiments of the same model. By default, it is `/scratch/$PROJECT/$USER/<model_name>`. `$PROJECT` and `$USER` are environment variables on _Gadi_ that points to your [default project](/getting_started/set_up_nci_account/#change-default-project-on-gadi) and your username respectively.
53-
See the section on [modifying the PBS resources](#modify-pbs-resources) to learn how to change the _laboratory_ location.
51+
This directory contains information to manage the simulation and the scientific options that define the algorithms used in the model component or the diagnostics saved by the model component. The simulation is orchestrated from a main `config.yaml` file contained in this directory. The files specific to each model component are either located directly in the _control_ directory if the model has one model component, or in subdirectories if the model has several model components. The `submodels` section of the `config.yaml` file specifies the name of the submodels and of the subdirectories containing the pertinent files. To modify these options please refer to the configurations documentation of the respective model component.
52+
- The _laboratory_ directory contains all data from _payu_ experiments of the same model. By default, it is `/scratch/$PROJECT/$USER/<model_name>`. `$PROJECT` and `$USER` are environment variables on _Gadi_ that points to your [default project](/getting_started/set_up_nci_account/#change-default-project-on-gadi) and your username respectively. This location can be changed.
53+
<!--See the section on [modifying the PBS resources](#modify-pbs-resources) to learn how to change the _laboratory_ location. (Should the include be cut to allow adding this sentence and link?)-->
5454

55-
On _Gadi_, it is good practice to put experiment _control_ directories in your `$HOME` directory as this is the only filesystem that is actively backed-up. There is a 10GB limit for home directories, but the _control_ directory only contains text files and symlinks, and so uses relatively little space (<1MB). The _laboratory_ directory is on `/scratch` which is optimised for fast reading and writing of large data and where there is adequate space available for large model output.
55+
!!! tip
56+
57+
On _Gadi_, it is good practice to put experiment _control_ directories in your `$HOME` directory as this is the only filesystem that is actively backed-up. There is a 10GB limit for home directories, but the _control_ directory only contains text files and symlinks, and so uses relatively little space (<1MB). The _laboratory_ directory is on `/scratch` which is optimised for fast reading and writing of large data and where there is adequate space available for large model output.
58+
59+
If you decide to locate your _control_ directory under `/g/data`, be aware of [some complications](https://forum.access-hive.org.au/t/changing-project-codes-for-payu-control-directories-under-g-data/6566) linked to that choice.
5660

5761
Inside the _laboratory_ directory, there are two subdirectories of particular interest:
5862

5963
- _work_ &rarr; for temporary storage of files needed by the model while it runs. _payu_ creates and removes directories and files in this directory upon successful completion of runs. It is left untouched in case of error to facilitate the identification of the cause of the model failure
6064
- _archive_ &rarr; for storing the output following each successful run. The output, log and restart files are automatically transferred from _work_ to _archive_ upon successful completion of runs.
6165

62-
Within each of the _work_ and _archive_ directories, _payu_ automatically creates a unique subdirectory for each experiment. Within each experiment sub-directory, the output and restart subfolders are called `outputXXX` and `restartXXX`, respectively, where _XXX_ is the run number starting from `000`. Model components are further separated into subdirectories within the output and restart directories.
63-
6466
The _archive_ and _work_ directories for an experiment are most easily accessed through the symbolic links created in the _control_ directory.
6567

6668
!!! warning
67-
Files on the `/scratch` drive, such as the _laboratory_ directory, might be deleted if not accessed for several days. All experiments which are to be kept should be moved to `/g/data/` by enabling the `sync` step in _payu_. To know more, refer to [Syncing output data](#syncing-output-data-to-long-term-storage).
69+
Files on the `/scratch` drive, such as the _laboratory_ directory, might be deleted if not accessed for several days. All experiments which are to be kept should be moved to `/g/data/` by enabling the `sync` step in _payu_.
70+
71+
##### Output and restart files organisation
72+
73+
Within each of the _work_ and _archive_ directories, _payu_ automatically creates a unique subdirectory for each experiment. Within each experiment subdirectory, the output and restart subfolders are called `outputXXX` and `restartXXX`, respectively, where _XXX_ is the run number starting from `000`. Model components are further separated into subdirectories within the output and restart directories.
74+
75+
##### Error and output log files
6876

69-
### Error and output log files
77+
**PBS output files**
7078

71-
#### PBS output files {: .no-toc }
7279
When the model fails or completes a run, PBS writes the standard output and error streams to two files inside the _control_ directory: `<jobname>.o<job-ID>` and `<jobname>.e<job-ID>`, respectively.
7380

7481
These files usually contain logs about _payu_ tasks, and give an overview of the resources used by the job.<br>
@@ -78,12 +85,13 @@ To move these files to the _archive_ directory, use the following commmand:
7885
payu sweep
7986
```
8087

81-
#### Model log files {: .no-toc }
88+
**Model log files**
8289

8390
While the model is running, the standard output and error streams are saved to file in the _control_ directory. You can examine the contents of these log files to check on the status of a run as it progresses (or after a failed run has completed).
8491

8592
!!! warning
8693
At the end of a successful run, the model log files are archived to the _archive_ directory and will no longer be found in the _control_ directory. If they remain in the _control_ directory after the PBS job for a run has completed, it means the run has failed.
94+
8795
<!--end:about-->
8896

8997
## Prerequisites for _payu_
@@ -112,7 +120,7 @@ After joining the _vk83_ project, load the _payu_ module:
112120
module load payu
113121

114122
To check that _payu_ is available, run:
115-
ø
123+
116124
payu --version
117125
<!--end:access-payu-->
118126

@@ -295,36 +303,37 @@ _payu_ provides the [`payu status`](https://payu.readthedocs.io/en/stable/usage.
295303
Example output from `payu status` for a running simulation:
296304

297305
```
298-
========================================
299-
Run: 8
300-
Job ID: running_example.gadi-pbs
301-
Run ID: xxxx
302-
Stage: model-run
303-
Current Expt Time: 1950-10-01T00:00:00
304-
Exit Status: 0 (Success)
305-
Model Exit Code: 0 (Success)
306-
Output Log: /home/189/USER/expt.o100
307-
Error Log: /home/189/USER/expt.3100
308-
Job File: /scratch/\$PROJECT/USER/archive/expt-branch—6dhash/payu_jobs/8/run/running_example.gadi-pbs.json
309-
========================================
306+
========================================
307+
Run: 8
308+
Job ID: running_example.gadi-pbs
309+
Run ID: xxxx
310+
Stage: model-run
311+
Current Expt Time: 1950-10-01T00:00:00
312+
Exit Status: 0 (Success)
313+
Model Exit Code: 0 (Success)
314+
Output Log: /home/189/USER/expt.o100
315+
Error Log: /home/189/USER/expt.3100
316+
Job File: /scratch/\$PROJECT/USER/archive/expt-branch—6dhash/payu_jobs/8/run/running_example.gadi-pbs.json
317+
========================================
310318
```
311319

312320
Example output from `payu status` for an archived simulation:
321+
313322

314323
```
315-
========================================
316-
Run: 8
317-
Job ID: archive_example.gadi-pbs
318-
Run ID: xxxx
319-
Stage: archive
320-
Total Queue Time: 0h 1m 7s
321-
Model Finish Time: 1950-10-01T00:00:00
322-
Exit Status: 0 (Success)
323-
Model Exit Code: 0 (Success)
324-
Output Log: /home/189/USER/expt.o100
325-
Error Log: /home/189/USER/expt.3100
326-
Job File: /scratch/\$PROJECT/USER/archive/expt-branch—6dhash/payu_jobs/8/run/archive_example.gadi-pbs.json
327-
========================================
324+
========================================
325+
Run: 8
326+
Job ID: archive_example.gadi-pbs
327+
Run ID: xxxx
328+
Stage: archive
329+
Total Queue Time: 0h 1m 7s
330+
Model Finish Time: 1950-10-01T00:00:00
331+
Exit Status: 0 (Success)
332+
Model Exit Code: 0 (Success)
333+
Output Log: /home/189/USER/expt.o100
334+
Error Log: /home/189/USER/expt.3100
335+
Job File: /scratch/\$PROJECT/USER/archive/expt-branch—6dhash/payu_jobs/8/run/archive_example.gadi-pbs.json
336+
========================================
328337
```
329338

330339
To monitor the current queue time of a queued job, use `payu status --update`.
@@ -361,9 +370,9 @@ To find out more about configuration settings for the `config.yaml` file, refer
361370
Adjusting the duration of the model run is one of the most common change to apply. However, models follow different ways to adapt the duration of the run. Please refer to the [Run a Model][Run a Model] page of the model of your choice for information<br>
362371

363372

373+
<!--start:payu-restart-choice-->
364374
### Start the run from a specific restart file {: id='specific-restart'}
365375

366-
<!--start:payu-restart-choice-->
367376
To configure the experiment to start from specific restart files, add a [`restart:` entry](https://payu.readthedocs.io/en/stable/config.html#miscellaneous) to the `config.yaml` file, specifying the path to a folder containing existing restart files.
368377
Or to do this automatically when setting up an experiment using `payu clone` interactive, give the restart path when prompted: `Do you want to specify a custom restart path?`.
369378

@@ -374,10 +383,10 @@ Or to do this automatically when setting up an experiment using `payu clone` int
374383
The restart option used here will only be applied if there is no restart directory in archive, and so does not have to be removed for subsequent submissions. See [Payu docs](https://payu.readthedocs.io/en/stable/config.html#miscellaneous) for further details.
375384
<!--end:payu-restart-choice-->
376385

377-
<!--start:payu-PBS-resources-->
378-
### Modify PBS resources
386+
<!--start:payu-compute-storage-project-->
387+
### Specify the compute project and storage location {: id='compute-storage-choice'}
379388

380-
If the model has been altered and needs more time or memory to complete, or needs to be submitted under a different NCI project, you will need to modify the following options in the `config.yaml`:
389+
If you want to submit an experiment or part of an experiment using a different project for the compute resources or a non-default location for the archive directory, you will need to modify the following entries in `config.yaml`:
381390

382391
```yaml
383392
# If submitting to a different project to your default, uncomment line below
@@ -386,26 +395,34 @@ If the model has been altered and needs more time or memory to complete, or need
386395

387396
# Force payu to always find, and save, files in this scratch project directory
388397
# shortpath: /scratch/PROJECT_CODE
389-
390-
queue: normal
391-
walltime: 3:00:00
392-
mem: 1000GB
393-
jobname: 1deg_jra55_ryf
394398
```
395399

396-
These lines can be edited to change the [PBS directives](https://opus.nci.org.au/display/Help/PBS+Directives+Explained) for the [PBS job][PBS job].
397-
398400
For example, to run under the {{WG_project}}, uncomment the line beginning with `# project` by deleting the `#` symbol and replace `PROJECT_CODE` with `{{WG_project_code}}`:
399401

400402
```yaml
401403
project: {{WG_project_code}}
402404
```
403405
404-
For model configurations and output to be saved to a `/scratch` storage allocation other than `project` (or your default if `project` is not set) then also set `shortpath` to the desired path.
406+
For model configurations and output to be saved to a `/scratch` storage location other than `project` (or your default if `project` is not set) then also set `shortpath` to the desired path.
405407

406408
!!! warning
407409
If changing the project providing the compute resources during an experiment, set the `shortpath` field so that it's the same for all runs of an experiment.
408410
Doing this will make sure the same `/scratch` location is used for the _laboratory_, regardless of which project is used to run the experiment.
411+
<!--end:payu-compute-storage-project-->
412+
413+
<!--start:payu-PBS-resources-->
414+
### Modify PBS resources
415+
416+
If the model has been altered and needs more time or memory to complete, or needs to be submitted under a different NCI project, you will need to modify the following options in the `config.yaml`:
417+
418+
```yaml
419+
queue: normal
420+
walltime: 3:00:00
421+
mem: 1000GB
422+
jobname: 1deg_jra55_ryf
423+
```
424+
425+
These lines can be edited to change the [PBS directives](https://opus.nci.org.au/display/Help/PBS+Directives+Explained) for the [PBS job][PBS job].
409426
<!--end:payu-PBS-resources-->
410427

411428
<!--start:payu-sync-->
@@ -460,7 +477,8 @@ restart_freq: '50YS'
460477

461478
The most recent sequential restarts are retained, and only deleted after a permanently archived restart file has been produced.
462479

463-
!!! note
480+
??? note "When `restart_freq` is not a multiplier of the model's restart frequency"
481+
464482
If `restart_freq` is not a multiplier of the model's restart frequency, _payu_ will keep the first restart passed `restart_freq`. For example, a model is set to write restart files every 3 years and produces restarts on the following dates:
465483

466484
- restart000: 01/01/2000
@@ -486,25 +504,19 @@ For more information, check [_payu_ Configuration Settings documentation](https:
486504
!!! warning
487505
The following sections in the `config.yaml` file control configuration options that are rarely modified, and often require a deeper understanding of how the model is structured to be safely changed.
488506

489-
#### Model configuration {: .no-toc }
507+
#### `model` section {: .no-toc }
490508

491509
This section tells _payu_ which driver to use for the main `model` configuration and the location of all `input` files that are common to all its model components.
492510

493-
```yaml
494-
name: common
495-
model: access-om2
496-
input: /g/data/ik11/inputs/access-om2/input_20201102/common_1deg_jra55
497-
```
498-
499511
The `name` field, for the model section, is not actually used for the configuration run, so it can be safely ignored. The `name` field is used for submodels (see below).
500512

501-
#### Submodels {: .no-toc }
513+
#### `submodels` section {: .no-toc }
502514

503515
Coupled models may deploy the model components as multiple submodels.
504516

505-
This section of the _payu_ configuration file specifies the submodels, the configuration options required to execute the model component correctly and the location of all inputs required for this submodel.
517+
This section of the _payu_ configuration file specifies the submodels, the configuration options required to execute the model component correctly and the location of all inputs required for this submodel. The configuration files specific to each submodel can be found in a `name/` subdirectory of the _control_ directory, where `name` is the value of this field in the `submodel` section of `config.yaml`.
506518

507-
#### Runlog {: .no-toc }
519+
#### `runlog` field {: .no-toc }
508520

509521
```yaml
510522
runlog: true
@@ -516,30 +528,19 @@ When running an experiment, if `runlog` is set to `true`, _payu_ saves a histor
516528
This should not be changed as it is an essential part of the provenance of an experiment.<br>
517529
_payu_ updates the manifest files for every run, and relies on `runlog` to save this information in the _git_ history, so there is a record of all inputs, restarts, and executables used in an experiment.
518530

519-
#### Userscripts {: .no-toc }
520-
521-
```yaml
522-
userscripts:
523-
error: tools/resub.sh
524-
run: rm -f resubmit.count
525-
sync: /g/data/vk83/apps/om2-scripts/concatenate_ice/concat_ice_daily.sh
526-
```
531+
#### `userscripts` section {: .no-toc }
527532

528533
They are used to run scripts or subcommands at various stages of a _payu_ submission:
529534

530-
- `error` gets called if the model does not run correctly and exits with an error.
531-
- `run` gets called after each model run successful execution, but prior to archiving the model output. If using `payu -n` for automatic resubmission, it is run for each submission.
532-
- `sync` gets called at the start of the sync PBS job. For more information refer to [Syncing output data](#syncing-output-data-to-long-term-storage).
535+
- `error` field: script is called if the model does not run correctly and exits with an error.
536+
- `run` field: script is called after each model run successful execution, but prior to archiving the model output. If using `payu -n` for automatic resubmission, it is run for each submission.
537+
- `sync` field: script is called at the start of the sync PBS job.
533538

534539
For more information about specific `userscripts` fields, check the relevant section of [_payu_ Configuration Settings documentation](https://payu.readthedocs.io/en/stable/config.html#postprocessing).
535540

536-
#### Postscripts {: .no-toc }
541+
#### `postscript` option {: .no-toc }
537542
Postprocessing scripts that run after _payu_ has completed all steps of each run (for example, with `payu run -n 10`, the postscript will run 10 times). Scripts that might alter the output directory, for example, can be run as postscripts. These run in PBS jobs separate from the main model simulation.
538543

539-
```yaml
540-
postscript: -v PAYU_CURRENT_OUTPUT_DIR,PROJECT -lstorage=${PBS_NCI_STORAGE} ./scripts/NetCDF-conversion/UM_conversion_job.sh
541-
```
542-
543544
#### Miscellaneous {: .no-toc }
544545

545546
The following configuration settings should never require changing:
@@ -550,6 +551,25 @@ qsub_flags: -W umask=027
550551
```
551552
<!--end:payu-advance-options-->
552553

554+
<!--start:payu-collate-->
555+
#### Collate {: .no-toc }
556+
557+
Rather than outputting a single diagnostic file over the whole model horizontal grid, the ocean component [MOM](/models/model_components/ocean/#modular-ocean-model-mom) typically generates diagnostic outputs as tiles, each of which spans a portion of model grid.
558+
559+
The `collate` section in the `config.yaml` file controls the process that combines these smaller files into a single outputfile.
560+
561+
```yaml
562+
# Collation
563+
collate:
564+
exe: mppnccombine.spack
565+
restart: true
566+
mem: 4GB
567+
walltime: 1:00:00
568+
mpi: false
569+
```
570+
Restart files are typically tiled in the same way and will also be combined together if the `restart` field is set to`true`.
571+
<!--end:payu-collate-->
572+
553573
<!--start:payu-component-configuration-->
554574
## Edit a model components' configuration
555575

0 commit comments

Comments
 (0)