Skip to content

Commit 695da72

Browse files
xylarxylar
committed
Flesh out the developer's guide
This includes: * adding a quick start for developers * adding a note on building the documentation * documenting the test suites * notes on how to make releases
1 parent 239549b commit 695da72

6 files changed

Lines changed: 374 additions & 1 deletion

File tree

‎docs/.gitignore‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,4 @@ design_docs/remapper.rst
1414
design_docs/template.rst
1515
design_docs/timekeeping_reorg.rst
1616
design_docs/variable_mapping_reorg.rst
17-
quick_start.rst
17+
/quick_start.rst

‎docs/developers_guide/docs.rst‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
Building the Documentation
2+
--------------------------
3+
4+
With the ``mpas_analysis_dev`` environment activated, you can run:
5+
6+
.. code-block:: bash
7+
8+
cd docs
9+
DOCS_VERSION=test make clean versioned-html
10+
11+
to build the docs locally in the ``_build/html`` subdirectory.
12+
13+
The docs should build cleanly. If they don't, please attempt to fix the
14+
errors and warnings even if they are not related to your changes. We want
15+
to keep the documentation in good shape.
16+
17+
Previewing the Documentation
18+
----------------------------
19+
20+
When generating documentation on HPC machines, you will want to copy the html
21+
output to the public web space to view it, or if the web portal is being
22+
cranky, scp it to your local machine.
23+
24+
To preview the documentation locally, open the ``index.html`` file in the
25+
``_build/html/test`` directory with your browser or try:
26+
27+
.. code-block:: bash
28+
29+
cd _build/html
30+
python -m http.server 8000
31+
32+
Then, open http://0.0.0.0:8000/test/ in your browser.

‎docs/developers_guide/quick_start.rst‎

Lines changed: 130 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,130 @@
1+
Quick Start for Developers
2+
==========================
3+
4+
This guide provides a condensed overview for developers to get started with
5+
MPAS-Analysis development.
6+
7+
1. Fork and Clone the Repository
8+
--------------------------------
9+
- Fork `MPAS-Analysis <https://github.com/MPAS-Dev/MPAS-Analysis>`_ on GitHub.
10+
- Clone the main repo and your fork locally:
11+
- Create a base directory (e.g., ``mpas-analysis``).
12+
- Clone the main repo:
13+
14+
.. code-block:: bash
15+
16+
git clone git@github.com:MPAS-Dev/MPAS-Analysis.git develop
17+
18+
- Add your fork as a remote:
19+
20+
.. code-block:: bash
21+
22+
git remote add <username>/MPAS-Analysis git@github.com:<username>/MPAS-Analysis.git
23+
24+
2. Configure Git
25+
----------------
26+
- Set up your ``~/.gitconfig`` with your name and email (must match your
27+
GitHub account).
28+
- Recommended: set editor, color, and useful aliases.
29+
30+
3. Set Up SSH Keys
31+
------------------
32+
- Add SSH keys to GitHub for push access.
33+
- See: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account
34+
35+
4. Create a Development Worktree
36+
--------------------------------
37+
- Fetch latest changes:
38+
39+
.. code-block:: bash
40+
41+
git fetch --all -p
42+
43+
- Create a worktree for your feature branch:
44+
45+
.. code-block:: bash
46+
47+
git worktree add ../<feature_branch>
48+
49+
- Enter the worktree directory:
50+
51+
.. code-block:: bash
52+
53+
cd ../<feature_branch>
54+
55+
5. Set Up Conda Environment
56+
---------------------------
57+
- Install Miniforge3 (recommended) or Miniconda.
58+
- For Miniconda, add ``conda-forge`` channel and set strict priority.
59+
- Create environment:
60+
61+
.. code-block:: bash
62+
63+
conda create -y -n mpas_analysis_dev --file dev-spec.txt
64+
65+
- Activate:
66+
67+
.. code-block:: bash
68+
69+
conda activate mpas_analysis_dev
70+
71+
- Install MPAS-Analysis in edit mode:
72+
73+
.. code-block:: bash
74+
75+
python -m pip install --no-deps --no-build-isolation -e .
76+
77+
6. Activate Environment (each session)
78+
--------------------------------------
79+
- For bash:
80+
81+
.. code-block:: bash
82+
83+
source ~/miniforge3/etc/profile.d/conda.sh; conda activate mpas_analysis_dev
84+
85+
- For csh:
86+
87+
.. code-block:: csh
88+
89+
source ~/miniforge3/etc/profile.d/conda.csh; conda activate mpas_analysis_dev
90+
91+
7. Configure and Run MPAS-Analysis
92+
----------------------------------
93+
- Copy and edit a config file (e.g., ``example_e3sm.cfg``) for your run.
94+
- Set required options: ``mainRunName``, ``baseDirectory``, ``mpasMeshName``, output paths.
95+
- Set ``mapMpiTasks = 1`` and ``mapParallelExec = None`` for development environments.
96+
- Export HDF5 file locking variable if needed:
97+
- Bash:
98+
99+
.. code-block:: bash
100+
101+
export HDF5_USE_FILE_LOCKING=FALSE
102+
103+
- Csh:
104+
105+
.. code-block:: csh
106+
107+
setenv HDF5_USE_FILE_LOCKING FALSE
108+
109+
- Run analysis:
110+
111+
.. code-block:: bash
112+
113+
mpas_analysis -m <machine> <your_config>.cfg
114+
115+
8. View Results
116+
---------------
117+
- Output is a set of web pages in your specified output directory.
118+
- On some systems, update permissions:
119+
120+
.. code-block:: bash
121+
122+
chmod -R ugo+rX <output_dir>
123+
124+
- See the main web page for links to results and provenance info.
125+
126+
Additional Recommendations
127+
--------------------------
128+
- Use VS Code for remote editing and linting (optional).
129+
130+
For more details, see the full :doc:`tutorials/dev_getting_started`.

‎docs/developers_guide/releasing.rst‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
.. _dev_releasing:
2+
3+
***********************
4+
Releasing a New Version
5+
***********************
6+
7+
This document describes the steps for maintainers to tag and release a new
8+
version of ``MPAS-Analysis``, and to update the conda-forge feedstock.
9+
10+
Version Bump and Dependency Updates
11+
===================================
12+
13+
1. **Update the Version Number**
14+
15+
- Open a pull request (PR) to update the version number in the following
16+
two files:
17+
- ``mpas_analysis/version.py``
18+
- ``ci/recipe/meta.yaml``
19+
20+
- Make sure the version follows `semantic versioning <https://semver.org/>`_.
21+
22+
2. **Check and Update Dependencies**
23+
24+
- Ensure that dependencies and their constraints are up-to-date and
25+
consistent in:
26+
- ``ci/recipe/meta.yaml`` (dependencies for the conda-forge release)
27+
- ``pyproject.toml`` (dependencies for PyPI; used as a sanity check)
28+
- ``dev-spec.txt`` (development dependencies; should be a superset of
29+
those for the conda-forge release)
30+
31+
- The dependencies in ``meta.yaml`` are the ones that will be used for the
32+
released package on conda-forge. The dependencies in ``pyproject.toml``
33+
are for PyPI and should be kept in sync as much as possible but are only
34+
there as a sanity check when we run ``pip check``. The ``dev-spec.txt``
35+
file should include all dependencies needed for development and testing.
36+
37+
- Review and update dependency versions and constraints as needed.
38+
39+
3. **Make a PR and merge it**
40+
41+
Tagging and Publishing a Release
42+
================================
43+
44+
4. **Tag the Release on GitHub**
45+
46+
- Go to https://github.com/MPAS-Dev/MPAS-Analysis/releases and click on
47+
"Draft a new release".
48+
- Enter the appropriate tag for the release, following semantic versioning
49+
(e.g., ``1.13.0``; **do not** include a ``v`` in front).
50+
- Enter a release title (typically the release version **with** a ``v`` in
51+
front, e.g., ``v1.13.0``).
52+
- Write a description and/or use the "Generate release notes" button to
53+
auto-generate release notes.
54+
- If the release is ready, click "Publish release". Otherwise, save it as a
55+
draft.
56+
57+
Updating the conda-forge Feedstock
58+
==================================
59+
60+
5. **Update the conda-forge Feedstock**
61+
62+
- After the release is published, update and merge a PR for the new release
63+
at the conda-forge feedstock:
64+
https://github.com/conda-forge/mpas-analysis-feedstock
65+
66+
- The conda-forge bot should automatically create a pull request for the
67+
new version within a few hours to a day after the release.
68+
69+
- Compare the dependencies in the new release to those in the previous
70+
release and update the recipe as needed. To do this:
71+
- Find the most recent release at
72+
https://github.com/MPAS-Dev/MPAS-Analysis/releases
73+
- Use the "Compare" feature to select the previous release.
74+
- Under "changed files", locate ``ci/recipe/meta.yaml`` to see
75+
any dependency changes.
76+
77+
- Review and update the feedstock PR as needed, then merge it.
78+
79+
- If you are not already a maintainer of the feedstock, you can request to
80+
be added by creating a new issue at
81+
https://github.com/conda-forge/mpas-analysis-feedstock/issues, choosing
82+
"Bot command", and putting
83+
``@conda-forge-admin, please add user @username`` as the subject.

‎docs/developers_guide/test_suite.rst‎

Lines changed: 123 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,123 @@
1+
Test Suite Infrastructure
2+
=========================
3+
4+
The `suite` directory provides a comprehensive infrastructure for testing
5+
MPAS-Analysis on supported machines (Anvil, Chrysalis, Perlmutter-CPU, and
6+
Compy). The suite is designed to ensure code changes do not introduce
7+
unexpected results and to validate MPAS-Analysis in various environments.
8+
9+
Overview of Test Scripts
10+
------------------------
11+
12+
There are three main scripts for running the test suite:
13+
14+
1. **run_dev_suite.bash** (Developer Testing)
15+
- Use this script after activating your development environment
16+
(must be named `mpas_analysis_dev`).
17+
- It builds the documentation and runs a series of analysis tasks on output
18+
from a low-resolution (QUwLI240) simulation.
19+
- Each task produces a web page with results, accessible via the web portal.
20+
- Example usage:
21+
22+
.. code-block:: bash
23+
24+
$ source ~/miniforge3/etc/profile.d/conda.sh
25+
$ conda activate mpas_analysis_dev
26+
$ ./suite/run_dev_suite.bash
27+
28+
- After completion, check for successful web page generation, e.g.:
29+
30+
.. code-block:: bash
31+
32+
$ tail -n 3 chrysalis_test_suite/main_py3.11/mpas_analysis.o793058
33+
34+
The last lines should include:
35+
36+
.. code-block:: none
37+
38+
Generating webpage for viewing results...
39+
Web page: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/<username>/analysis_testing/chrysalis/<branch>/main_py3.11/
40+
41+
- To quickly identify unfinished or failed tasks:
42+
43+
.. code-block:: bash
44+
45+
$ grep -L "Web page:" chrysalis_test_suite/*/mpas_analysis.o*
46+
47+
- Developers should run this suite manually on each pull request before
48+
merging and link the results in the PR.
49+
50+
2. **run_suite.bash** (Package Build & Test)
51+
- Use this script to build the MPAS-Analysis conda package and test it in
52+
fresh environments.
53+
- It creates conda environments for multiple Python versions, runs tests,
54+
builds documentation, and executes the analysis suite.
55+
- Recommended for more thorough validation, especially before releases.
56+
- Example usage:
57+
58+
.. code-block:: bash
59+
60+
$ ./suite/run_suite.bash
61+
62+
3. **run_e3sm_unified_suite.bash** (E3SM-Unified Deployment Testing)
63+
- Used during test deployments of E3SM-Unified to verify MPAS-Analysis
64+
works as expected within the deployment.
65+
- Typically run by E3SM-Unified maintainers during deployment testing.
66+
- Example usage:
67+
68+
.. code-block:: bash
69+
70+
$ ./suite/run_e3sm_unified_suite.bash
71+
72+
Supported Machines
73+
------------------
74+
75+
The suite is designed to run only on supported machines:
76+
- Anvil
77+
- Chrysalis
78+
- Perlmutter-CPU (`pm-cpu`)
79+
- Compy
80+
81+
If you attempt to run the suite on an unsupported machine, you will receive an
82+
error.
83+
84+
Modifying the Test Suite
85+
------------------------
86+
87+
Developers may need to update the suite for new requirements:
88+
89+
- **Python Versions**:
90+
- The Python versions tested are defined in the scripts (e.g.,
91+
`main_py=3.11`, `alt_py=3.10`).
92+
- To test additional versions, add them to the relevant script variables and
93+
loops.
94+
95+
- **Adding New Machines**:
96+
- Update the machine detection logic in `suite/setup.py` and add appropriate
97+
input/output paths for the new machine.
98+
- Ensure the new machine is supported in the scripts and the web portal
99+
configuration.
100+
101+
- **Adding/Modifying Tests**:
102+
- To add new tests, update the list of runs in the scripts and
103+
provide corresponding config files in the `suite` directory.
104+
- New tests could change which analysis tasks are run, the configuration for
105+
running tasks overall (e.g. how climatologies are computed), or how
106+
individual tasks are configured (e.g. focused on polar regions vs. global)
107+
108+
- **Changing Simulation Data**:
109+
- Update the simulation name and mesh in `suite/setup.py` if you wish to
110+
test on different output.
111+
112+
Best Practices
113+
--------------
114+
115+
- Always run the test suite before merging a pull request.
116+
- Link the results web page in your PR for reviewers.
117+
- Use the quick check (`grep -L "Web page:" ...`) to ensure all tasks
118+
completed successfully.
119+
- Update the suite scripts and configs as needed to keep pace with
120+
MPAS-Analysis development.
121+
122+
For more details, see the comments and documentation within each script and
123+
config file in the `suite` directory.

‎docs/index.rst‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,11 @@ used those components.
2828
:caption: Developer's guide
2929
:maxdepth: 2
3030

31+
developers_guide/quick_start
32+
developers_guide/docs
33+
developers_guide/test_suite
34+
developers_guide/releasing
35+
3136
developers_guide/api
3237

3338
design_docs/index

0 commit comments

Comments
 (0)