Skip to content

Commit 61d3a64

Browse files
xylarxylar
committed
Flesh out the developer's guide
This includes: * adding a quick start for developers * adding a note on building the documentation * documenting the test suites * notes on how to make releases
1 parent 239549b commit 61d3a64

6 files changed

Lines changed: 301 additions & 1 deletion

File tree

‎docs/.gitignore‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,4 @@ design_docs/remapper.rst
1414
design_docs/template.rst
1515
design_docs/timekeeping_reorg.rst
1616
design_docs/variable_mapping_reorg.rst
17-
quick_start.rst
17+
/quick_start.rst

‎docs/developers_guide/docs.rst‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
Building the Documentation
2+
--------------------------
3+
4+
With the ``mpas_analysis_dev`` environment activated, you can run:
5+
6+
.. code-block:: bash
7+
8+
cd docs
9+
DOCS_VERSION=test make clean versioned-html
10+
11+
to build the docs locally in the ``_build/html`` subdirectory.
12+
13+
Previewing the Documentation
14+
----------------------------
15+
16+
When generating documentation on HPC machines, you will want to copy the html
17+
output to the public web space to view it, or if the web portal is being
18+
cranky, scp it to your local machine.
19+
20+
To preview the documentation locally, open the ``index.html`` file in the
21+
``_build/html/test`` directory with your browser or try:
22+
23+
.. code-block:: bash
24+
25+
cd _build/html
26+
python -m http.server 8000
27+
28+
Then, open http://0.0.0.0:8000/test/ in your browser.

‎docs/developers_guide/quick_start.rst‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
Quick Start for Developers
2+
==========================
3+
4+
This guide provides a condensed overview for developers to get started with
5+
MPAS-Analysis development.
6+
7+
1. Fork and Clone the Repository
8+
--------------------------------
9+
- Fork `MPAS-Analysis <https://github.com/MPAS-Dev/MPAS-Analysis>`_ on GitHub.
10+
- Clone the main repo and your fork locally:
11+
- Create a base directory (e.g., ``mpas-analysis``).
12+
- Clone the main repo:
13+
``git clone git@github.com:MPAS-Dev/MPAS-Analysis.git develop``
14+
- Add your fork as a remote:
15+
``git remote add <username>/MPAS-Analysis git@github.com:<username>/MPAS-Analysis.git``
16+
17+
2. Configure Git
18+
----------------
19+
- Set up your ``~/.gitconfig`` with your name and email (must match your
20+
GitHub account).
21+
- Recommended: set editor, color, and useful aliases.
22+
23+
3. Set Up SSH Keys
24+
------------------
25+
- Add SSH keys to GitHub for push access.
26+
- See: https://docs.github.com/en/authentication/connecting-to-github-with-ssh/adding-a-new-ssh-key-to-your-github-account
27+
28+
4. Create a Development Worktree
29+
--------------------------------
30+
- Fetch latest changes: ``git fetch --all -p``
31+
- Create a worktree for your feature branch: ``git worktree add ../<feature_branch>``
32+
- Enter the worktree directory: ``cd ../<feature_branch>``
33+
34+
5. Set Up Conda Environment
35+
---------------------------
36+
- Install Miniforge3 (recommended) or Miniconda.
37+
- For Miniconda, add ``conda-forge`` channel and set strict priority.
38+
- Create environment: ``conda create -y -n mpas_analysis_dev --file dev-spec.txt``
39+
- Activate: ``conda activate mpas_analysis_dev``
40+
- Install MPAS-Analysis in edit mode: ``python -m pip install --no-deps --no-build-isolation -e .``
41+
42+
6. Activate Environment (each session)
43+
--------------------------------------
44+
- For bash: ``source ~/miniforge3/etc/profile.d/conda.sh; conda activate mpas_analysis_dev``
45+
- For csh: ``source ~/miniforge3/etc/profile.d/conda.csh; conda activate mpas_analysis_dev``
46+
47+
7. Configure and Run MPAS-Analysis
48+
----------------------------------
49+
- Copy and edit a config file (e.g., ``example_e3sm.cfg``) for your run.
50+
- Set required options: ``mainRunName``, ``baseDirectory``, ``mpasMeshName``, output paths.
51+
- Set ``mapMpiTasks = 1`` and ``mapParallelExec = None`` for development environments.
52+
- Export HDF5 file locking variable if needed:
53+
- Bash: ``export HDF5_USE_FILE_LOCKING=FALSE``
54+
- Csh: ``setenv HDF5_USE_FILE_LOCKING FALSE``
55+
- Run analysis: ``mpas_analysis -m <machine> <your_config>.cfg``
56+
57+
8. View Results
58+
---------------
59+
- Output is a set of web pages in your specified output directory.
60+
- On some systems, update permissions: ``chmod -R ugo+rX <output_dir>``
61+
- See the main web page for links to results and provenance info.
62+
63+
Additional Recommendations
64+
--------------------------
65+
- Use VS Code for remote editing and linting (optional).
66+
67+
For more details, see the full :doc:`tutorials/dev_getting_started`.

‎docs/developers_guide/releasing.rst‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
.. _dev_releasing:
2+
3+
***********************
4+
Releasing a New Version
5+
***********************
6+
7+
This document describes the steps for maintainers to tag and release a new
8+
version of ``MPAS-Analysis``, and to update the conda-forge feedstock.
9+
10+
Version Bump and Dependency Updates
11+
===================================
12+
13+
1. **Update the Version Number**
14+
15+
- Open a pull request (PR) to update the version number in the following
16+
two files:
17+
- ``mpas_analysis/version.py``
18+
- ``ci/recipe/meta.yaml``
19+
20+
- Make sure the version follows [semantic versioning](https://semver.org/).
21+
22+
2. **Check and Update Dependencies**
23+
24+
- Ensure that dependencies and their constraints are up-to-date and
25+
consistent in:
26+
- ``ci/recipe/meta.yaml`` (dependencies for the conda-forge release)
27+
- ``pyproject.toml`` (dependencies for PyPI; used as a sanity check)
28+
- ``dev-spec.txt`` (development dependencies; should be a superset of
29+
those for the conda-forge release)
30+
31+
- The dependencies in ``meta.yaml`` are the ones that will be used for the
32+
released package on conda-forge. The dependencies in ``pyproject.toml``
33+
are for PyPI and should be kept in sync as much as possible but are only
34+
there as a sanity check when we run ```pip check``. The ``dev-spec.txt``
35+
file should include all dependencies needed for development and testing.
36+
37+
- Review and update dependency versions and constraints as needed.
38+
39+
3. **Make a PR and merge it**
40+
41+
Tagging and Publishing a Release
42+
================================
43+
44+
4. **Tag the Release on GitHub**
45+
46+
- Go to https://github.com/MPAS-Dev/MPAS-Analysis/releases and click on
47+
"Draft a new release".
48+
- Enter the appropriate tag for the release, following semantic versioning
49+
(e.g., ``1.13.0``; **do not** include a ``v`` in front).
50+
- Enter a release title (typically the release version **with** a ``v`` in
51+
front, e.g., ``v1.13.0``).
52+
- Write a description and/or use the "Generate release notes" button to
53+
auto-generate release notes.
54+
- If the release is ready, click "Publish release". Otherwise, save it as a
55+
draft.
56+
57+
Updating the conda-forge Feedstock
58+
==================================
59+
60+
5. **Update the conda-forge Feedstock**
61+
62+
- After the release is published, update and merge a PR for the new release
63+
at the conda-forge feedstock:
64+
https://github.com/conda-forge/mpas-analysis-feedstock
65+
66+
- The conda-forge bot should automatically create a pull request for the
67+
new version within a few hours to a day after the release.
68+
69+
- Compare the dependencies in the new release to those in the previous
70+
release and update the recipe as needed. To do this:
71+
- Find the most recent release at
72+
https://github.com/MPAS-Dev/MPAS-Analysis/releases
73+
- Use the "Compare" feature to select the previous release.
74+
- Under "changed files", locate ``ci/recipe/meta.yaml`` to see
75+
any dependency changes.
76+
77+
- Review and update the feedstock PR as needed, then merge it.
78+
79+
- If you are not already a maintainer of the feedstock, you can request to
80+
be added by creating a new issue at
81+
https://github.com/conda-forge/mpas-analysis-feedstock/issues, choosing
82+
"Bot command", and putting
83+
``@conda-forge-admin, please add user @username`` as the subject.

‎docs/developers_guide/test_suite.rst‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
Test Suite Infrastructure
2+
=========================
3+
4+
The `suite` directory provides a comprehensive infrastructure for testing
5+
MPAS-Analysis on supported machines (Anvil, Chrysalis, Perlmutter-CPU, and
6+
Compy). The suite is designed to ensure code changes do not introduce
7+
unexpected results and to validate MPAS-Analysis in various environments.
8+
9+
Overview of Test Scripts
10+
------------------------
11+
12+
There are three main scripts for running the test suite:
13+
14+
1. **run_dev_suite.bash** (Developer Testing)
15+
- Use this script after activating your development environment
16+
(must be named `mpas_analysis_dev`).
17+
- It builds the documentation and runs a series of analysis tasks on output
18+
from a low-resolution (QUwLI240) simulation.
19+
- Each task produces a web page with results, accessible via the web portal.
20+
- Example usage:
21+
.. code-block:: bash
22+
23+
$ source ~/miniforge3/etc/profile.d/conda.sh
24+
$ conda activate mpas_analysis_dev
25+
$ ./suite/run_dev_suite.bash
26+
27+
- After completion, check for successful web page generation, e.g.:
28+
.. code-block:: bash
29+
30+
$ tail -n 3 chrysalis_test_suite/main_py3.11/mpas_analysis.o793058
31+
32+
The last lines should include:
33+
.. code-block:: none
34+
35+
Generating webpage for viewing results...
36+
Web page: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/<username>/analysis_testing/chrysalis/<branch>/main_py3.11/
37+
38+
- To quickly identify unfinished or failed tasks:
39+
.. code-block:: bash
40+
41+
$ grep -L "Web page:" chrysalis_test_suite/*/mpas_analysis.o*
42+
43+
- Developers should run this suite manually on each pull request before
44+
merging and link the results in the PR.
45+
46+
2. **run_suite.bash** (Package Build & Test)
47+
- Use this script to build the MPAS-Analysis conda package and test it in
48+
fresh environments.
49+
- It creates conda environments for multiple Python versions, runs tests,
50+
builds documentation, and executes the analysis suite.
51+
- Recommended for more thorough validation, especially before releases.
52+
- Example usage:
53+
.. code-block:: bash
54+
55+
$ ./suite/run_suite.bash
56+
57+
3. **run_e3sm_unified_suite.bash** (E3SM-Unified Deployment Testing)
58+
- Used during test deployments of E3SM-Unified to verify MPAS-Analysis
59+
works as expected within the deployment.
60+
- Typically run by E3SM-Unified maintainers during deployment testing.
61+
- Example usage:
62+
.. code-block:: bash
63+
64+
$ ./suite/run_e3sm_unified_suite.bash
65+
66+
Supported Machines
67+
------------------
68+
69+
The suite is designed to run only on supported machines:
70+
- Anvil
71+
- Chrysalis
72+
- Perlmutter-CPU (`pm-cpu`)
73+
- Compy
74+
75+
If you attempt to run the suite on an unsupported machine, you will receive an
76+
error.
77+
78+
Modifying the Test Suite
79+
------------------------
80+
81+
Developers may need to update the suite for new requirements:
82+
83+
- **Python Versions**:
84+
- The Python versions tested are defined in the scripts (e.g.,
85+
`main_py=3.11`, `alt_py=3.10`).
86+
- To test additional versions, add them to the relevant script variables and
87+
loops.
88+
89+
- **Adding New Machines**:
90+
- Update the machine detection logic in `suite/setup.py` and add appropriate
91+
input/output paths for the new machine.
92+
- Ensure the new machine is supported in the scripts and the web portal
93+
configuration.
94+
95+
- **Adding/Modifying Tests**:
96+
- To add new tests, update the list of runs in the scripts and
97+
provide corresponding config files in the `suite` directory.
98+
- New tests could change which analysis tasks are run, the configuration for
99+
running tasks overall (e.g. how climatologies are computed), or how
100+
individual tasks are configured (e.g. focused on polar regions vs. global)
101+
102+
- **Changing Simulation Data**:
103+
- Update the simulation name and mesh in `suite/setup.py` if you wish to
104+
test on different output.
105+
106+
Best Practices
107+
--------------
108+
109+
- Always run the test suite before merging a pull request.
110+
- Link the results web page in your PR for reviewers.
111+
- Use the quick check (`grep -L "Web page:" ...`) to ensure all tasks
112+
completed successfully.
113+
- Update the suite scripts and configs as needed to keep pace with
114+
MPAS-Analysis development.
115+
116+
For more details, see the comments and documentation within each script and
117+
config file in the `suite` directory.

‎docs/index.rst‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,11 @@ used those components.
2828
:caption: Developer's guide
2929
:maxdepth: 2
3030

31+
developers_guide/quick_start
32+
developers_guide/docs
33+
developers_guide/test_suite
34+
developers_guide/releasing
35+
3136
developers_guide/api
3237

3338
design_docs/index

0 commit comments

Comments
 (0)