From 1b6f63ae130f661969f3ca51168ff5b509179a9e Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Tue, 15 Jul 2025 11:46:04 -0700 Subject: [PATCH 01/11] environments: update for pearc25 --- outputs/environments.sh | 2 +- tutorial_environments.rst | 47 +++++++++++++++++++++++++++++---------- 2 files changed, 36 insertions(+), 13 deletions(-) diff --git a/outputs/environments.sh b/outputs/environments.sh index b76d039318..879d4f14fa 100755 --- a/outputs/environments.sh +++ b/outputs/environments.sh @@ -91,7 +91,7 @@ example --tee environments/env-swap-1 "spack env activate myproject" spack env activate myproject example environments/env-swap-1 "spack find" -example environments/config-get-1 "spack config get" +example environments/config-edit-1 'cat "$(spack location -e)/spack.yaml"' # The file is edited by hand here # We mock that by using `spack config add` diff --git a/tutorial_environments.rst b/tutorial_environments.rst index 728a6caa16..b326f7e029 100644 --- a/tutorial_environments.rst +++ b/tutorial_environments.rst @@ -299,9 +299,14 @@ In this section, we'll learn how to enforce that all the packages in our environ We can customize the selection of the ``mpi`` provider using `concretization preferences `_ to change the behavior of the concretizer. -Let's start by examining our environment's configuration using ``spack config get``: +.. note:: + + Before proceeding, make sure your ``EDITOR`` environment variable + is set to the path of your preferred text editor. -.. literalinclude:: outputs/environments/config-get-1.out +Let's start by examining our environment's configuration using ``spack config edit``: + +.. literalinclude:: outputs/environments/config-edit-1.out :emphasize-lines: 8-13 The output shows the special ``spack.yaml`` configuration file that Spack uses to store environment configurations. @@ -331,14 +336,9 @@ The ``concretizer:unify:true`` setting controls how Spack resolves dependencies Editing environment configuration ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -.. note:: - - Before proceeding, make sure your ``EDITOR`` environment variable - is set to the path of your preferred text editor. - -Let's edit ``spack.yaml`` to *require* ``mpich`` as our ``mpi`` provider using ``spack config edit``. +Let's edit ``spack.yaml`` to *require* ``mpich`` as our ``mpi`` provider. -You should now have the above file open in your editor. +You should have the above file open in your editor. Change it to include the ``packages:mpi:require`` entry below: .. code-block:: yaml @@ -388,10 +388,23 @@ Let's run ``spack concretize --force`` (or ``-f`` in short) to make Spack re-con All the specs are now concrete **and** ready to be installed with ``mpich`` as the MPI implementation. -Re-concretization is sometimes also necessary when creating an environment *incrementally* with unification enabled. -Spack makes sure that already concretized specs in the environment are not modified when adding something new. -Adding and installing specs one by one leads to greedy concretization. +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Creating an environment incrementally +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We can also add and install specs to an environment incrementally. For example: + +.. code-block:: console + + $ spack install --add python + $ spack install --add py-numpy@1.20 + +If we create environments incrementally, Spack ensures that already installed roots are not re-concretized. +So, adding specs to an environment at a later point in time will not cause existing packages to rebuild. + +Adding and installing specs incrementally leads to greedy concretization, meaning that the environment may concain different package versions compared to an environment created all at once. + When you first install ``python`` in an environment, Spack will pick a recent version for it. If you then add ``py-numpy``, it may be in conflict with the ``python`` version already installed, and fail to concretize: @@ -403,6 +416,16 @@ The solution is to re-concretize the environment as a whole, which causes ``pyth .. literalinclude:: outputs/environments/incremental-2.out :language: console +.. note:: + There are advantages to concretizing and installing an environment all at once: + + * If you have a number of specs that can be installed together, + adding them first and installing them together enables them to + share dependencies and reduces total installation time. + + * You can launch all builds in parallel by taking advantage of Spack's + `install-level build parallelism `_. + ------------------------ Building in environments ------------------------ From 8f8355d106e2f748a0a69ff88bb2df118d7b352d Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Tue, 15 Jul 2025 11:58:30 -0700 Subject: [PATCH 02/11] undo changes to output to get docs to build? --- outputs/environments.sh | 2 +- tutorial_environments.rst | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/outputs/environments.sh b/outputs/environments.sh index 879d4f14fa..262f813ab1 100755 --- a/outputs/environments.sh +++ b/outputs/environments.sh @@ -91,7 +91,7 @@ example --tee environments/env-swap-1 "spack env activate myproject" spack env activate myproject example environments/env-swap-1 "spack find" -example environments/config-edit-1 'cat "$(spack location -e)/spack.yaml"' +example environments/config-get-1 "spack config get" # The file is edited by hand here # We mock that by using `spack config add` diff --git a/tutorial_environments.rst b/tutorial_environments.rst index b326f7e029..10f9e00fb7 100644 --- a/tutorial_environments.rst +++ b/tutorial_environments.rst @@ -306,7 +306,7 @@ We can customize the selection of the ``mpi`` provider using `concretization pre Let's start by examining our environment's configuration using ``spack config edit``: -.. literalinclude:: outputs/environments/config-edit-1.out +.. literalinclude:: outputs/environments/config-get-1.out :emphasize-lines: 8-13 The output shows the special ``spack.yaml`` configuration file that Spack uses to store environment configurations. From d87fb8d31a5a2ec1a08dfdbb98c82e636c891e27 Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Tue, 15 Jul 2025 11:58:55 -0700 Subject: [PATCH 03/11] space --- outputs/environments.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/outputs/environments.sh b/outputs/environments.sh index 262f813ab1..b76d039318 100755 --- a/outputs/environments.sh +++ b/outputs/environments.sh @@ -91,7 +91,7 @@ example --tee environments/env-swap-1 "spack env activate myproject" spack env activate myproject example environments/env-swap-1 "spack find" -example environments/config-get-1 "spack config get" +example environments/config-get-1 "spack config get" # The file is edited by hand here # We mock that by using `spack config add` From bed0154f081678845b1ca8a103172b2abd031228 Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Tue, 15 Jul 2025 12:08:26 -0700 Subject: [PATCH 04/11] outputs --- outputs/environments.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/outputs/environments.sh b/outputs/environments.sh index b76d039318..cc156d9330 100755 --- a/outputs/environments.sh +++ b/outputs/environments.sh @@ -91,7 +91,7 @@ example --tee environments/env-swap-1 "spack env activate myproject" spack env activate myproject example environments/env-swap-1 "spack find" -example environments/config-get-1 "spack config get" +example environments/config-get-1 'cat "$(spack location -e)/spack.yaml"' # The file is edited by hand here # We mock that by using `spack config add` From 7b5b1810451a6c59acef4364af5dc0d904a99e28 Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Tue, 15 Jul 2025 12:15:24 -0700 Subject: [PATCH 05/11] try using outputs workflow --- .github/workflows/ci.yml | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fd9f791b11..0c2b1d4f25 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -50,15 +50,15 @@ jobs: needs: changes uses: ./.github/workflows/container.yml - # outputs: - # if: | - # !cancelled() - # && contains(needs.*.result, 'success') - # && !contains(needs.*.result, 'failure') - # && (needs.changes.outputs.outputs == 'true' - # || startsWith(github.ref, 'refs/tags/')) - # needs: [ changes, container ] - # uses: ./.github/workflows/outputs.yml + outputs: + if: | + !cancelled() + && contains(needs.*.result, 'success') + && !contains(needs.*.result, 'failure') + && (needs.changes.outputs.outputs == 'true' + || startsWith(github.ref, 'refs/tags/')) + needs: [ changes, container ] + uses: ./.github/workflows/outputs.yml style: if: ${{ needs.changes.outputs.style == 'true' }} From 214ebb1c10d443630f95db81549169ce710dba67 Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Tue, 15 Jul 2025 12:32:14 -0700 Subject: [PATCH 06/11] recomment --- .github/workflows/ci.yml | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0c2b1d4f25..fd9f791b11 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -50,15 +50,15 @@ jobs: needs: changes uses: ./.github/workflows/container.yml - outputs: - if: | - !cancelled() - && contains(needs.*.result, 'success') - && !contains(needs.*.result, 'failure') - && (needs.changes.outputs.outputs == 'true' - || startsWith(github.ref, 'refs/tags/')) - needs: [ changes, container ] - uses: ./.github/workflows/outputs.yml + # outputs: + # if: | + # !cancelled() + # && contains(needs.*.result, 'success') + # && !contains(needs.*.result, 'failure') + # && (needs.changes.outputs.outputs == 'true' + # || startsWith(github.ref, 'refs/tags/')) + # needs: [ changes, container ] + # uses: ./.github/workflows/outputs.yml style: if: ${{ needs.changes.outputs.style == 'true' }} From 93d2acd33022d23bb2ff42ec5439f7213349932e Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Tue, 15 Jul 2025 12:34:52 -0700 Subject: [PATCH 07/11] spelling --- tutorial_environments.rst | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/tutorial_environments.rst b/tutorial_environments.rst index 10f9e00fb7..4cf458fec7 100644 --- a/tutorial_environments.rst +++ b/tutorial_environments.rst @@ -52,12 +52,18 @@ Goals of this Tutorial ---------------------- This tutorial will teach you the fundamentals of creating and using Spack environments. + We'll cover: + 1. Command line basics -- Creating and managing environments with Spack commands. + 2. Configuration files -- Editing ``spack.yaml`` and understanding ``spack.lock``. + 3. Environment types -- Understanding Spack-managed vs. independent environments. + 4. Reproducible builds -- Sharing and recreating environments across systems. + ------------------- Environment Basics ------------------- @@ -403,9 +409,9 @@ We can also add and install specs to an environment incrementally. For example: If we create environments incrementally, Spack ensures that already installed roots are not re-concretized. So, adding specs to an environment at a later point in time will not cause existing packages to rebuild. -Adding and installing specs incrementally leads to greedy concretization, meaning that the environment may concain different package versions compared to an environment created all at once. +Adding and installing specs incrementally leads to greedy concretization, meaning that the environment may have different package versions compared to an environment created all at once. -When you first install ``python`` in an environment, Spack will pick a recent version for it. +When you first install ``python`` in an environment, Spack will pick a recent version. If you then add ``py-numpy``, it may be in conflict with the ``python`` version already installed, and fail to concretize: .. literalinclude:: outputs/environments/incremental-1.out @@ -417,7 +423,7 @@ The solution is to re-concretize the environment as a whole, which causes ``pyth :language: console .. note:: - There are advantages to concretizing and installing an environment all at once: + There are other advantages to concretizing and installing an environment all at once: * If you have a number of specs that can be installed together, adding them first and installing them together enables them to From 5cd795903357e30432fb2d29bb73ca2b90e01c9a Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Wed, 16 Jul 2025 16:04:33 -0700 Subject: [PATCH 08/11] test building outputs --- .github/workflows/ci.yml | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index fd9f791b11..0c2b1d4f25 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -50,15 +50,15 @@ jobs: needs: changes uses: ./.github/workflows/container.yml - # outputs: - # if: | - # !cancelled() - # && contains(needs.*.result, 'success') - # && !contains(needs.*.result, 'failure') - # && (needs.changes.outputs.outputs == 'true' - # || startsWith(github.ref, 'refs/tags/')) - # needs: [ changes, container ] - # uses: ./.github/workflows/outputs.yml + outputs: + if: | + !cancelled() + && contains(needs.*.result, 'success') + && !contains(needs.*.result, 'failure') + && (needs.changes.outputs.outputs == 'true' + || startsWith(github.ref, 'refs/tags/')) + needs: [ changes, container ] + uses: ./.github/workflows/outputs.yml style: if: ${{ needs.changes.outputs.style == 'true' }} From 2a90caf31574f5f332fa51896d17534013e8d889 Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Sun, 20 Jul 2025 09:50:23 -0700 Subject: [PATCH 09/11] revert output changes --- .github/workflows/ci.yml | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0c2b1d4f25..fd9f791b11 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -50,15 +50,15 @@ jobs: needs: changes uses: ./.github/workflows/container.yml - outputs: - if: | - !cancelled() - && contains(needs.*.result, 'success') - && !contains(needs.*.result, 'failure') - && (needs.changes.outputs.outputs == 'true' - || startsWith(github.ref, 'refs/tags/')) - needs: [ changes, container ] - uses: ./.github/workflows/outputs.yml + # outputs: + # if: | + # !cancelled() + # && contains(needs.*.result, 'success') + # && !contains(needs.*.result, 'failure') + # && (needs.changes.outputs.outputs == 'true' + # || startsWith(github.ref, 'refs/tags/')) + # needs: [ changes, container ] + # uses: ./.github/workflows/outputs.yml style: if: ${{ needs.changes.outputs.style == 'true' }} From b2b46f0c67003c7c522fe0eee0551c3f82f38b3c Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Sun, 20 Jul 2025 10:51:23 -0700 Subject: [PATCH 10/11] use head -12 instead of cat for spack config get --- outputs/environments.sh | 2 +- tutorial_environments.rst | 51 +++++++++------------------------------ 2 files changed, 12 insertions(+), 41 deletions(-) diff --git a/outputs/environments.sh b/outputs/environments.sh index cc156d9330..516387e322 100755 --- a/outputs/environments.sh +++ b/outputs/environments.sh @@ -91,7 +91,7 @@ example --tee environments/env-swap-1 "spack env activate myproject" spack env activate myproject example environments/env-swap-1 "spack find" -example environments/config-get-1 'cat "$(spack location -e)/spack.yaml"' +example environments/config-get-1 "spack config get | head -12" # The file is edited by hand here # We mock that by using `spack config add` diff --git a/tutorial_environments.rst b/tutorial_environments.rst index 4cf458fec7..b626bf4e71 100644 --- a/tutorial_environments.rst +++ b/tutorial_environments.rst @@ -76,6 +76,7 @@ Let's look at the output of ``spack find`` at this point in the tutorial. This is a complete, but cluttered list of all our installed packages and their dependencies. It contains packages built with both ``openmpi`` and ``mpich``, as well as multiple variants of other packages, like ``hdf5`` and ``zlib-ng``. + The query mechanism we learned about with ``spack find`` can help, but it would be nice if we could see only the software that is relevant to our current project instead of seeing everything on the machine. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -155,6 +156,8 @@ Let's try it: :language: console Now, ``tcl`` and ``trilinos`` have been registered as **root specs** in our environment. **Root specs** are packages that we've explicitly requested to be installed in an environment. + + They're called **"roots"** because they sit at the top of the dependency graph---when Spack installs these packages, with their respective dependency packages sitting below them. Now, let's install: @@ -173,38 +176,6 @@ Let's now confirm the contents of the environment using ``spack find``: We can see that the roots and all their dependencies have been installed. -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Creating an environment incrementally -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -We can also add and install specs to an environment incrementally. For example: - -.. code-block:: console - - $ spack install --add tcl - $ spack install --add trilinos - -If we create environments incrementally, Spack ensures that already installed roots are not re-concretized. -So, adding specs to an environment at a later point in time will not cause existing packages to rebuild. - -.. note:: - - Incrementally creating an environment may give us different package - versions from an environment created all at once. - We'll cover this later in the tutorial after we've discussed different - concretization strategies. - - Further, there are two other advantages of concretizing and installing an - environment all at once: - - - * If you have a number of specs that can be installed together, - adding them first and installing them together enables them to - share dependencies and reduces total installation time. - - * You can launch all builds in parallel by taking advantage of Spack's - `install-level build parallelism `_. - ^^^^^^^^^^^^^^ Using Packages ^^^^^^^^^^^^^^ @@ -305,15 +276,10 @@ In this section, we'll learn how to enforce that all the packages in our environ We can customize the selection of the ``mpi`` provider using `concretization preferences `_ to change the behavior of the concretizer. -.. note:: - Before proceeding, make sure your ``EDITOR`` environment variable - is set to the path of your preferred text editor. - -Let's start by examining our environment's configuration using ``spack config edit``: +Let's start by examining our environment's configuration using ``spack config get``: .. literalinclude:: outputs/environments/config-get-1.out - :emphasize-lines: 8-13 The output shows the special ``spack.yaml`` configuration file that Spack uses to store environment configurations. @@ -342,9 +308,14 @@ The ``concretizer:unify:true`` setting controls how Spack resolves dependencies Editing environment configuration ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Let's edit ``spack.yaml`` to *require* ``mpich`` as our ``mpi`` provider. +.. note:: + + Before proceeding, make sure your ``EDITOR`` environment variable + is set to the path of your preferred text editor. + +Let's edit ``spack.yaml`` to *require* ``mpich`` as our ``mpi`` provider using ``spack config edit``. -You should have the above file open in your editor. +You should now have the above file open in your editor. Change it to include the ``packages:mpi:require`` entry below: .. code-block:: yaml From 559defcf6ef4df7ab4111b159e659c8b47886811 Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Sun, 20 Jul 2025 10:52:50 -0700 Subject: [PATCH 11/11] rm extra newline --- tutorial_environments.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/tutorial_environments.rst b/tutorial_environments.rst index b626bf4e71..2b5c0042e9 100644 --- a/tutorial_environments.rst +++ b/tutorial_environments.rst @@ -157,7 +157,6 @@ Let's try it: Now, ``tcl`` and ``trilinos`` have been registered as **root specs** in our environment. **Root specs** are packages that we've explicitly requested to be installed in an environment. - They're called **"roots"** because they sit at the top of the dependency graph---when Spack installs these packages, with their respective dependency packages sitting below them. Now, let's install: @@ -276,7 +275,6 @@ In this section, we'll learn how to enforce that all the packages in our environ We can customize the selection of the ``mpi`` provider using `concretization preferences `_ to change the behavior of the concretizer. - Let's start by examining our environment's configuration using ``spack config get``: .. literalinclude:: outputs/environments/config-get-1.out