From 15342f40e85098da3f2860d3df2a02510c7d80a8 Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Mon, 2 Jun 2025 15:59:54 -0700 Subject: [PATCH 1/5] harmen/google suggestions applied --- tutorial_configuration.rst | 44 ++--- tutorial_developer_workflows.rst | 276 ++++++++++++++----------------- 2 files changed, 147 insertions(+), 173 deletions(-) diff --git a/tutorial_configuration.rst b/tutorial_configuration.rst index 1c6f786259..fbcd119fbc 100644 --- a/tutorial_configuration.rst +++ b/tutorial_configuration.rst @@ -24,30 +24,30 @@ A partial list of some key configuration sections is provided below. * - config - General settings (install location, number of build jobs, etc) * - concretizer - - Specializaiton of the concretizer behavior (reuse, unification, etc) + - Specialization of the concretizer behavior (reuse, unification, etc) * - compilers - Define the compilers that Spack can use (required and system specific) * - mirrors - - Locations where spack can look for stashed source or binary distributions + - Locations where Spack can look for stashed source or binary distributions * - packages - Specific settings and rules for packages * - modules - Naming, location and additional configuration of Spack generated modules The full list of sections can be viewed with ``spack config list``. -For further education we encourage you to explore the spack `documentation on configuration files `_. +For more details, we encourage you to explore the Spack `documentation on configuration files `_. -The principle goals of this section of the tutorial are: +The principal goals of this section of the tutorial are: 1. Introduce the configuration sections and scope hierarchy 2. Demonstrate how to manipulate configurations -3. Show how to configure system assets with spack (compilers and packages) +3. Show how to configure system assets with Spack (compilers and packages) -As such we will primarily focus on the ``compilers`` and ``packages`` configuration sections in this portion of the tutorial. +As such, we will primarily focus on the ``compilers`` and ``packages`` configuration sections in this portion of the tutorial. We will explain this by first covering how to manipulate configurations from the command line and then show how this impacts the configuration file hierarchy. We will then move into compiler and package configurations to help you develop skills for getting the builds you want on your system. -Finally, we will give some brief attention to more generalized spack configurations in the ``config`` section. +Finally, we will give some brief attention to more generalized Spack configurations in the ``config`` section. For all of these features, we will demonstrate how we build up a full configuration file. For some, we will then demonstrate how the configuration affects the install command, and for others we will use the ``spack spec`` command to demonstrate how the configuration changes have affected Spack's concretization algorithm. @@ -58,7 +58,7 @@ Configuration from the command line ----------------------------------- You can run ``spack config blame [section]`` at any point in time to see what your current configuration is. -If you omit the section then spack will dump all the configurations settings to your screen. +If you omit the section, Spack will show all configuration settings. Let's go ahead and run this for the ``concretizer`` section. .. code-block:: console @@ -80,10 +80,10 @@ If we rerun ``spack config blame concretizer`` we can see that the change was ap $ spack config blame concretizer -Notice that the reference file on for this option is now different. -This indicates the scope where the configuration was set in, and we will discuss how spack chooses the default scope shortly. +Notice that the reference file for this option is now different. +This indicates the scope where the configuration was set, and we will discuss how Spack chooses the default scope shortly. For now, it is important to note that the ``spack config`` command accepts an optional ``--scope`` flag so we can be more precise in the configuration process. -This will make more sense after the next section which provides the definition of spack's configuration scopes and their hierarchy. +This will make more sense after the next section, which provides the definition of Spack's configuration scopes and their hierarchy. .. _configs-tutorial-scopes: @@ -131,14 +131,14 @@ Some facilities manage multiple platforms from a single shared file system. In order to handle this, each of the configuration scopes listed above has two *sub-scopes*: platform-specific and platform-independent. For example, compiler settings can be stored in the following locations: -#. ``$ENVIRONMENT_ROOT/spack.yaml`` -#. ``~/.spack//compilers.yaml`` -#. ``~/.spack/compilers.yaml`` -#. ``$SPACK_ROOT/etc/spack//compilers.yaml`` -#. ``$SPACK_ROOT/etc/spack/compilers.yaml`` +#. ``$ENVIRONMENT_ROOT/spack.yaml`` +#. ``~/.spack//compilers.yaml`` +#. ``~/.spack/compilers.yaml`` +#. ``$SPACK_ROOT/etc/spack//compilers.yaml`` +#. ``$SPACK_ROOT/etc/spack/compilers.yaml`` #. ``/etc/spack//compilers.yaml`` -#. ``/etc/spack/compilers.yaml`` -#. ``$SPACK_ROOT/etc/defaults//compilers.yaml`` +#. ``/etc/spack/compilers.yaml`` +#. ``$SPACK_ROOT/etc/defaults//compilers.yaml`` #. ``$SPACK_ROOT/etc/defaults/compilers.yaml`` These files are listed in decreasing order of precedence, so files in ``~/.spack/`` will override settings in ``~/.spack``. @@ -243,8 +243,8 @@ We start with no active environment, so this will open a ``compilers.yaml`` file paths: cc: /usr/bin/gcc-10 cxx: /usr/bin/g++-10 - f77: usr/bin/gfortran-10 - fc: usr/bin/gfortran-10 + f77: /usr/bin/gfortran-10 + fc: /usr/bin/gfortran-10 flags: {} operating_system: ubuntu22.04 target: x86_64 @@ -308,7 +308,7 @@ We can verify that our new compiler works by invoking it now: This new compiler also works on Fortran codes. -We'll show it by compiling a small package using as a build dependency ``cmake%gcc@11.4.0`` since it is already available in our binary cache: +We'll show it by compiling a small package using ``cmake%gcc@11.4.0`` as a build dependency since it is already available in our binary cache: .. code-block:: console @@ -793,7 +793,7 @@ If we uninstall and reinstall zlib-ng, we see that it now uses only 2 cores: Obviously, if you want to build everything in serial for whatever reason, you would set ``build_jobs`` to 1. -Last we'll unset ``concretizer:reuse:false`` since we'll want to enable concretizer reuse for the rest of this tutorial. +Last, we'll unset ``concretizer:reuse:false``, as we'll want to enable concretizer reuse for the rest of this tutorial. .. code-block:: yaml diff --git a/tutorial_developer_workflows.rst b/tutorial_developer_workflows.rst index 5611f3cc9c..3dea5f7954 100644 --- a/tutorial_developer_workflows.rst +++ b/tutorial_developer_workflows.rst @@ -11,31 +11,39 @@ Developer Workflows Tutorial ============================ -This tutorial will guide you through the process of using the ``spack develop`` command to develop software from local source code within a spack environment. -With this command spack will manage your dependencies while you focus on testing changes to your library and/or application. - +This tutorial will guide you through using the ``spack develop`` command to develop software from local source code within a Spack environment. +With this command, Spack manages your dependencies while you focus on testing changes to your library and/or application. ----------------------------- -Installing from local source +Installing from Local Source ----------------------------- -The ``spack install`` command, as you know, fetches source code from a mirror or the internet before building and installing your package. -As developers, we want to build from local source, which we will constantly change, build, and test. +The ``spack install`` command typically fetches source code from a mirror or the internet before building and installing your package. +As developers, however, we often want to build from local source code that we continuously modify, build, and test. + +Let's imagine we're working on ``scr``—a library used to implement scalable checkpointing in application codes. +It supports fast, efficient checkpoint read/write operations using MPI and high-bandwidth file I/O. +We want to test changes to ``scr`` within an actual application, so we'll use ``macsio``, a proxy application that mimics typical HPC I/O workloads. + +We've chosen ``scr`` and ``macsio`` because they are both quick to build. + +We'll begin by creating a Spack environment for our development work. +We need to build ``macsio`` with ``scr`` support, and for now, we want everything built without Fortran support. +Let's set up this development workflow: + +.. literalinclude:: outputs/dev/setup-scr.out + :language: console -Let's imagine for a second we're working on ``scr``. ``scr`` is a library used to implement scalable checkpointing in application codes. -It supports writing/reading checkpoints quickly and efficiently using MPI and high-bandwidth file I/O. -We'd like to test changes to scr within an actual application so we'll test with ``macsio``, a proxy application written to mimic typical HPC I/O workloads. -We've chosen ``scr`` and ``macsio`` because together they are quick to build. +Before making any changes, we verify that everything builds correctly. +Spack builds the entire development tree as specified and links all components together for you. -We'll start by making an environment for our development. -We need to build ``macsio`` with ``scr`` support, and we'd like everything to be built without fortran support for the time being. -Let's set up that development workflow. .. literalinclude:: outputs/dev/setup-scr.out :language: console -Before we do any work, we verify that this all builds. -Spack ends up building the entire development tree below, and links everything together for you. + +Before making any changes, we verify that everything builds correctly. +Spack builds the entire development tree as specified and links all components together for you. .. graphviz:: @@ -82,158 +90,106 @@ Spack ends up building the entire development tree below, and links everything t "smoyzzo2qhzpn6mg6rd3l2p7b23enshg" [label="zlib"] "sz72vygmht66khd5aa4kihz5alg4nrbm" [label="macsio"] - "wbqbc5vw5sxzwhvu56p6x5nd5n4abrvh" -> "jdxbjftheiotj6solpomva7dowrhlerl" - "zqwfzhw5k2ollygh6nrjpsi7u4d4g6lu" -> "4sh6pymrm2ms4auu3ajbjjr6fiuhz5g7" - "sz72vygmht66khd5aa4kihz5alg4nrbm" -> "vfrf7asfclt7epufnoxibfqbkntbk5k3" - "vfrf7asfclt7epufnoxibfqbkntbk5k3" -> "t54jzdy2jj4snltjazlm3br2urcilc6v" - "crhlefo3dv7lmsv5pf4icsy4gepkdorm" -> "4sh6pymrm2ms4auu3ajbjjr6fiuhz5g7" - "7tkgwjvu2mi4ea2wsdetunq7g4k4r2nh" -> "mkc3u4x2p2wie6jfhuku7g5rkovcrxps" - "sz72vygmht66khd5aa4kihz5alg4nrbm" -> "pmsyupw6w3gql4loaor25gfumlmvkl25" - "zqwfzhw5k2ollygh6nrjpsi7u4d4g6lu" -> "wbqbc5vw5sxzwhvu56p6x5nd5n4abrvh" - "7tkgwjvu2mi4ea2wsdetunq7g4k4r2nh" -> "d2krmb5gweivlnztcymhklzsqbrpatt6" - "es377uqsqougfc67jyg7yfjyyuukin52" -> "smoyzzo2qhzpn6mg6rd3l2p7b23enshg" - "bltycqwh5oofai4f6o42q4uuj4w5zb3j" -> "crhlefo3dv7lmsv5pf4icsy4gepkdorm" - "mm33a3ocsv3jsh2tfxc4mlab4xsurtdd" -> "zfdvt2jjuaees43ffrrtphqs2ky3o22t" - "es377uqsqougfc67jyg7yfjyyuukin52" -> "zfdvt2jjuaees43ffrrtphqs2ky3o22t" - "7tkgwjvu2mi4ea2wsdetunq7g4k4r2nh" -> "jdxbjftheiotj6solpomva7dowrhlerl" - "mkc3u4x2p2wie6jfhuku7g5rkovcrxps" -> "lbrx7lnfz46ukewxbhxnucmx76g23c6q" - "bltycqwh5oofai4f6o42q4uuj4w5zb3j" -> "es377uqsqougfc67jyg7yfjyyuukin52" - "vedchc5aoqyu3ydbp346qrbpe6kg46rq" -> "smoyzzo2qhzpn6mg6rd3l2p7b23enshg" - "wbqbc5vw5sxzwhvu56p6x5nd5n4abrvh" -> "d2krmb5gweivlnztcymhklzsqbrpatt6" - "zfdvt2jjuaees43ffrrtphqs2ky3o22t" -> "4av4gywgpaspkhy3dvbb62nulqogtzbb" - "vedchc5aoqyu3ydbp346qrbpe6kg46rq" -> "pmsyupw6w3gql4loaor25gfumlmvkl25" - "d2krmb5gweivlnztcymhklzsqbrpatt6" -> "mm33a3ocsv3jsh2tfxc4mlab4xsurtdd" - "bob4o5m3uku6vtdil5imasprgy775zg7" -> "jdxbjftheiotj6solpomva7dowrhlerl" - "yn2r3wfhiilelyulh5toteicdtxjhw7d" -> "komekkmyciga3kl24edjmredhj3uyt7v" - "pmsyupw6w3gql4loaor25gfumlmvkl25" -> "smoyzzo2qhzpn6mg6rd3l2p7b23enshg" - "wbqbc5vw5sxzwhvu56p6x5nd5n4abrvh" -> "mm33a3ocsv3jsh2tfxc4mlab4xsurtdd" - "vfrf7asfclt7epufnoxibfqbkntbk5k3" -> "vedchc5aoqyu3ydbp346qrbpe6kg46rq" - "bob4o5m3uku6vtdil5imasprgy775zg7" -> "gs6ag7ktdoiirb62t7bcagjw62szrrg2" - "d2krmb5gweivlnztcymhklzsqbrpatt6" -> "zfdvt2jjuaees43ffrrtphqs2ky3o22t" - "7tkgwjvu2mi4ea2wsdetunq7g4k4r2nh" -> "mm33a3ocsv3jsh2tfxc4mlab4xsurtdd" - "vfrf7asfclt7epufnoxibfqbkntbk5k3" -> "smoyzzo2qhzpn6mg6rd3l2p7b23enshg" - "zfdvt2jjuaees43ffrrtphqs2ky3o22t" -> "4ihuiazsglf22f3pntq5hc4kyszqzexn" - "bob4o5m3uku6vtdil5imasprgy775zg7" -> "4sh6pymrm2ms4auu3ajbjjr6fiuhz5g7" - "vfrf7asfclt7epufnoxibfqbkntbk5k3" -> "pmsyupw6w3gql4loaor25gfumlmvkl25" - "zqwfzhw5k2ollygh6nrjpsi7u4d4g6lu" -> "bob4o5m3uku6vtdil5imasprgy775zg7" - "yn2r3wfhiilelyulh5toteicdtxjhw7d" -> "jearpk4xci4zc7dkrza4fufaqfkq7rfl" - "sz72vygmht66khd5aa4kihz5alg4nrbm" -> "bltycqwh5oofai4f6o42q4uuj4w5zb3j" - "pmsyupw6w3gql4loaor25gfumlmvkl25" -> "wbqbc5vw5sxzwhvu56p6x5nd5n4abrvh" - "sz72vygmht66khd5aa4kihz5alg4nrbm" -> "7tkgwjvu2mi4ea2wsdetunq7g4k4r2nh" - "yn2r3wfhiilelyulh5toteicdtxjhw7d" -> "smoyzzo2qhzpn6mg6rd3l2p7b23enshg" - "t54jzdy2jj4snltjazlm3br2urcilc6v" -> "crhlefo3dv7lmsv5pf4icsy4gepkdorm" - "pmsyupw6w3gql4loaor25gfumlmvkl25" -> "zqwfzhw5k2ollygh6nrjpsi7u4d4g6lu" - "4av4gywgpaspkhy3dvbb62nulqogtzbb" -> "t54jzdy2jj4snltjazlm3br2urcilc6v" - "jdxbjftheiotj6solpomva7dowrhlerl" -> "mkc3u4x2p2wie6jfhuku7g5rkovcrxps" - "yn2r3wfhiilelyulh5toteicdtxjhw7d" -> "4sh6pymrm2ms4auu3ajbjjr6fiuhz5g7" - "mm33a3ocsv3jsh2tfxc4mlab4xsurtdd" -> "mkc3u4x2p2wie6jfhuku7g5rkovcrxps" - "zqwfzhw5k2ollygh6nrjpsi7u4d4g6lu" -> "yn2r3wfhiilelyulh5toteicdtxjhw7d" - "pmsyupw6w3gql4loaor25gfumlmvkl25" -> "4sh6pymrm2ms4auu3ajbjjr6fiuhz5g7" - "wbqbc5vw5sxzwhvu56p6x5nd5n4abrvh" -> "mkc3u4x2p2wie6jfhuku7g5rkovcrxps" - } +"wbqbc5vw5sxzwhvu56p6x5nd5n4abrvh" -> "jdxbjftheiotj6solpomva7dowrhlerl" "zqwfzhw5k2ollygh6nrjpsi7u4d4g6lu" -> "4sh6pymrm2ms4auu3ajbjjr6fiuhz5g7" "sz72vygmht66khd5aa4kihz5alg4nrbm" -> "vfrf7asfclt7epufnoxibfqbkntbk5k3" "vfrf7asfclt7epufnoxibfqbkntbk5k3" -> "t54jzdy2jj4snltjazlm3br2urcilc6v" "crhlefo3dv7lmsv5pf4icsy4gepkdorm" -> "4sh6pymrm2ms4auu3ajbjjr6fiuhz5g7" "7tkgwjvu2mi4ea2wsdetunq7g4k4r2nh" -> "mkc3u4x2p2wie6jfhuku7g5rkovcrxps" "sz72vygmht66khd5aa4kihz5alg4nrbm" -> "pmsyupw6w3gql4loaor25gfumlmvkl25" "zqwfzhw5k2ollygh6nrjpsi7u4d4g6lu" -> "wbqbc5vw5sxzwhvu56p6x5nd5n4abrvh" "7tkgwjvu2mi4ea2wsdetunq7g4k4r2nh" -> "d2krmb5gweivlnztcymhklzsqbrpatt6" "es377uqsqougfc67jyg7yfjyyuukin52" -> "smoyzzo2qhzpn6mg6rd3l2p7b23enshg" "bltycqwh5oofai4f6o42q4uuj4w5zb3j" -> "crhlefo3dv7lmsv5pf4icsy4gepkdorm" "mm33a3ocsv3jsh2tfxc4mlab4xsurtdd" -> "zfdvt2jjuaees43ffrrtphqs2ky3o22t" "es377uqsqougfc67jyg7yfjyyuukin52" -> "zfdvt2jjuaees43ffrrtphqs2ky3o22t" "7tkgwjvu2mi4ea2wsdetunq7g4k4r2nh" -> "jdxbjftheiotj6solpomva7dowrhlerl" "mkc3u4x2p2wie6jfhuku7g5rkovcrxps" -> "lbrx7lnfz46ukewxbhxnucmx76g23c6q" "bltycqwh5oofai4f6o42q4uuj4w5zb3j" -> "es377uqsqougfc67jyg7yfjyyuukin52" "vedchc5aoqyu3ydbp346qrbpe6kg46rq" -> "smoyzzo2qhzpn6mg6rd3l2p7b23enshg" "wbqbc5vw5sxzwhvu56p6x5nd5n4abrvh" -> "d2krmb5gweivlnztcymhklzsqbrpatt6" "zfdvt2jjuaees43ffrrtphqs2ky3o22t" -> "4av4gywgpaspkhy3dvbb62nulqogtzbb" "vedchc5aoqyu3ydbp346qrbpe6kg46rq" -> "pmsyupw6w3gql4loaor25gfumlmvkl25" "d2krmb5gweivlnztcymhklzsqbrpatt6" -> "mm33a3ocsv3jsh2tfxc4mlab4xsurtdd" "bob4o5m3uku6vtdil5imasprgy775zg7" -> "jdxbjftheiotj6solpomva7dowrhlerl" "yn2r3wfhiilelyulh5toteicdtxjhw7d" -> "komekkmyciga3kl24edjmredhj3uyt7v" "pmsyupw6w3gql4loaor25gfumlmvkl25" -> "smoyzzo2qhzpn6mg6rd3l2p7b23enshg" "wbqbc5vw5sxzwhvu56p6x5nd5n4abrvh" -> "mm33a3ocsv3jsh2tfxc4mlab4xsurtdd" "vfrf7asfclt7epufnoxibfqbkntbk5k3" -> "vedchc5aoqyu3ydbp346qrbpe6kg46rq" "bob4o5m3uku6vtdil5imasprgy775zg7" -> "gs6ag7ktdoiirb62t7bcagjw62szrrg2" "d2krmb5gweivlnztcymhklzsqbrpatt6" -> "zfdvt2jjuaees43ffrrtphqs2ky3o22t" "7tkgwjvu2mi4ea2wsdetunq7g4k4r2nh" -> "mm33a3ocsv3jsh2tfxc4mlab4xsurtdd" "vfrf7asfclt7epufnoxibfqbkntbk5k3" -> "smoyzzo2qhzpn6mg6rd3l2p7b23enshg" "zfdvt2jjuaees43ffrrtphqs2ky3o22t" -> "4ihuiazsglf22f3pntq5hc4kyszqzexn" "bob4o5m3uku6vtdil5imasprgy775zg7" -> "4sh6pymrm2ms4auu3ajbjjr6fiuhz5g7" "vfrf7asfclt7epufnoxibfqbkntbk5k3" -> "pmsyupw6w3gql4loaor25gfumlmvkl25" "zqwfzhw5k2ollygh6nrjpsi7u4d4g6lu" -> "bob4o5m3uku6vtdil5imasprgy775zg7" "yn2r3wfhiilelyulh5toteicdtxjhw7d" -> "jearpk4xci4zc7dkrza4fufaqfkq7rfl" "sz72vygmht66khd5aa4kihz5alg4nrbm" -> "bltycqwh5oofai4f6o42q4uuj4w5zb3j" "pmsyupw6w3gql4loaor25gfumlmvkl25" -> "wbqbc5vw5sxzwhvu56p6x5nd5n4abrvh" "sz72vygmht66khd5aa4kihz5alg4nrbm" -> "7tkgwjvu2mi4ea2wsdetunq7g4k4r2nh" "yn2r3wfhiilelyulh5toteicdtxjhw7d" -> "smoyzzo2qhzpn6mg6rd3l2p7b23enshg" "t54jzdy2jj4snltjazlm3br2urcilc6v" -> "crhlefo3dv7lmsv5pf4icsy4gepkdorm" "pmsyupw6w3gql4loaor25gfumlmvkl25" -> "zqwfzhw5k2ollygh6nrjpsi7u4d4g6lu" "4av4gywgpaspkhy3dvbb62nulqogtzbb" -> "t54jzdy2jj4snltjazlm3br2urcilc6v" "jdxbjftheiotj6solpomva7dowrhlerl" -> "mkc3u4x2p2wie6jfhuku7g5rkovcrxps" "yn2r3wfhiilelyulh5toteicdtxjhw7d" -> "4sh6pymrm2ms4auu3ajbjjr6fiuhz5g7" "mm33a3ocsv3jsh2tfxc4mlab4xsurtdd" -> "mkc3u4x2p2wie6jfhuku7g5rkovcrxps" "zqwfzhw5k2ollygh6nrjpsi7u4d4g6lu" -> "yn2r3wfhiilelyulh5toteicdtxjhw7d" "pmsyupw6w3gql4loaor25gfumlmvkl25" -> "4sh6pymrm2ms4auu3ajbjjr6fiuhz5g7" "wbqbc5vw5sxzwhvu56p6x5nd5n4abrvh" -> "mkc3u4x2p2wie6jfhuku7g5rkovcrxps" } + +Now we are ready to begin work on the actual application. Now we are ready to begin work on the actual application. ----------------------------- -Development iteration cycles +Development Iteration Cycle ----------------------------- -Let's assume that scr has a bug, and we'd like to patch scr to find out what the problem is. -First, we tell spack that we'd like to check out the version of scr that we want to work on. -In this case, it will be the 3.1.0 release that we want to write a patch for: +Let's assume that ``scr`` has a bug, and we'd like to patch it to find out what the problem is. +First, we tell Spack that we want to check out the version of ``scr`` we intend to work on. +In this case, it's the 3.1.0 release that we want to patch: .. literalinclude:: outputs/dev/develop-1.out :language: console -The spack develop command marks the package as being a "development" package in the spack.yaml. -This adds a special ``dev_path=`` attribute to the spec for the package, so spack remembers where the source code for this package is located. -The develop command also downloads/checks out the source code for the package. -By default, the source code is downloaded into a subdirectory of the environment. -You can change the location of this source directory by modifying the ``path:`` attribute of the develop configuration in the environment. - -There are a few gotchas with the spack develop command - -* You often specify the package version manually when specifying a - package as a dev package. Spack needs to know the version of the dev - package so it can supply the correct flags for the package's build - system. If a version is not supplied then spack will take the maximum version - defined in the package where where `infinity versions `_ like ``develop`` and ``main`` - have a higher value than the numeric versions. -* You should ensure a spec for the package you are developing appears in the DAG of at least one of the roots of the environment with the same version that you are developing. - ``spack add `` with the matching version you want to develop is a way to ensure - the develop spec is satisfied.the ``spack.yaml`` environments file. This is because - develop specs are not concretization constraints but rather a criteria for adding - the ``dev_path=`` variant to existing spec. -* You'll need to re-concretize the environment so that the version - number and the ``dev_path=`` attributes are properly added to the - cached spec in ``spack.lock``. +The ``spack develop`` command marks the package as a "development" package in the ``spack.yaml``. +This adds a special ``dev_path=`` attribute to the spec for the package, so Spack remembers where the source code is located. +The ``develop`` command also downloads or checks out the source code for the package. +By default, the source code is downloaded into a subdirectory of the environment. +You can change the location of this source directory by modifying the ``path:`` attribute of the ``develop`` configuration in the environment. + +There are a few gotchas with the ``spack develop`` command: + +* You often need to specify the package version manually when marking a package as a development package. + Spack requires this version so it can supply the correct flags for the package's build system. + If no version is supplied, Spack uses the highest version defined in the package—where `infinity versions `_ like ``develop`` and ``main`` rank higher than numeric versions. + +* You must ensure that a spec for the package you are developing appears in the DAG of at least one of the environment's roots, using the same version that you are developing. + ``spack add @`` is one way to ensure that the ``develop`` spec is satisfied in the ``spack.yaml``. + This is necessary because develop specs are not treated as concretization constraints, but rather as criteria for adding the ``dev_path=`` variant to an existing spec. + +* You'll need to re-concretize the environment so that the version number and ``dev_path=`` attributes are properly added to the cached spec in ``spack.lock``. .. literalinclude:: outputs/dev/develop-conc.out :language: console -Now that we have this done, we tell spack to rebuild both ``scr`` and ``macsio`` by running ``spack install``. +Now that this is done, we tell Spack to rebuild both ``scr`` and ``macsio`` by running ``spack install``. .. literalinclude:: outputs/dev/develop-2.out :language: console -This rebuilds ``scr`` from the subdirectory we specified. -If your package uses cmake, spack will build the package in a build directory that matches the hash for your package. -From here you can change into the appropriate directory and perform your own build/test cycles. +This rebuilds ``scr`` from the subdirectory we specified. +If your package uses CMake, Spack will build it in a directory corresponding to the hash of your package. +From there, you can change into the appropriate directory and perform your own build/test cycles. -Now, we can develop our code. -For the sake of this demo, we're just going to intentionally introduce an error. -Let's edit a file and remove the first semi-colon we find. +Now we can develop our code. +For the sake of this demo, we're going to intentionally introduce an error. +Let’s edit a file and remove the first semicolon we find. .. literalinclude:: outputs/dev/edit-1.out :language: console -Once you have a development package, ``spack install`` also works much like "make". -Since spack knows the source code directory of the package, it checks the filetimes on the source directory to see if we've made recent changes. -If the file times are newer, it will rebuild ``scr`` and any other package that depends on ``scr``. +Once you have a development package, ``spack install`` works like ``make``. +Because Spack knows the source code directory of the package, it checks file timestamps to detect recent changes. +If any files have been modified, Spack will rebuild ``scr`` and its dependents. .. literalinclude:: outputs/dev/develop-3.out :language: console -Here, the build failed as expected. -We can look at the output for the build in ``scr/spack-build-out.txt`` to find out why, or we can launch a shell directly with the appropriate environment variables to figure out what went wrong by using ``spack build-env scr@2.0 -- bash``. -If that's too much to remember, then sourcing ``scr/spack-build-env.txt`` will also set all the appropriate environment variables so we can diagnose the build ourselves. -Now let's fix it and rebuild directly. +Here, the build failed as expected. +We can inspect the build output in ``scr/spack-build-out.txt``. +Alternatively, to debug interactively, we can launch a shell within the build environment using: + +``spack build-env scr@2.0 -- bash`` + +If that’s too much to remember, sourcing ``scr/spack-build-env.txt`` will set the appropriate environment variables so you can diagnose the build manually. +Now let's fix the issue and rebuild directly. .. literalinclude:: outputs/dev/develop-4.out :language: console -You'll notice here that spack rebuilt both ``scr`` and ``macsio``, as expected. +You'll notice that Spack rebuilt both ``scr`` and ``macsio``, as expected. -Taking advantage of iterative builds with spack requires cooperation from your build system. -When spack performs a rebuild on a development package, it reruns all the build stages for your package without cleaning the source and build directories to a pristine state. -If your build system can take advantage of the previously compiled object files then you'll end up with an iterative build. +Taking advantage of iterative builds with Spack requires cooperation from your build system. +When Spack performs a rebuild on a development package, it reruns all the build stages without cleaning the source and build directories to a pristine state. +If your build system can reuse previously compiled object files, you’ll benefit from an iterative build. -- If your package just uses make, you also should get iterative builds - for free when running ``spack develop``. -- If your package uses cmake with the typical ``cmake`` / ``build`` / - ``install`` build stages, you'll get iterative builds for free with - spack because cmake doesn’t modify the filetime on the - ``CMakeCache.txt`` file if your cmake flags haven't changed. -- If your package uses autoconf, then rerunning the typical - ``autoreconf`` stage typically modifies the filetime of - ``config.h``, which can trigger a cascade of rebuilding. +- If your package uses ``make``, you should get iterative builds automatically when using ``spack develop``. +- If your package uses CMake with the standard ``cmake`` / ``build`` / ``install`` stages, you'll also get iterative builds automatically. + This is because CMake doesn’t modify the file time of ``CMakeCache.txt`` unless your CMake flags change. +- If your package uses Autoconf, rerunning the typical ``autoreconf`` stage will usually modify the timestamp of ``config.h``, which may trigger a full rebuild. -Multiple packages can also be marked as develop. -If we were co-developing ``macsio``, we could run +Multiple packages can also be marked as development packages. +If we were co-developing ``macsio``, we could run: .. literalinclude:: outputs/dev/develop-5.out :language: console -Using development workflows also lets us ship our whole development process to another developer on the team. -They can simply take our spack.yaml, create a new environment, and use this to replicate our build process. -For example, we'll make another development environment here. +Using development workflows also allows us to share our full development setup with other team members. +They can simply use our ``spack.yaml`` to create a new environment and replicate the entire build process. +For example, here we create another development environment: .. literalinclude:: outputs/dev/otherdevel.out :language: console -Here, ``spack develop`` with no arguments will check out or download the source code and place it in the appropriate places. +In this case, running ``spack develop`` with no arguments will check out or download the source code and place it in the correct directories. -When we're done developing, we simply tell spack that it no longer needs to keep a development version of the package. +When we're done developing, we simply tell Spack that it no longer needs to keep a development version of the package: .. literalinclude:: outputs/dev/wrapup.out :language: console @@ -242,30 +198,33 @@ When we're done developing, we simply tell spack that it no longer needs to keep Workflow Summary ------------------- -Use the ``spack develop`` command with an environment to make a reproducible build environment for your development workflow. -Spack will set up all the dependencies for you and link all your packages together. -Within a development environment, ``spack install`` works similar to ``make`` in that it will check file times to rebuild the minimum number of spack packages necessary to reflect the changes to your build. +Use the ``spack develop`` command within an environment to create a reproducible build setup for your development workflow. +Spack will manage all dependencies and link your packages together automatically. +Within a development environment, ``spack install`` behaves similarly to ``make``: it checks file timestamps and rebuilds only the minimum set of Spack packages required to reflect your changes. ------------------- Optional: Tips and Tricks ------------------- -This section will cover some additional features that are useful additions to the core tutorial above. -Many of these items are very useful to specific projects and developers. -A list of the options for the ``spack develop`` can be viewed below: +This section covers additional features that complement the core tutorial above. +Many of these tips are especially useful for specific projects and developer workflows. + +A list of available options for the ``spack develop`` command can be viewed below: .. code-block:: console $ spack develop --help + Source Code Management ----------- +---------------------- -``spack develop`` allows users to manipulate the source code locations The default behavior is to let spack manage its location and cloning operations, but software developers often want more control over these. +The ``spack develop`` command allows users to control the location of source code. +By default, Spack manages source locations and handles cloning automatically, but software developers often prefer more control. -The source directory can be set with the ``--path`` argument when calling ``spack develop``. -If this directory already exists then ``spack develop`` will not attempt to fetch the code for you. -This allows developers to pre-clone the software or use preferred paths as they wish. +You can specify the source directory using the ``--path`` argument when invoking ``spack develop``. +If the specified directory already exists, Spack will not attempt to fetch the source code. +This allows developers to pre-clone repositories or use preferred directory paths as needed. .. code-block:: console @@ -277,14 +236,18 @@ This allows developers to pre-clone the software or use preferred paths as they $ spack concretize -f Navigation and the Build Environment ----------- +------------------------------------ + +Diving into the build environment was introduced earlier in the packaging section using the ``spack build-env scr -- bash`` command. +This is a helpful feature because it allows you to run commands inside the package’s build environment. + +In the packaging section, this was combined with ``spack cd`` to demonstrate a manual build process outside of Spack’s automated workflow. +This approach is especially useful in developer environments, providing a streamlined workflow for iterating on a single package without the overhead of the ``spack install`` command. + +The extra features of the ``spack install`` command are often unnecessary when rapidly iterating between building and testing a specific package. + +For example, the development workflow for modifying ``scr`` that we just covered can be simplified as follows: -Diving into the build environment was introduced previously in the packaging section with the ``spack build-env scr -- bash`` command. -This is a helpful function because it allows you to run commands inside the build environment. -In the packages section of the tutorial this was combined with ``spack cd`` to produce a manual build outside of Spack's automated Process. -This command is particularly useful in developer environments -- it allows developers a streamlined workflow when iterating on a single package without the overhead of the ``spack install`` command. -The additional features of the install command are unnecessary when tightly iterating between building and testing a particular package. -For example, the workflow modifying ``scr`` that we just went through can be simplified to: .. code-block:: console @@ -303,15 +266,17 @@ For example, the workflow modifying ``scr`` that we just went through can be sim $ make test $ exit -Working with the build environment and along with spack navigation features provides a nice way to iterate quickly and navigate through the hash heavy spack directory structures. + +Working directly within the build environment, combined with Spack's navigation features, provides a powerful way to iterate quickly and navigate through Spack’s hash-heavy directory structures. Combinatorics ------------- +------------- -The final note we will look at in this tutorial will be the power of combinatoric development builds. -There are many instances where developers want to see how a single set of changes affects multiple builds i.e. ``+cuda`` vs ``~cuda``, ``%gcc`` vs ``%clang``, ``build_type=Release`` vs ``build_type=Debug``, etc. +The final topic in this tutorial highlights the power of combinatorial development builds. +There are many cases where developers want to see how a single set of changes affects multiple build configurations—for example: +``+cuda`` vs ``~cuda``, ``%gcc`` vs ``%clang``, or ``build_type=Release`` vs ``build_type=Debug``. -Developers can achieve builds of both cases from a single ``spack install`` as long as the develop spec is generic enough to cover the packages' spec variations +Developers can build all of these configurations with a single ``spack install`` call, as long as the develop spec is generic enough to cover the spec variations of the packages. .. code-block:: console @@ -323,8 +288,17 @@ Developers can achieve builds of both cases from a single ``spack install`` as l # Inspect the graph for multiple dev_path= $ spack concretize -f -While we won't build out this example it illustrates how the ``dev_path`` for ``build_type=Release`` and ``build_type=Debug`` points to the same source code. +While we won't build out this example, it illustrates how the ``dev_path`` for both ``build_type=Release`` and ``build_type=Debug`` points to the same source code. + +If we want to do most of our incremental builds using the ``Release`` configuration and periodically check results using the ``Debug`` build, we can combine the workflow from the previous example: +First, enter the ``Release`` build environment using: + +``spack build-env scr build_type=Release -- bash`` + +Then, navigate to the build directory with: + +``spack cd -b scr build_type=Release`` + +Note that since there are two ``scr`` specs in the environment, we must disambiguate which one we want when using these commands. -Now if we want to do most of our incremental builds using the ``Release`` build and periodically check the results using the ``Debug`` build we can combine the workflow from the previous example: dive into the ``Release`` versions build environment using ``spack build-env scr build_type=Release -- bash`` and navigate with ``spack cd -b scr build_type=Release``. -Note that since there are two ``scr`` specs in the environment we must distinguish which one we want for these commands. -When we are ready to check our changes for the debug build we can exit out of the build environment subshell, rerun ``spack install`` to rebuild everything, and then inspect the debug build through our method of choice. +When we're ready to check our changes in the ``Debug`` build, we can exit the build environment subshell, rerun ``spack install`` to rebuild everything, and then inspect the debug build using our preferred method. From 222afe685e4fec32cf1e405ab2288f172fd66190 Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Mon, 2 Jun 2025 16:34:44 -0700 Subject: [PATCH 2/5] scripting changes --- tutorial_scripting.rst | 135 +++++++++++++++++++++-------------------- 1 file changed, 70 insertions(+), 65 deletions(-) diff --git a/tutorial_scripting.rst b/tutorial_scripting.rst index 26a7d2d96b..12b3e98dc0 100644 --- a/tutorial_scripting.rst +++ b/tutorial_scripting.rst @@ -11,49 +11,51 @@ Scripting with Spack ==================== -This tutorial introduces advanced Spack features related to scripting. -Specifically, we will show you how to write scripts using ``spack find`` and ``spack python``. -Earlier sections of the tutorial demonstrated using ``spack find`` to list and search installed packages. -The ``spack python`` command gives you access to all of Spack's `internal APIs `_, allowing you to write more complex queries, for example. +This tutorial introduces advanced Spack features related to scripting. +Specifically, we'll show how to write scripts using ``spack find`` and ``spack python``. -Since Spack has an extensive API, we'll only scratch the surface here. -We'll give you enough information to start writing your own scripts and to find what you need, with a little digging. +Earlier sections demonstrated using ``spack find`` to list and search installed packages. +The ``spack python`` command provides access to all of Spack's `internal APIs `_, allowing you to write more complex queries. + +Since Spack has an extensive API, we'll only scratch the surface here. +Our goal is to give you enough information to start writing your own scripts and to help you discover what you need—with a little digging. ----------------------------- Scripting with ``spack find`` ----------------------------- -So far, the output we've seen from ``spack find`` has been for human consumption. -But you can take advantage of some advanced options of the command to generate machine-readable output suitable for piping to a script. +So far, the output we've seen from ``spack find`` has been intended for human consumption. +However, the command also provides options for generating machine-readable output that can be piped into scripts. ^^^^^^^^^^^^^^^^^^^^^^^ ``spack find --format`` ^^^^^^^^^^^^^^^^^^^^^^^ -The main job of ``spack find`` is to show the user a bunch of concrete specs that correspond to installed packages. -By default, we display them with some default attributes, like the ``@version`` suffix you're used to seeing in the output. +The main purpose of ``spack find`` is to display information about concrete specs corresponding to installed packages. +By default, it shows these specs with a set of standard attributes, such as the familiar ``@version`` suffix. + +The ``--format`` argument allows you to customize the output string for each found package. +Format strings let you specify which *parts* of each spec you want to display. -The ``--format`` argument allows you to display the specs however you choose, using custom format strings. -Format strings let you specify the names of particular *parts* of the specs you want displayed. -Let's see the first option in action. +Let's see this option in action. -Suppose you only want to display the *name*, *version*, and first ten (10) characters of the *hash* for every package installed in your Spack instance. +Suppose you only want to display the *name*, *version*, and the first ten (10) characters of the *hash* for every package installed in your Spack instance. You can generate that output with the following command: .. literalinclude:: outputs/scripting/find-format.out :language: console :emphasize-lines: 1 -Note that ``name``, ``version``, and ``hash`` are attributes of Spack's internal ``Spec`` object and enclosing them in braces ensures they are output according to your format string. +Note that ``name``, ``version``, and ``hash`` are attributes of Spack's internal ``Spec`` object, and enclosing them in braces ensures they are rendered according to your format string. -Using ``spack find --format`` allows you to retrieve just the information you need to do things like pipe the output to typical UNIX command-line tools like ``sort`` or ``uniq``. +Using ``spack find --format`` allows you to retrieve only the information you need—making it easy to pipe the output into standard UNIX command-line tools like ``sort`` or ``uniq``. ^^^^^^^^^^^^^^^^^^^^^ ``spack find --json`` ^^^^^^^^^^^^^^^^^^^^^ -Alternatively, you can get a serialized version of Spec objects in the `JSON` format using the ``--json`` option. -For example, you can get attributes for all installations of ``zlib-ng`` by entering: +Alternatively, you can get a serialized version of ``Spec`` objects in `JSON` format using the ``--json`` option. +For example, to retrieve attributes for all installations of ``zlib-ng``, you can run: .. literalinclude:: outputs/scripting/find-json.out :language: console @@ -64,51 +66,51 @@ You can pipe its output to JSON filtering tools like ``jq`` to extract just the Check out the `basic usage docs `_ for more examples. - ---------------------------------------- Introducing the ``spack python`` command ---------------------------------------- What if we need to perform more advanced queries? -Spack provides the ``spack python`` command to launch a python interpreter with Spack's python modules available to import. -It uses the underlying python for the rest of its commands. -So you can write scripts to: +Spack provides the ``spack python`` command, which launches a Python interpreter with Spack's modules available for import. +This interpreter uses the same underlying Python environment that Spack uses for its other commands. + +Using this interface, you can write scripts to: -- run Spack commands; -- explore abstract and concretized specs; and +- run Spack commands; +- explore abstract and concretized specs; and - directly access other internal components of Spack. -Let's launch a Spack-aware python interpreter by entering: +Let's launch a Spack-aware Python interpreter by entering: .. literalinclude:: outputs/scripting/spack-python-1.out :language: console :emphasize-lines: 1,5 -Since we are in a python interpreter, use ``exit()`` to end the session and return to the terminal. +Since we are in a Python interpreter, use ``exit()`` to end the session and return to the terminal. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Accessing the ``Spec`` object ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Now let's take a look at the internal representation of the Spack ``Spec``. -As you already know, specs can be either *abstract* or *concrete*. -The specs you've seen in ``package.py`` files (e.g., in the ``install()`` method) have been *concrete*, or fully specified. -The specs you've typed on the command line have been *abstract*. -Understanding the differences between the two types is key to using Spack's internal API. +Now let's take a look at the internal representation of the Spack ``Spec``. +As you may already know, specs can be either *abstract* or *concrete*. +The specs you've seen in ``package.py`` files (e.g., in the ``install()`` method) have been *concrete*—fully specified with resolved dependencies, compilers, and variants. +In contrast, the specs you've typed on the command line have been *abstract*. -Let's open another python interpreter with ``spack python``, instantiate the ``zlib`` spec, and check a few properties of an abstract spec: +Understanding the difference between these two forms is key to effectively using Spack's internal API. + +Let's open another Python interpreter using ``spack python``, instantiate the ``zlib`` spec, and inspect a few properties of an abstract spec: .. literalinclude:: outputs/scripting/spack-python-abstract.out :language: console :emphasize-lines: 1-3,5,11,13 -Notice that there are ``Spec`` properties and methods that are not accessible to abstract specs; specifically: +Notice that certain ``Spec`` properties and methods are not accessible on abstract specs: -- an exception -- ``SpecError`` -- is raised if we try to access its - ``version``; -- there are no associated ``versions``; and -- the spec's operating system is ``None``. +- An exception—``SpecError``—is raised if you try to access the ``version``. +- There are no associated ``versions``. +- The spec's operating system is ``None``. Now, without exiting the interpreter, let's concretize the spec and try again: @@ -119,10 +121,10 @@ Now, without exiting the interpreter, let's concretize the spec and try again: Notice that the concretized spec now: - has a ``version``; -- has a single entry in its ``versions`` list; and -- the operating system is now ``ubuntu22.04``. +- includes a single entry in its ``versions`` list; and +- has ``ubuntu22.04`` as its operating system. -It is not necessary to store the intermediate abstract spec -- you can use the ``.concretized()`` method as shorthand: +It's not necessary to store the intermediate abstract spec—you can use the ``.concretized()`` method as a shorthand: .. literalinclude:: outputs/scripting/spack-python-sans-intermediate.out :language: console @@ -132,10 +134,11 @@ It is not necessary to store the intermediate abstract spec -- you can use the ` Querying the Spack database ^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Even more powerful queries are available when we look at the information stored in the Spack database. -The ``Database`` object in Spack is in the ``spack.store.STORE.db`` variable. -We'll interact with it mainly through the ``query()`` method. -Let's see the documentation available for ``query()`` using python's built-in ``help()`` function: +More powerful queries become available when interacting with Spack's installation database. +The ``Database`` object is accessible via the ``spack.store.STORE.db`` variable. +We'll primarily interact with it through the ``query()`` method. +Let's view the documentation for ``query()`` using Python's built-in ``help()`` function: + .. literalinclude:: outputs/scripting/spack-python-db-query-help.out :language: console @@ -143,13 +146,14 @@ Let's see the documentation available for ``query()`` using python's built-in `` We will primarily make use of the ``query_spec`` argument. -Recall that queries using the ``spack find`` command are limited to queries of attributes with matching values, not values they do *not* have. -In other words, we cannot use the ``spack find`` command for all packages that *do not* satisfy a certain criterion. +Recall that queries using the ``spack find`` command are primarily intended to *match* packages based on specified criteria. +However, it's more difficult to use ``spack find`` for queries that involve *excluding* packages based on complex logic (e.g., “does *not* depend on X”). + +Using the Python interface, we *can* write these more advanced queries. +For example, let's find all packages that were compiled with ``gcc`` but do **not** depend on ``mpich``. + +We'll use ``spack.cmd.display_specs`` to print the results, replicating the display behavior of the ``spack find`` command: -We *can* use the python interface to write these types of queries. -For example, let's find all packages that were compiled with ``gcc`` but do not depend on ``mpich``. -We can do this by using custom python code and Spack database queries. -We will use the ``spack.cmd.display_specs`` for output to achieve the same printing functionality as the ``spack find`` command: .. literalinclude:: outputs/scripting/spack-python-db-query-exclude.out :language: console @@ -169,15 +173,15 @@ before generalizing the functionality for reuse. Using scripts ^^^^^^^^^^^^^ -Now let's parameterize our script to accept arguments on the command line. -With a few generalizations to use the include and exclude specs as arguments, we can create a powerful, general-purpose query script. +Now let's parameterize our script to accept command-line arguments. +With a few generalizations, we can use the include and exclude specs as arguments to create a powerful, general-purpose query script. -Open a file called ``find_exclude.py`` in your preferred editor and add the following code: +Open a file named ``find_exclude.py`` in your preferred editor and add the following code: .. literalinclude:: outputs/scripting/0.find_exclude.py.example :language: python -Notice we added importing and using the system package (``sys``) to access the first and second command line arguments. +Notice that we've imported the ``sys`` module and used it to access the first and second command-line arguments. Now we can run our new script by entering the following: @@ -191,12 +195,12 @@ This is *great* for us, as long as we remember to use Spack's ``python`` command Using the ``spack-python`` executable ------------------------------------- -What if we want to make our script available for others to use without the hassle of having to remember to use ``spack python``? +What if we want to make our script available for others to use—without requiring them to remember to run it with ``spack python``? -We can take advantage of the shebang line typically added as the first line of python executable files. -But there is a catch, as we will soon see. +We can take advantage of a shebang line, typically included as the first line in executable Python scripts. +However, there's a catch, as we'll see shortly. -Open the ``find_exclude.py`` script we created above in your preferred editor and add the shebang line with ``spack python`` as the arguments to ``env``: +Open the ``find_exclude.py`` script you created earlier in your preferred editor, and add a shebang line that invokes ``spack python`` using ``env``: .. literalinclude:: outputs/scripting/1.find_exclude.py.example :language: python @@ -208,25 +212,26 @@ Then exit our editor and add execute permissions to the script before running it :language: console :emphasize-lines: 1-2 -If you are lucky, it worked on your system, but there is no guarantee. -Some systems only support a single argument on the shebang line (see `here `_). ``spack-python``, which is a wrapper script for ``spack python``, solves this issue. +If you're lucky, the script worked on your system—but there's no guarantee. +Some systems only support a single argument on the shebang line (see `here `_). +``spack-python``, a wrapper script for ``spack python``, solves this limitation. -Bring up the file in your editor again and change the ``env`` argument to ``spack-python`` as follows: +Open the script in your editor again and change the ``env`` argument to ``spack-python`` as shown below: .. literalinclude:: outputs/scripting/2.find_exclude.py.example :language: python :emphasize-lines: 1 -Exit your editor and let's run the script again: +Exit your editor, and let's run the script again: .. literalinclude:: outputs/scripting/find-exclude-3.out :language: console :emphasize-lines: 1 -Congratulations! It will now work on any system with Spack installed. +Congratulations! It will now work on any system with Spack installed. -You now have the basic tools to create your own custom Spack queries and prototype ideas. -We hope one day you'll contribute them back to Spack. +You now have the basic tools to write your own custom Spack queries and prototype new ideas. +We hope you'll consider contributing them back to Spack in the future. .. LocalWords: LLC Spack's APIs hdf zlib literalinclude json uniq jq .. LocalWords: docs concretized REPL API SpecError spec's py ubuntu From 0b881eb9ddd3d791b2adb945391545b36e9cb74c Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Mon, 2 Jun 2025 16:45:59 -0700 Subject: [PATCH 3/5] stacks --- tutorial_stacks.rst | 316 +++++++++++++++++++++++--------------------- 1 file changed, 169 insertions(+), 147 deletions(-) diff --git a/tutorial_stacks.rst b/tutorial_stacks.rst index 1f2516a134..9ddf3aedc5 100644 --- a/tutorial_stacks.rst +++ b/tutorial_stacks.rst @@ -11,18 +11,20 @@ Stacks Tutorial =============== -So far, we've talked about Spack environments in the context of a unified user environment. -But environments in Spack have much broader capabilities. -In this tutorial we will consider how to use Spack environments to manage large deployments of software. +So far, we've discussed Spack environments in the context of unified user workflows. +However, Spack environments have much broader capabilities. -What usually differs between a typical environment for a single user, and an environment used to manage large deployments, is that in the latter case we often have a set of packages we want to install across a wide range of MPIs, LAPACKs or compilers. +In this tutorial, we'll explore how to use Spack environments to manage large software deployments. -In the following we'll mimic the creation of a software stack built onto a cross-product of different LAPACK and MPI libraries, with a compiler that is more recent than the one provided by the host system. +The main difference between a typical environment for a single user and one used for large deployments is scope: +in the latter case, we often need to install a set of packages across a wide range of MPI implementations, LAPACK libraries, or compilers. -In the first part we'll focus on how to properly configure and install the software we want. -We'll learn how to pin certain requirements, and how to write a cross product of specs in a compact, and expressive, way. +In the following sections, we'll demonstrate how to create a software stack built as a cross-product of different LAPACK and MPI libraries, using a compiler that is newer than the one provided by the host system. -Then we'll consider how the software we install might be consumed by our users, and see the two main mechanisms that Spack provides for that: views and module files. +The first part focuses on how to properly configure and install the desired software. +We'll learn how to pin certain requirements and how to express a cross-product of specs in a compact and flexible way. + +Then, we'll consider how the installed software might be consumed by users, and examine the two main mechanisms that Spack provides for that: **views** and **module files**. .. note:: @@ -37,27 +39,27 @@ Then we'll consider how the software we install might be consumed by our users, Setup the compiler ------------------ -The first step to build our stack is to setup the compiler we want to use later. -This is, currently, an iterative process that can be done in two ways: +The first step in building our stack is to set up the compiler we want to use later. +This is currently an iterative process that can be done in one of two ways: - 1. Install the compiler first, then register it in the environment - 2. Use a second environment just for the compiler + 1. Install the compiler first, then register it in the environment + 2. Use a separate environment dedicated to the compiler -In the following we'll use the first approach. -For people interested, an example of the latter approach can be found `at this link `_. +In the following, we'll use the first approach. +For those interested, an example of the second approach can be found `at this link `_. Let's start by creating an environment in a directory of our choice: .. literalinclude:: outputs/stacks/setup-0.out :language: console -Now we can add from the command line a new compiler. -We'll also disable the generation of views for the time being, as we'll come back to this topic later in the tutorial: +Now we can add a new compiler from the command line. +We'll also disable view generation for now, as we'll return to this topic later in the tutorial: .. literalinclude:: outputs/stacks/setup-1.out :language: console -What you should see on screen now is the following ``spack.yaml`` file: +You should now see the following ``spack.yaml`` file on screen: .. literalinclude:: outputs/stacks/examples/0.spack.stack.yaml :language: yaml @@ -68,7 +70,7 @@ The next step is to concretize and install our compiler: .. literalinclude:: outputs/stacks/setup-2.out :language: console -Finally, let's register it as a new compiler in the environment: +Finally, let's register the new compiler in the environment: .. literalinclude:: outputs/stacks/compiler-find-0.out :language: console @@ -78,35 +80,35 @@ The ``spack location -i`` command returns the installation prefix for the spec b .. literalinclude:: outputs/stacks/compiler-find-1.out :language: console -This might be useful in general when scripting Spack commands, as the example above shows. -Listing the compilers now shows the presence of ``gcc@12.3.0``: +This is generally useful when scripting Spack commands, as the example above shows. +Listing the compilers now confirms the presence of ``gcc@12.3.0``: .. literalinclude:: outputs/stacks/compiler-list-0.out :language: console -The manifest file at this point looks like: +At this point, the manifest file looks like this: .. literalinclude:: outputs/stacks/examples/1.spack.stack.yaml :language: yaml -We are ready to build more software with our newly installed GCC! +We are now ready to build more software with our newly installed GCC! ------------------------ Install a software stack ------------------------ -Now that we have a compiler ready, the next objective is to build software with it. -We'll start by trying to add different versions of ``netlib-scalapack``, linked against different MPI implementations: +Now that we have a compiler ready, the next objective is to build software with it. +We'll start by adding different versions of ``netlib-scalapack``, each linked against a different MPI implementation: .. literalinclude:: outputs/stacks/unify-0.out :language: console -If we try to concretize the environment, we'll get an error: +If we try to concretize the environment at this point, we'll encounter an error: .. literalinclude:: outputs/stacks/unify-1.out :language: console -The error message is quite verbose, and admittedly complicated, but at the end it gives a useful hint: +The error message is quite verbose—and admittedly a bit complex—but it ends with a useful hint: .. code-block:: @@ -118,73 +120,74 @@ Let's see what that means. Tuning concretizer options for a stack ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Whenever we concretize an environment with more than one root spec, we can configure Spack to be more or less strict with duplicate nodes in the sub-DAG obtained by following link and run edges starting from the roots. -We usually call this sub-DAG the *root unification set*. +Whenever we concretize an environment with more than one root spec, we can configure Spack to be more or less strict about allowing duplicate nodes in the sub-DAG formed by following link and run dependencies from the roots. +This subgraph is commonly referred to as the *root unification set*. -A diagram might help to better visualize the concept: +A diagram may help to better visualize the concept: .. image:: _static/images/stacks-unify.svg -The image above represents the current environment, with our three root specs highlighted by a thicker dashed line. -Any node that could be reached following a link or run edge is part of the root unification set. -Pure build dependencies might fall outside of it. +The image above represents our current environment, with the three root specs highlighted using a thicker dashed outline. +Any node that can be reached via a link or run dependency from a root spec is part of the root unification set. +Pure build dependencies may fall outside of it. -The config option determining which nodes are allowed to be in the root unification set is ``concretizer:unify``. -Let's check its value: +The configuration option that controls unification behavior is ``concretizer:unify``. +Let's check its current value: .. literalinclude:: outputs/stacks/unify-2.out :language: console -``concretizer:unify:true`` means that only a single configuration for each package can be present. -This value is good for single project environments, since it ensures we can construct a view of all the software, with the usual structure expected on a Unix-ish system, and without risks of collisions between installations. +Setting ``concretizer:unify: true`` means that only one configuration of each package is allowed in the environment. +This setting is ideal for single-project environments, since it ensures a unified view of all installed software, resembling a traditional Unix filesystem layout, without the risk of collisions between installations. -Clearly, we can't respect this requirement, since our roots already contain two different configurations of ``netlib-scalapack``. -Let's set the value to ``false``, and try to re-concretize: +However, in our case, this strict unification is not feasible—our root specs already require two different configurations of ``netlib-scalapack``. +Let's update the setting to ``false`` and try to re-concretize: .. literalinclude:: outputs/stacks/unify-3.out :language: console -This time concretization succeeded. -Setting ``concretizer:unify:false`` is effectively concretizing each root spec on its own, and then merging the results into the environment. -This allows us to have the duplicates we need. +This time, concretization succeeds. +Setting ``concretizer:unify: false`` effectively tells Spack to concretize each root spec independently, then merge the results into the environment. +This allows us to retain multiple versions or configurations of the same package when necessary. .. note:: - If the environment is expected to have only a few duplicate nodes, then there's another value we might consider: + If the environment is expected to contain only a few duplicate nodes, there's another unification strategy worth considering: .. code-block:: console $ spack config add concretizer:unify:when_possible -With this option Spack will try to unify the environment in an eager way, solving it in multiple rounds. -The concretization at round ``n`` will contain all the specs that could not be unified at round ``n-1``, and will consider all the specs from previous rounds for reuse. + With this option, Spack will attempt to unify the environment eagerly by solving it in multiple rounds. + The concretization at round ``n`` includes all specs that could not be unified at round ``n-1``, and considers previous rounds' results for reuse. + ^^^^^^^^^^^^^ Spec matrices ^^^^^^^^^^^^^ -Let's expand our stack further and consider also linking against different LAPACK providers. +Let's expand our stack further to also link against different LAPACK providers. We could, of course, add new specs explicitly: .. literalinclude:: outputs/stacks/unify-4.out :language: console -This way of proceeding, though, will become very tedious as soon as more software is requested. -The best way to express a cross-product like this in Spack is instead through a matrix: +However, this approach becomes tedious as soon as additional software is required. +The best way to express a cross-product like this in Spack is by using a **matrix**: .. literalinclude:: outputs/stacks/examples/2.spack.stack.yaml :language: yaml :emphasize-lines: 8-12 -Matrices will expand to the cross-product of their rows, so this matrix: +Matrices expand to the cross-product of their rows. So the following matrix: .. code-block:: yaml - matrix: - - ["netlib-scalapack"] - - ["^openmpi", "^mpich"] - - ["^openblas", "^netlib-lapack"] - - ["%gcc@12"] + - ["netlib-scalapack"] + - ["^openmpi", "^mpich"] + - ["^openblas", "^netlib-lapack"] + - ["%gcc@12"] is equivalent to this list of specs: @@ -195,54 +198,57 @@ is equivalent to this list of specs: - "netlib-scalapack %gcc@12 ^netlib-lapack ^openmpi" - "netlib-scalapack %gcc@12 ^netlib-lapack ^mpich" -We are now ready to concretize and install the environment: +We're now ready to concretize and install the environment: .. literalinclude:: outputs/stacks/concretize-0.out :language: console -Let's double check which specs we have installed so far: +Let's double-check which specs have been installed so far: .. literalinclude:: outputs/stacks/concretize-01.out :language: console -As we can see we have our four variations of ``netlib-scalapack`` installed. +As we can see, all four variations of ``netlib-scalapack`` have been successfully installed. ^^^^^^^^^^^^^^^^^^^^ Reusable definitions ^^^^^^^^^^^^^^^^^^^^ -So far, we have seen how we can use spec matrices to generate cross-product specs from rows containing a list of constraints. -A common situation you will encounter with large deployments is the necessity to add multiple matrices to the list of specs, that possibly share some of those rows. +So far, we've seen how spec matrices can generate cross-product specs from rows containing lists of constraints. +In large deployments, it's common to include multiple matrices in the spec list—often sharing some of the same rows. + +To reduce duplication in the manifest file and lower the maintenance burden, Spack allows you to *define* reusable lists of constraints under the ``definitions`` attribute, and expand them later wherever needed. -To reduce the amount of duplication needed in the manifest file, and thus the maintenance burden for people maintaining it, Spack allows to *define* lists of constraints under the ``definitions`` attribute, and expand them later when needed. -Let's rewrite our manifest in that sense: +Let's rewrite our manifest using this feature: .. literalinclude:: outputs/stacks/examples/3.spack.stack.yaml :language: yaml :emphasize-lines: 6-10,14-18 -And check that re-concretizing won't change the environment: +Next, let's verify that re-concretizing the environment results in no changes: .. literalinclude:: outputs/stacks/concretize-1.out :language: console -Now we can use those definitions to add e.g. serial packages built against the LAPACK libraries. -Let's try to do that by using ``py-scypy`` as an example: +Now we can use those definitions to add, for example, serial packages built against the LAPACK libraries. +Let's demonstrate this using ``py-scipy`` as an example. -Another ability that is often useful, is that of excluding specific entries from a cross-product matrix. -We can do that with the ``exclude`` keyword, in the same item as the ``matrix``. -Let's try to remove ``py-scipy ^netlib-lapack`` from our matrix: +Another useful feature is the ability to exclude specific entries from a cross-product matrix. +This can be done using the ``exclude`` keyword within the same item as the ``matrix``. + +Let's exclude ``py-scipy ^netlib-lapack`` from the matrix: .. literalinclude:: outputs/stacks/examples/4bis.spack.stack.yaml :language: yaml :emphasize-lines: 11,20-25 -Let's concretize the environment and install the specs once again: +Let's concretize the environment and install the specs again: .. literalinclude:: outputs/stacks/concretize-3.out :language: console -At this point the environment contains only ``py-scipy ^openblas``. Let's verify it: +At this point, the environment should contain only ``py-scipy ^openblas``. +Let's verify that: .. literalinclude:: outputs/stacks/concretize-4.out :language: console @@ -251,123 +257,133 @@ At this point the environment contains only ``py-scipy ^openblas``. Let's verify Conditional definitions ^^^^^^^^^^^^^^^^^^^^^^^ -Spec list definitions can also be conditioned on a ``when`` clause. -The ``when`` clause is a python conditional that is evaluated in a restricted environment. -The variables available in ``when`` clauses are: - -================= =========== -variable name value -================= =========== -``platform`` The spack platform name for this machine -``os`` The default spack os name and version string for this machine -``target`` The default spack target string for this machine -``architecture`` The default spack architecture string platform-os-target for this machine -``arch`` Alias for ``architecture`` -``env`` A dictionary representing the users environment variables -``re`` The python ``re`` module for regex -``hostname`` The hostname of this node -================= =========== - -Let's say we only want to limit to just use ``mpich``, unless the ``SPACK_STACK_USE_OPENMPI`` environment variable is set. -To do so we could write the following ``spack.yaml``: +Spec list definitions can also be conditioned using a ``when`` clause. +The ``when`` clause is a Python conditional that is evaluated in a restricted context. +The following variables are available for use in ``when`` clauses: + +================= ============================================ +Variable Name Value +================= ============================================ +``platform`` The Spack platform name for this machine +``os`` The default Spack OS name and version string +``target`` The default Spack target string +``architecture`` The default Spack architecture string (platform-os-target) +``arch`` Alias for ``architecture`` +``env`` A dictionary representing the user's environment variables +``re`` The Python ``re`` module for regular expressions +``hostname`` The hostname of the current node +================= ============================================ + +Let's say we want to restrict the MPI provider to just ``mpich``, unless the ``SPACK_STACK_USE_OPENMPI`` environment variable is set. +To accomplish this, we could write the following ``spack.yaml``: .. literalinclude:: outputs/stacks/examples/5.spack.stack.yaml :language: yaml :emphasize-lines: 7-9 -Different definitions of lists with the same name are concatenated, so we can define our MPI list in one place unconditionally, and then conditionally append one or more values to it. +Different definitions for the same list name are concatenated. +This allows you to define the base list unconditionally and then append additional values conditionally using separate ``when`` blocks. -Let's first check what happens when we concretize and don't set any environment variable: +Let's first see what happens when we concretize without setting the environment variable: .. literalinclude:: outputs/stacks/concretize-5.out :language: console -As we expected now we are only using ``mpich`` as an MPI provider. -To get ``openmpi`` back we just need to set the appropriate environment variable: +As expected, only ``mpich`` is used as the MPI provider. +To include ``openmpi``, we simply set the appropriate environment variable: .. literalinclude:: outputs/stacks/concretize-6.out :language: console -There is no need to install this time, since all the specs were still in the store. +There's no need to reinstall in this case, since all the specs are already present in the store. + ^^^^^^^^^^^^^^^^^^^^^ Other useful features ^^^^^^^^^^^^^^^^^^^^^ -Sometimes it might be useful to create a local source mirror for the specs installed in an environment. +Sometimes it can be helpful to create a local source mirror for the specs installed in an environment. If the environment is active, this is as simple as: .. code-block:: console $ spack mirror create --all -d ./stacks-mirror -This command fetches all the tarballs for the packages in the ``spack.lock`` file, and puts them in the directory passed as argument. -Later you can move this mirror to e.g. an air-gapped machine and: +This command fetches all the tarballs for the packages listed in the ``spack.lock`` file and places them in the specified directory. +Later, you can move this mirror to an air-gapped machine and run: .. code-block:: console $ spack mirror add -to be able to re-build the specs from sources. If instead you want to create a buildcache you can: +This allows you to rebuild the specs from source. +If instead you want to create a buildcache, you can do the following: .. code-block:: console $ spack gpg create $ spack buildcache push ./mirror -In that case, don't forget to set an appropriate value for the padding of the install tree, see `how to setup relocation `_ in our documentation. +In that case, don't forget to set an appropriate value for the padding of the install tree. +See the documentation on `how to set up relocation `_ for details. + +By default, Spack installs one package at a time, using the ``-j`` option where possible. +If you are installing a large environment and have access to a powerful build node, you might want to launch multiple builds in parallel to make better use of available resources. -By default, Spack installs one package at a time, using the ``-j`` option where it can. -If you are installing a large environment, and have at disposal a beefy build node, you might need to start more installations in parallel to make an optimal use of the resources. -This can be done by creating a ``depfile``, when the environment is active: +This can be done by generating a ``depfile`` while the environment is active: .. code-block:: console $ spack env depfile -o Makefile -The result is a makefile that starts multiple Spack instances, and the resources are shared through the GNU jobserver. -More information of this feature can be found `in our documentation `_. -This might cut down your build time by a fair amount, if you build frequently from sources. +This generates a Makefile that starts multiple Spack instances, sharing resources via the GNU jobserver. +More details on this feature can be found in the `Spack documentation `_. + +Using this approach can significantly reduce build time, especially if you frequently build from source. ----------------------------------- Make the software stack easy to use ----------------------------------- -Now that the software stack has been installed, we need to focus on how it can be used by our customers. -We'll first see how we can configure views to project a subset of the specs we installed onto a filesystem folder with the usual Unix structure. -Then we'll have a similar discussion for module files. -Which of the two approaches is better depends strongly on the use case at hand. +Now that the software stack has been installed, we need to focus on how it can be used by end users. +We'll first look at how to configure views to project a subset of installed specs onto a directory structure that resembles a typical Unix filesystem layout. +Then we'll discuss an alternative approach using module files. +Which of these methods is more appropriate depends heavily on the specific use case. ^^^^^^^^^^^^^^^^ View descriptors ^^^^^^^^^^^^^^^^ -At the beginning, we configured Spack not to create a view for this stack because simple views won't work with stacks. -We've been concretizing multiple packages of the same name, and they would conflict if linked into the same view. +At the beginning of this tutorial, we configured Spack not to create a view for this stack. +That's because simple views won't work well with software stacks: we've been concretizing multiple packages with the same name, and they would conflict if linked into a single view. + +Instead, we can create *multiple views* using view descriptors. +This allows us to control exactly which packages are included in a view—and how they're laid out. -What we can do is create *multiple views*, using view descriptors. -This would allows us to define which packages are linked into the view, and how. -Let's edit our ``spack.yaml`` file again. +Let's edit our ``spack.yaml`` file again: .. literalinclude:: outputs/stacks/examples/6.spack.stack.yaml :language: yaml :emphasize-lines: 44-54 -In the configuration above we created two views, named ``default`` and ``full``. -The ``default`` view consists of all the packages that are compiled with ``gcc@12``, but do not depend on either ``mpich`` or ``netlib-lapack``. -As we can see, we can both *include* and *exclude* specs using constrains. +In the configuration above, we define two views: ``default`` and ``full``. + +The ``default`` view includes all packages built with ``gcc@12``, but excludes those that depend on ``mpich`` or ``netlib-lapack``. +As shown, we can use both *include* and *exclude* spec constraints in our view descriptors. -The ``full`` view contains a more complex projection, so to put each spec into an appropriate subdirectory, according to the first constraint that the spec matches. ``all`` is the default projection, and has always the lowest priority, independent of the order in which it appears. -To avoid confusion, we advise to always keep it last in projections. +The ``full`` view uses a more complex projection layout. +This places each spec in a subdirectory according to the first matching constraint. +The ``all`` pattern acts as a fallback and always has the lowest priority, regardless of where it appears. +To avoid confusion, we recommend placing ``all`` last in the projection list. -Let's concretize to regenerate the views, and check their structure: +Let's re-concretize the environment to regenerate the views, and check their structure: .. literalinclude:: outputs/stacks/view-0.out :language: console -The view descriptor also contains a ``link`` key. -The default behavior, as we have seen, is to link all packages, including implicit link and run dependencies, into the view. -If we set the option to "roots", Spack links only the root packages into the view. +View descriptors also support a ``link`` key. +By default, Spack links all packages—including implicit link and run dependencies—into the view. +If we set this option to ``roots``, Spack links only the root packages into the view. .. literalinclude:: outputs/stacks/examples/7.spack.stack.yaml :language: yaml @@ -376,21 +392,22 @@ If we set the option to "roots", Spack links only the root packages into the vie .. literalinclude:: outputs/stacks/view-1.out :language: console -Now we see only the root libraries in the default view. -The rest are hidden, but are still available in the full view. -The complete documentation on view can be found `here `_. +Now, only the root libraries appear in the ``default`` view. +All other packages are excluded from the view but remain available in the ``full`` view. + +You can find the complete documentation on views in the Spack manual `here `_. ^^^^^^^^^^^^ Module files ^^^^^^^^^^^^ -Module files are another very popular way to use software on HPC systems. -In this section we'll show how to configure and generate a hierarchical module structure, suitable for ``lmod``. +Module files are a widely used mechanism for managing software environments on HPC systems. +In this section, we'll show how to configure and generate a hierarchical module structure suitable for use with ``lmod``. -A more in-depth tutorial, focused only on module files, can be found at :ref:`modules-tutorial`. -There we discuss the general architecture of module file generation in Spack and we highlight differences between ``environment-modules`` and ``lmod`` that won't be covered in this section. +A more detailed tutorial focused exclusively on module files is available at :ref:`modules-tutorial`. +That tutorial covers Spack's module file architecture in depth and compares ``environment-modules`` and ``lmod``—a distinction we won't address here. -So, let's start by adding ``lmod`` to the software installed with the system compiler: +Let's start by installing ``lmod`` using the system compiler: .. code-block:: console @@ -398,71 +415,76 @@ So, let's start by adding ``lmod`` to the software installed with the system com $ spack concretize $ spack install -Once that is done, let's add the ``module`` command to our shell like this: +Once installed, initialize the ``module`` command in your shell: .. code-block:: console $ . $(spack location -i lmod)/lmod/lmod/init/bash -If everything worked out correctly you should now have the module command available in you shell: +If everything worked correctly, the ``module`` command should now be available in your shell: .. literalinclude:: outputs/stacks/modules-1.out :language: console -The next step is to add some basic configuration to our ``spack.yaml`` to generate module files: +Next, let's add basic configuration to our ``spack.yaml`` file to enable module file generation: .. literalinclude:: outputs/stacks/examples/8.spack.stack.yaml :language: yaml :emphasize-lines: 45-54 -In these few lines of additional configuration we told Spack to generate ``lmod`` module files in a subdirectory named ``modules``, using a hierarchy comprising both ``lapack`` and ``mpi``. +In these lines, we instruct Spack to generate ``lmod`` module files in a subdirectory named ``modules``, using a hierarchy based on ``lapack`` and ``mpi``. -We can generate the module files and use them with the following commands: +To generate and use the module files, run: .. code-block:: console $ spack module lmod refresh -y $ module use $PWD/stacks/modules/linux-ubuntu22.04-x86_64/Core -Now we should be able to see the module files that have been generated: +You should now see the generated module files: .. literalinclude:: outputs/stacks/modules-2.out :language: console -The sets of modules is already usable, and the hierarchy already works. -For instance we can load the ``gcc`` compiler and check that we have ``gcc`` in out path and we have a lot more modules available - all the ones compiled with ``gcc@12.3.0``: +This basic setup is already functional. +For example, you can load the ``gcc`` compiler module and confirm that it adjusts your path and exposes additional modules compiled with ``gcc@12.3.0``: .. literalinclude:: outputs/stacks/modules-3.out :language: console -There are a few issues though. -For once, we have a lot of modules generated from dependencies of ``gcc`` that are cluttering the view, and won't likely be needed directly by users. -Then, module names contain hashes, which go against users being able to reuse the same script in similar, but not equal, environments. +However, there are still some issues: -Also, some of the modules might need to set custom environment variables, which are specific to the deployment aspects that don't enter the hash - for instance a policy at the deploying site. +- Many modules generated from compiler dependencies clutter the view and are not useful to end users. +- Module names include hashes, which makes them harder to reuse across similar environments. +- Some modules may need to set custom environment variables based on site-specific deployment policies—information that isn't encoded in the spec hash. -To address all these needs we can complicate out ``modules`` configuration a bit more: +To address these issues, we can enhance the ``modules`` configuration: .. literalinclude:: outputs/stacks/examples/9.spack.stack.yaml :language: yaml :emphasize-lines: 55-70 -Let's regenerate the modules once again: +Let's regenerate the module files with the updated configuration: .. literalinclude:: outputs/stacks/modules-4.out :language: console -Now we have a set of module files without hashes, with a correct hierarchy, and with all our custom modifications: +Now, the generated module files: + +- exclude hashes from names, +- follow a clean and correct hierarchy, +- and include our custom modifications. .. literalinclude:: outputs/stacks/modules-5.out :language: console -This concludes the quick tour of module file generation, and the tutorial on stacks. +This concludes our brief tour of module file generation—and the tutorial on software stacks. ------- Summary ------- -In this tutorial, we configured Spack to install a stack of software built on a cross-product of different MPI and LAPACK libraries. -We used the spec matrix syntax to express in a compact way the specs to be installed, and spec list definitions to reuse the same matrix rows in different places. -Then, we discussed how to make the software easy to use, leveraging either filesystem views or module files. +In this tutorial, we configured Spack to install a stack of software built as a cross-product of different MPI and LAPACK libraries. +We used spec matrix syntax to compactly define the specs to install, and leveraged spec list definitions to reuse matrix rows in multiple contexts. + +Finally, we discussed how to make the installed software accessible to users, using either filesystem views or module files—choosing the best approach depending on your use case. From 533001583632cbc32eb5fb5df62dc94b8c6cf094 Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Mon, 2 Jun 2025 17:16:31 -0700 Subject: [PATCH 4/5] environments --- tutorial_environments.rst | 622 +++++++++++++++++++------------------- 1 file changed, 312 insertions(+), 310 deletions(-) diff --git a/tutorial_environments.rst b/tutorial_environments.rst index c48d20277b..a7dd191131 100644 --- a/tutorial_environments.rst +++ b/tutorial_environments.rst @@ -11,297 +11,293 @@ Environments Tutorial ===================== -We've covered how to install, remove, and list packages with Spack using the commands: +We've covered how to install, remove, and list packages in Spack using the following commands: * `spack install `_ to install packages; -* `spack uninstall `_ to remove them; and -* `spack find `_ to look at and query what is installed. +* `spack uninstall `_ to remove packages; and +* `spack find `_ to list and query installed packages. -.. Customizing Spack's installation with configuration files, like - `packages.yaml `_, was also discussed. +.. Customizing Spack's installation with configuration files such as + `packages.yaml `_ was also discussed. -This section of the tutorial introduces **Spack Environments**, which allow you to work with independent groups of packages separately, in a reproducible way. -In some ways, Spack environments are similar to *virtual environments* in other systems (e.g., `Python venv `_), but they are based around file formats (``spack.yaml`` and ``spack.lock``) that can be shared easily and re-used by others across systems. +This section of the tutorial introduces **Spack Environments**, which let you manage independent groups of packages in a reproducible way. +In some ways, Spack environments are similar to *virtual environments* in other systems (e.g., `Python venv `_), but they are built around file formats (``spack.yaml`` and ``spack.lock``) that can be easily shared and reused across systems. -Administering properly configured software involving lots of packages and/or varying configuration requirements (e.g., different implementations of ``mpi``) for multiple projects and efforts can be overwhelming. -Spack environments allow you to readily: +Managing software that involves many packages and varying configuration requirements (e.g., different ``mpi`` implementations) for multiple projects can quickly become overwhelming. +Spack environments help by enabling you to: -* establish standard software requirements for your project(s); -* set up run environments for users; -* support your usual development environment(s); -* set up packages for CI/CD; -* reproduce builds (approximately or exactly) on other machines; and +* define and standardize software requirements for your projects; +* configure user runtime environments; +* support consistent development setups; +* define packages for CI/CD workflows; +* reproduce builds—approximately or exactly—on other systems; and * much more. -This tutorial introduces the basics of creating and using environments, then explains how to expand, configure, and build software in them. -We will start with the command line interface, then cover editing key environment file directly. -We will describe the difference between Spack-managed and independent environments, then finish with a section on reproducible builds. +This tutorial introduces the basics of creating and using environments, explains how to expand, configure, and build software within them, and distinguishes between Spack-managed and standalone environments. +We'll begin with the command-line interface, then cover editing key environment files directly, and conclude with guidance on reproducible builds. ------------------- Environment Basics ------------------- -Let's look at the output of ``spack find`` at this point in the tutorial. +Let's take a look at the output of ``spack find`` at this point in the tutorial: .. literalinclude:: outputs/environments/find-no-env-1.out :language: console +This is a complete—but cluttered—list of installed packages and their dependencies. +It includes packages built with both ``openmpi`` and ``mpich``, along with multiple variants of other packages such as ``hdf5`` and ``zlib-ng``. -This is a complete, but cluttered list of the installed packages and their dependencies. -It contains packages built with both ``openmpi`` and ``mpich``, as well as multiple variants of other packages, like ``hdf5`` and ``zlib-ng``. -The query mechanism we learned about with ``spack find`` can help, but it would be nice if we could start from a clean slate without losing what we've already installed. +While the ``spack find`` query mechanisms we've covered can help navigate this, it would be even more useful to start from a clean slate—without losing what we've already installed. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Creating and activating environments ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The ``spack env`` command can help. +The ``spack env`` command is used to manage environments. Let's create a new environment called ``myproject``: .. literalinclude:: outputs/environments/env-create-1.out :language: console +An environment is like a virtualized Spack instance that aggregates package installations for a specific project or purpose. +It has an associated *view*, which is a single directory where all packages from the environment are linked. -An environment is like a virtualized Spack instance that you can use to aggregate package installations for a project or other purpose. -It has an associated *view*, which is a single prefix where all packages from the environment are linked. - -You can see the environments we've created so far using the ``spack env list`` command: +You can list the environments you've created so far with the ``spack env list`` command: .. literalinclude:: outputs/environments/env-list-1.out :language: console - -Now let's **activate** our environment. -You can use ``spack env activate`` command: +Now, let's **activate** our environment. +Use the ``spack env activate`` command: .. literalinclude:: outputs/environments/env-activate-1.out :language: console .. note:: - If you use the ``-p`` option for ``spack env activate``, Spack - will prepend the environment name to the prompt. This is a handy - way to be reminded if and which environment you are in. -You can also use the shorter ``spacktivate`` alias for ``spack env activate``. + If you use the ``-p`` option with ``spack env activate``, Spack will prepend the environment name to your shell prompt. + This is a helpful reminder of which environment is currently active. + +You can also use the shorter alias ``spacktivate`` for ``spack env activate``. .. note:: - Alias behavior may vary depending on the shell interpreter used. In Bash, - use of aliases requires the shell option ``expand_aliases`` to be set. - This option is set by default, but it is not set in non-interactive shells, - e.g., when executing a Bash script. In a Bash script, ``expand_aliases`` - can be set with the command ``shopt -s expand_aliases``. + Alias behavior may vary depending on your shell. In Bash, alias use requires the ``expand_aliases`` option to be enabled. + This option is set by default in interactive shells, but not in non-interactive ones (e.g., when running a Bash script). + To enable it in a script, run: ``shopt -s expand_aliases``. -Once you activate an environment, ``spack find`` only shows what is in the current environment. -We just created this environment, so it does not contain any installed packages. +Once you activate an environment, ``spack find`` shows only the packages in that environment. +Since we just created this one, it does not contain any packages yet: .. literalinclude:: outputs/environments/find-env-1.out :language: console -The output from ``spack find`` is now *slightly* different. -It tells you that you're in the ``myproject`` environment, so there is no need to panic when you see that none of the previously installed packages are available. -It also states that there are **no** *root specs*. -We'll get back to what that means later. +The output from ``spack find`` is now *slightly* different: +It tells you that you're in the ``myproject`` environment—so don't panic if none of your previously installed packages are listed. +It also notes that there are **no** *root specs* (we'll explain that term shortly). -If you *only* want to check what environment you are in, you can use ``spack env status``: +If you only want to check which environment is currently active, use: .. literalinclude:: outputs/environments/env-status-1.out :language: console +To leave the environment, use ``spack env deactivate`` or its alias ``despacktivate``. -If you want to leave this environment, you can use ``spack env deactivate`` or the ``despacktivate`` alias for short. - -After deactivating, we can see everything installed in this Spack instance: +After deactivating, ``spack find`` will again show all packages installed in your Spack instance: .. literalinclude:: outputs/environments/env-status-2.out :language: console +Notice that we are no longer in an environment—and all your previously installed packages are still available. -Notice that we are no longer in an environment and all our packages are still installed. ^^^^^^^^^^^^^^^^^^^ Installing packages ^^^^^^^^^^^^^^^^^^^ -Now that we understand how creation and activation work, let's go back to ``myproject`` and *install* a couple of packages, specifically, ``tcl`` and ``trilinos``. +Now that we understand how environment creation and activation work, let's return to the ``myproject`` environment and *install* a couple of packages: ``tcl`` and ``trilinos``. -Try the usual install command first: +First, try the usual install command: .. literalinclude:: outputs/environments/env-fail-install-1.out :language: console -Environments are special in that you must *add* specs to them before installing. ``spack add`` allows us to do queue up several specs to be installed together. -Let's try it: +Environments are a bit different—before installing packages, you must first *add* them. +The ``spack add`` command queues up specs to be installed in the environment. Let's try it: .. literalinclude:: outputs/environments/env-add-1.out :language: console -Now, ``tcl`` and ``trilinos`` have been registered as **root specs** in this environment. -That is because we explicitly asked for them to be installed, so they are the **roots** of the combined graph of all packages we'll install. +Now, ``tcl`` and ``trilinos`` have been registered as **root specs** in this environment. +Root specs are the packages you explicitly request for installation—they serve as the **roots** of the environment's dependency graph. -Now, let's install: +Let's install them: .. literalinclude:: outputs/environments/env-install-1.out :language: console +We see that ``tcl`` and the dependencies of ``trilinos`` were already installed, while ``trilinos`` itself was newly built. +The environment's view was also updated to reflect the new installations. -We see that ``tcl`` and the dependencies of ``trilinos`` are already installed, and that ``trilinos`` was newly installed. -We also see that the environment's view was updated to include the new installations. - -Now confirm the contents of the environment using ``spack find``: +Now, confirm the environment contents using ``spack find``: .. literalinclude:: outputs/environments/find-env-2.out :language: console -We can see that the roots and all their dependencies have been installed. +As shown, the root specs and all their dependencies are now installed within the ``myproject`` environment. + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Creating an environment incrementally ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -As a short-hand, you can use the ``install --add`` flag to accomplish the same thing in one step: +As a shorthand, you can use the ``install --add`` flag to add and install specs in a single step: .. code-block:: console $ spack install --add tcl trilinos -This both adds the specs to the environment and installs them. +This command adds the specs to the environment and installs them immediately. -You can also add and install specs to an environment incrementally. For example: +You can also add and install specs incrementally, one at a time: .. code-block:: console $ spack install --add tcl $ spack install --add trilinos -If you create environments incrementally, Spack ensures that already installed roots are not re-concretized. -So, adding specs to an environment at a later point in time will not cause existing packages to rebuild. +When building environments incrementally, Spack ensures that already installed root specs are not re-concretized. +This means that adding new specs later will not cause previously installed packages to be rebuilt. -Do note however that incrementally creating an environment can give you different package versions from an environment created all at once. -We will cover this after we've discussed different concretization strategies. +However, incrementally adding and installing specs may result in different package versions than installing them all at once. +We'll revisit this topic when we discuss different concretization strategies. -Further, there are two other advantages of concretizing and installing an environment all at once: +Additionally, there are two key benefits to concretizing and installing an environment all at once: -* If you have a number of specs that can be installed together, - adding them first and installing them together enables them to - share dependencies and reduces total installation time. +* If several specs can share dependencies, installing them together can reduce total installation time. +* You can take advantage of Spack's `install-level build parallelism `_ to launch builds in parallel. -* You can launch all builds in parallel by taking advantage of Spack's `install-level build parallelism `_. ^^^^^^^^^^^^^^ Using packages ^^^^^^^^^^^^^^ -Environments provide a convenient way for using installed packages. -Running ``spack env activate`` gives you everything in the environment on your ``PATH``. -Otherwise, you would need to use `spack load `_ or `module load `_ for each package in order to set up the environment for the package (and its dependencies). +Environments provide a convenient way to use installed packages. +Activating an environment with ``spack env activate`` automatically places everything in the environment on your ``PATH``. + +Without environments, you would need to manually use `spack load `_ or `module load `_ for each package to set up the appropriate environment for the package and its dependencies. -When you install packages into an environment, they are, by default, linked into a single prefix, or *view*. -Activating the environment with ``spack env activate`` results in subdirectories from the view being added to ``PATH``, ``MANPATH``, ``CMAKE_PREFIX_PATH``, and other environment variables. -This makes the environment easier to use. +When you install packages into an environment, they are—by default—linked into a single directory prefix known as a *view*. +Activating the environment with ``spack env activate`` adds relevant subdirectories from this view to ``PATH``, ``MANPATH``, ``CMAKE_PREFIX_PATH``, and other environment variables, making the environment easier to use. Let's try it out. -We just installed ``tcl`` into our ``myproject`` environment. ``Tcl`` includes a shell-like application called ``tclsh``. -You can see the path to ``tclsh`` using ``which``: + +We just installed ``tcl`` in the ``myproject`` environment. +Tcl includes a shell-like executable called ``tclsh``. +You can check its location with ``which``: .. literalinclude:: outputs/environments/use-tcl-1.out :language: console +Notice that the path includes the name of our environment and a ``view`` subdirectory. -Notice its path includes the name of our environment *and* a ``view`` subdirectory. - -You can now run ``tclsh`` like you would any other program that is in your path: +You can now run ``tclsh`` like any other command available in your ``PATH``: .. code-block:: console - $ tclsh - % echo "hello world!" - hello world! - % exit + $ tclsh + % puts "hello world!" + hello world! + % exit ^^^^^^^^^^^^^^^^^^^^^ Uninstalling packages ^^^^^^^^^^^^^^^^^^^^^ -We can uninstall packages from an environment without affecting other environments. -This is possible since, while Spack shares common installations, environments only link to those installations. +Spack environments let you uninstall or remove packages *within* one environment without affecting other environments or globally installed packages. +This is because environments primarily manage which packages are part of their configuration and view. -Let's demonstrate this feature by creating another environment. -Suppose ``myproject`` requires ``trilinos`` but we have another project that has it installed but no longer requires it. +Let's demonstrate this by creating another environment. +Suppose ``myproject`` requires ``trilinos``, but we have another project that also installed it and no longer needs it. -Start by creating a ``myproject2`` environment with the installed packages ``scr`` and ``trilinos``. +Start by creating a ``myproject2`` environment and adding the installed packages ``scr`` and ``trilinos``: .. literalinclude:: outputs/environments/env-create-2.out :language: console +Now we have two environments: -Now we have two environments. -The ``myproject`` environment has ``tcl`` and ``trilinos`` while the ``myproject2`` environment has ``scr`` and ``trilinos``. +- ``myproject`` contains ``tcl`` and ``trilinos`` +- ``myproject2`` contains ``scr`` and ``trilinos`` -Now let's try to uninstall ``trilinos`` from ``myproject2`` and review the contents of the environment: +Let's try uninstalling ``trilinos`` from ``myproject2`` and review the environment contents: .. literalinclude:: outputs/environments/env-uninstall-1.out :language: console - -We can see that ``trilinos`` won't be uninstalled because it is still referenced in another environment managed by spack. -If we want to remove it from the roots list we need to use ``spack remove``: +We see that ``trilinos`` was not uninstalled because it is still referenced by another environment. +To remove it from the list of root specs in ``myproject2``, use ``spack remove``: .. literalinclude:: outputs/environments/env-remove-1.out :language: console -When the spec is first removed, we see that it is no longer a root but is still present in the installed specs. -Once we reconcretize, the vestigial spec is removed. -Now, it is no longer a root and will need to be re-added before being installed as part of this environment. +After removing it, you'll see that ``trilinos`` is no longer listed as a root spec—but it still appears in the environment until we re-concretize. +Once we do, the spec is fully removed. -We know ``trilinos`` is still needed for the ``myproject`` environment, so let's switch back to confirm that it is still installed in that environment. +To confirm ``trilinos`` is still available in the ``myproject`` environment, switch back: .. literalinclude:: outputs/environments/env-swap-1.out :language: console - -Phew! -We see that ``myproject`` still has ``trilinos`` as a root spec. -Spack uses reference counting to ensure that we don't remove ``trilinos`` when it is still needed by ``myproject``. +Phew! +``trilinos`` is still present as a root spec in ``myproject``. +Spack uses reference counting to ensure packages aren't uninstalled while they're still required elsewhere. .. note:: - Trilinos would only have been uninstalled by Spack if it were - no longer needed by any environments or their package dependencies. + Spack only uninstalls a package when it is no longer needed by any environment or their dependencies. + +You can also uninstall a package and remove it from the environment in one step with: + +.. code-block:: console -You can also uninstall a package and remove it from the environment in one go with ``spack uninstall --remove trilinos``. + $ spack uninstall --remove trilinos ----------------------- The ``spack.yaml`` file ----------------------- -An environment is more than just a list of root specs. -It includes *configuration* settings that affect the way Spack behaves when the environment is activated. -So far, ``myproject`` relies on configuration defaults that can be overridden. -Here we'll look at how to add specs and ensure all the packages depending on ``mpi`` build with ``mpich``. -We can customize the selection of the ``mpi`` provider using `concretization preferences `_ to change the behavior of the concretizer. +An environment is more than just a list of root specs. +It also includes *configuration* settings that affect Spack's behavior when the environment is active. + +So far, ``myproject`` has relied on configuration defaults, but these can be overridden. +For example, we may want to ensure that all packages depending on ``mpi`` use ``mpich`` instead of another provider. -Let's start by looking at the configuration of our environment using ``spack config get``: +We can customize provider selection using `concretization preferences `_. + +Let's start by examining the current environment configuration with: .. literalinclude:: outputs/environments/config-get-1.out :emphasize-lines: 8-13 -The output shows the special ``spack.yaml`` configuration file that Spack uses to store the environment configuration. +This output shows the ``spack.yaml`` configuration file used by Spack to store environment state. + +Key parts of this file include: -There are several important parts of this file: +* ``specs:`` — the list of root specs to install +* ``view:`` — controls whether the environment has a *view*; set to ``false`` to disable view generation +* ``concretizer:unify:`` — controls how specs in the environment are concretized -* ``specs:``: the list of specs to install -* ``view:``: this controls whether the environment has a *view*. You can - set it to ``false`` to disable view generation. -* ``concretizer:unify:``: This controls how the specs in the environment - are concretized. +The ``specs`` list should look familiar—these are the packages we've been managing with ``spack add``. -The ``specs`` list should look familiar; these are the specs we've been modifying with ``spack add``. +The ``concretizer:unify`` setting controls how the concretizer resolves versions: -``concretizer:unify:true``, the default, means that they are concretized *together*, so that there is only one version of each package in the environment. -Other options for ``unify`` are ``false`` and ``when_possible``. ``false`` means that the specs are concretized *independently*, so that there may be multiple versions of the same package in the environment. ``when_possible`` lies between those options. -In this case, Spack will unify as many packages in the environment, but will not fail if it cannot unify all of them. +- ``true`` (default): Spack tries to unify dependencies so that only one version of each package is used +- ``false``: specs are concretized independently, allowing multiple versions +- ``when_possible``: Spack unifies packages where it can but allows divergence if necessary +This setting helps determine whether all specs share the same dependency versions or not. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Editing environment configuration @@ -309,13 +305,13 @@ Editing environment configuration .. note:: - Before proceeding, make sure your ``EDITOR`` environment variable + Before proceeding, ensure that your ``EDITOR`` environment variable is set to the path of your preferred text editor. -Let's edit ``spack.yaml`` to *require* ``mpich`` as our ``mpi`` provider using ``spack config edit``. +Let's edit ``spack.yaml`` to *require* ``mpich`` as the ``mpi`` provider using the ``spack config edit`` command. -You should now have the above file open in your editor. -Change it to include the ``packages:mpi:require`` entry below: +This should open the file in your configured editor. +Modify it to include the ``packages:mpi:require`` entry as shown below: .. code-block:: yaml :emphasize-lines: 6-8 @@ -332,47 +328,50 @@ Change it to include the ``packages:mpi:require`` entry below: # add package specs to the `specs` list specs: [tcl, trilinos] - .. note:: - We introduce this here to show you how environment configuration - can affect concretization. Configuration options are covered in much - more detail in the :ref:`configuration tutorial `. - + We introduce this configuration to demonstrate how environment settings + can affect concretization. Configuration is covered in more detail + in the :ref:`configuration tutorial `. -We've only scratched the surface here by requiring a specific ``mpi`` provider for packages depending on ``mpi``. -There are many other customizations you can make to an environment. -Refer to the links at the end of this section for more information. +Here, we've only scratched the surface by requiring a specific ``mpi`` provider for packages that depend on ``mpi``. +Spack environments support many additional customizations—refer to the links at the end of this section for further exploration. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Re-concretizing the environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -You may need to re-install packages in the environment after making significant changes to the configuration, such as changing virtual providers. -This can be accomplished by forcing Spack to re-concretize the environment and re-install the specs. +You may need to re-install packages in an environment after making significant configuration changes—such as switching virtual providers. +This is done by forcing Spack to *re-concretize* the environment and re-install the affected specs. -For example, the packages installed in our ``myproject`` environment are now out of sync with our new configuration since we already installed part of the environment with ``openmpi``. -Suppose we want to install everything in ``myproject`` with ``mpich``. +For example, the packages currently installed in our ``myproject`` environment are now out of sync with the configuration, because we previously installed part of the environment with ``openmpi``. +Suppose we now want everything in ``myproject`` to use ``mpich`` instead. -Let's run ``spack concretize --force`` (or ``-f`` in short) to make Spack re-concretize all the environment's specs: +We can run: + +.. code-block:: console + + $ spack concretize --force + +(or use the shorthand ``-f``) to force Spack to re-concretize all specs in the environment: .. literalinclude:: outputs/environments/concretize-f-1.out :language: console +All specs are now concrete and ready to be installed with ``mpich`` as the selected MPI provider. -All the specs are now concrete **and** ready to be installed with ``mpich`` as the MPI implementation. +Re-concretization is also necessary when building environments *incrementally* with unification enabled. +By default, Spack avoids modifying already concretized specs when new ones are added. -Re-concretization is sometimes also necessary when creating an environment *incrementally* with unification enabled. -Spack makes sure that already concretized specs in the environment are not modified when adding something new. - -Adding and installing specs one by one leads to greedy concretization. -When you first install ``python`` in an environment, Spack will pick a recent version for it. -If you then add ``py-numpy``, it may be in conflict with the ``python`` version already installed, and fail to concretize: +Incrementally adding and installing specs leads to *greedy concretization*. +For example, if you install ``python`` first, Spack may choose the most recent version. +Later, if you add ``py-numpy``, it might conflict with the previously selected ``python`` version and fail to concretize: .. literalinclude:: outputs/environments/incremental-1.out :language: console -The solution is to re-concretize the environment as a whole, which causes ``python`` to downgrade to a version compatible with ``py-numpy``: +To fix this, re-concretize the entire environment. +Spack will then select compatible versions for all specs—for example, downgrading ``python`` if needed: .. literalinclude:: outputs/environments/incremental-2.out :language: console @@ -381,146 +380,165 @@ The solution is to re-concretize the environment as a whole, which causes ``pyth Building in environments ------------------------ -Activated environments allow you to invoke any programs installed in them as if they were installed on the system. -In this section we will take advantage of that feature. +Activated environments allow you to use any installed programs as if they were installed system-wide. +In this section, we'll take advantage of that feature. -Suppose you want to compile some MPI programs. -We have an MPI implementation installed in our ``myproject2`` environment, so ``mpicc`` is available in our path. +Suppose you want to compile an MPI program. +Since our ``myproject2`` environment includes an MPI implementation, ``mpicc`` is available on the ``PATH``. We can confirm this using ``which``: .. literalinclude:: outputs/environments/show-mpicc-1.out :language: console +As mentioned earlier, activating an environment sets several key environment variables. +These include ``PATH``, ``MANPATH``, and ``CMAKE_PREFIX_PATH``, which allow tools and compilers to easily locate the executables, headers, and libraries installed in the environment. -As mentioned before, activating the environment sets a number of environment variables. -That includes variables like ``PATH``, ``MANPATH``, and ``CMAKE_PREFIX_PATH``, which allows you to easily find package executables and libraries installed in the environment. - -Let's look specifically at path-related environment variables using ``env | grep PATH``: +Let's inspect the relevant environment variables using: .. literalinclude:: outputs/environments/show-paths-1.out :language: console +We'll now demonstrate how this environment setup enables building a simple MPI program. -We can demonstrate use of these environment settings by building a really simple MPI program. - -Let's create a program called ``mpi-hello.c`` that contains the following code: +Create a source file named ``mpi-hello.c`` with the following code: .. code-block:: c - #include - #include - #include + #include + #include + #include - int main(int argc, char **argv) { - int rank; - MPI_Init(&argc, &argv); + int main(int argc, char **argv) { + int rank; + MPI_Init(&argc, &argv); - MPI_Comm_rank(MPI_COMM_WORLD, &rank); - printf("Hello world from rank %d\n", rank); + MPI_Comm_rank(MPI_COMM_WORLD, &rank); + printf("Hello world from rank %d\n", rank); - if (rank == 0) { - printf("zlib version: %s\n", ZLIB_VERSION); - printf("zlib-ng version: %s\n", ZLIBNG_VERSION); - } + if (rank == 0) { + printf("zlib version: %s\n", ZLIB_VERSION); + printf("zlib-ng version: %s\n", ZLIBNG_VERSION); + } - MPI_Finalize(); - } + MPI_Finalize(); + } -This program includes headers from ``mpi`` and ``zlib``. -It also prints out a message from each MPI rank and the version of ``zlib``. +This program includes headers from both ``mpi`` and ``zlib``, prints a message from each MPI rank, and reports the version of ``zlib`` used. -Let's build and run our program: +Let's build and run the program: .. literalinclude:: outputs/environments/use-mpi-1.out :language: console +Notice that we only needed to pass the include path to the compiler—no additional setup was required. +The output confirms that: -Notice that we only needed to pass the include path to the compiler. -We also see that ``Hello world`` is output for each of the ranks and the version of ``zlib`` used to build the program is printed. +- Each MPI rank prints a “Hello world” message. +- Rank 0 prints the versions of ``zlib`` and ``zlib-ng``. -We can confirm the version of ``zlib`` used to build the program is in our environment using ``spack find``: +We can confirm which version of ``zlib`` was used by checking our environment with ``spack find``: .. literalinclude:: outputs/environments/myproject-zlib-ng-1.out :language: console -Note that the reported version *does* match that of our installation. +As expected, the reported version matches the one installed in the environment. ------------------ Reproducing builds ------------------ -Spack environments provide users with *virtual environments* similar to `Python venv `_ and `Conda environments `_). -The goal is to ensure packages in one environment are kept separate from those of another. -These environments can be managed by Spack or independent. -In either case, their environment files can be used to reproduce builds by other users and on other machines. -Since those files are key to reproducing builds, let's start with them. +Spack environments provide users with *virtual environments*—similar to +`Python venv `_ and +`Conda environments `_. + +These environments help ensure that packages in one project are isolated from those in another. +They can be either Spack-managed or standalone, but in both cases, their configuration files allow builds to be reproduced by other users or on different machines. + +Since these files are key to reproducibility, let's begin by reviewing them. ^^^^^^^^^^^^^^^^^ Environment files ^^^^^^^^^^^^^^^^^ -There are two key files tracking the contents of environments: ``spack.yaml`` and ``spack.lock``. -The ``spack.yaml`` file holds the environment configuration that we previously edited through ``spack config edit``. -The ``spack.lock`` file is automatically generated during concretization. +There are two key files that track the contents of a Spack environment: ``spack.yaml`` and ``spack.lock``. -The two files represent two fundamental concepts: +- ``spack.yaml`` stores the environment configuration, including package specs and any overrides (as we previously edited using ``spack config edit``). +- ``spack.lock`` is automatically generated during concretization and captures the fully resolved dependency graph. -* ``spack.yaml``: *abstract* specs and configuration to install; and -* ``spack.lock``: all fully *concrete* specs. +These two files represent distinct but complementary concepts: +* ``spack.yaml``: defines *abstract* specs and configuration to install +* ``spack.lock``: records all *concrete* specs (versions, variants, dependencies) -These files are intended to be used by developers and administrators to manage the environments in a reproducible way. -We will cover their reuse later. +Together, they allow developers and administrators to manage and reproduce software environments consistently. .. note:: - Both environment files can be versioned in repositories, shared, and - used to install the same set of software by different users and on - other machines. + Both environment files can be version-controlled, shared, and reused + to install the same set of packages across different machines and by different users. + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Managed versus independent environments ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Environments are either Spack-managed or independent. -Both types of environments are defined by their environment files. -So far we have only created managed environments. -This section describes their differences. +Spack environments can be either *managed* or *independent*. +Both types are defined by their environment files (``spack.yaml`` and optionally ``spack.lock``), but they differ in how they are created and referenced. + +So far, we have only worked with **managed environments**. +This section explains the difference between the two types. + +**Managed environments** are created using: + +.. code-block:: console -*Managed environments* are created using ``spack env create ``. -They are automatically created in the ``var/spack/environments`` subdirectory and can be referenced by their names. + $ spack env create -*Independent environments* can be created in one of two ways. -First, the Spack environment file(s) can be placed in any directory (other than ``var/spack/environments``). -Alternatively, you can use ``spack env create -d `` to specify the directory (````) in which the files should reside. -Independent environments are not named. +These environments are automatically placed under the ``var/spack/environments`` directory in your Spack installation. +They are referenced by their given names. + +**Independent environments** are created outside of Spack's internal directory structure. +You can create one in two ways: + +1. By placing a ``spack.yaml`` (and optionally ``spack.lock``) file in any directory *other than* ``var/spack/environments``. +2. By explicitly specifying the directory using: + + .. code-block:: console + + $ spack env create -d + +Independent environments are *not* referenced by name; instead, they are activated or accessed by their path. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Reviewing a managed environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -We created the ``myproject`` environment earlier using ``spack env create myproject`` so let's mainly focus on its environment files in this section. +We created the ``myproject`` environment earlier using: + +.. code-block:: console + + $ spack env create myproject -Earlier, when we changed the environment's configuration using ``spack config edit``, we were actually editing its ``spack.yaml`` file. -We can move to the directory containing the file using ``spack cd``: +So let's focus on reviewing its environment files. + +When we modified the environment's configuration with ``spack config edit``, we were actually editing its ``spack.yaml`` file. +To navigate to the directory containing that file, use: .. literalinclude:: outputs/environments/filenames-1.out :language: console - -Notice that ``myproject`` is a subdirectory of ``var/spack/environments`` within the Spack installation making it a *managed* environment. -Consequently, it can be referenced by name. -It will also show up when running ``spack env list``: +As shown, ``myproject`` is a subdirectory under ``var/spack/environments`` in your Spack installation—making it a *managed* environment. +Because of this, it can be referenced by name and will appear in the list produced by: .. literalinclude:: outputs/environments/env-list-2.out :language: console -which indicates the active environment by highlighting it in green. +The active environment will be highlighted in green in this listing. -We can also see from the listing above that the current environment directory contains both of the environment files: ``spack.yaml`` and ``spack.lock``. -This is because ``spack.lock`` was generated when we concretized the environment. +We can also see from the listing above that the environment directory contains both ``spack.yaml`` and ``spack.lock``. +The ``spack.lock`` file was generated automatically when we concretized the environment. -If we ``cat`` the ``spack.yaml`` file, we'll see the same specs and view options previously shown by ``spack config get``: +If we examine the contents of ``spack.yaml`` using ``cat``, we'll see the same specs and view options previously shown by ``spack config get``: .. literalinclude:: outputs/environments/cat-config-1.out :language: console @@ -530,41 +548,36 @@ If we ``cat`` the ``spack.yaml`` file, we'll see the same specs and view options Creating an independent environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Environments do not have to be created in or managed by a Spack instance. -Rather, their environment files can be placed in any directory. -This feature can be quite helpful for use cases such as environment-based software releases and CI/CD. +Environments do not need to be created within or managed by a Spack instance. +Instead, their environment files (``spack.yaml`` and optionally ``spack.lock``) can be placed in any directory. +This flexibility is useful for workflows like software releases and CI/CD pipelines. Let's create an *independent* environment from scratch for a simple project: .. literalinclude:: outputs/environments/independent-create-1.out :language: console +The output confirms that Spack created the environment, updated the view, and printed the command required to activate it. +Because the environment is independent, it must be referenced by its directory path when activating it. -Notice that the command shows Spack created the environment, updated the view, and printed the command needed to activate it. -As we can see in the activation command, since the environment is independent, it must be referenced by its directory path. - -Let's see what really happened with this command by listing the directory contents and looking at the configuration file: +Let's inspect the directory contents and view the configuration file: .. literalinclude:: outputs/environments/independent-create-2.out :language: console +As shown, Spack created a ``spack.yaml`` file in the ``code`` directory. +This file contains an empty spec list (``[]``), which is intended to hold only the *root specs* of the environment. -Notice that Spack created a ``spack.yaml`` file in the *code* directory. -Also note that the configuration file has an empty spec list (i.e., ``[]``). -That list is intended to contain only the *root specs* of the environment. - -We can confirm that it is not a managed environment by running ``spack env list``: +To confirm this is not a managed environment, run: .. literalinclude:: outputs/environments/env-list-3.out :language: console -and noting that the path does not appear in the output. +You'll see that the environment path does not appear in the output—indicating it is not registered under Spack's ``var/spack/environments``. -Now let's add some specs to the environment. -Suppose your project depends on ``trilinos`` and ``openmpi``. -Add these packages to the spec list using your favorite text editor. -The dash syntax for a YAML list is used in our example. -Your package should now contain the following entries: +Next, let's add some specs to the environment. +Suppose your project depends on ``trilinos`` and ``openmpi``. +Edit the ``spack.yaml`` file using your preferred text editor, and add the following under ``specs:``: .. code-block:: yaml @@ -586,166 +599,155 @@ Now activate the environment and install the packages: .. literalinclude:: outputs/environments/install-independent-1.out :language: console +As shown, Spack concretizes the specs and installs them along with their dependencies. +It also updates the environment's view. +In this case, since the packages were already installed elsewhere, Spack reused them and simply linked them into the environment. -Notice that Spack concretized the specs before installing them and their dependencies. -It also updated the environment's view. -Since we already installed all these packages outside of the environment, their links were simply added to it. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Updating an installed environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Spack supports tweaking an environment even after the initial specs are installed. -You are free to add and remove specs just as you would outside of the environment using the command line interface as before. +Spack allows you to modify an environment even after it has been installed. +You can add or remove specs just as you would outside an environment—using the same command-line interface. -For example, let's add ``hdf5`` and look at our file: +For example, let's add ``hdf5`` and inspect the result: .. literalinclude:: outputs/environments/add-independent-1.out :language: console - -Notice that ``spack add`` added the package to our active environment and it appears in the configuration file's spec list. +Notice that ``spack add`` updated the active environment and ``hdf5`` now appears in the ``specs`` list of the configuration file. .. note:: - You'll need to run ``spack install`` to install added packages - in your environment because ``spack add`` only adds it to the - configuration. + ``spack add`` only updates the configuration—it does not install the package. + You must run ``spack install`` separately to install the newly added spec. -Now use ``spack remove`` to remove the spec from the configuration: +Now, let's remove the spec from the configuration using ``spack remove``: .. literalinclude:: outputs/environments/remove-independent-1.out :language: console -and we see that the spec *was* removed from the spec list of our environment. +As shown, the spec *was* removed from the ``specs`` list of the environment. .. note:: - You can also edit the ``spack.yaml`` file directly instead of - using the ``spack add`` and ``spack remove`` commands. + You can also directly edit the ``spack.yaml`` file to add or remove specs, + instead of using ``spack add`` and ``spack remove``. + ^^^^^^^^^^^^^^^^^^^^^^^^ Reviewing ``spack.lock`` ^^^^^^^^^^^^^^^^^^^^^^^^ -Now let's turn our attention from the abstract to the concrete. +Now let's shift focus from the *abstract* configuration to the *concrete* environment state. -Our focus so far has been on the abstract environment configuration represented by the ``spack.yaml`` file. -Once that file is concretized, Spack *generates* a corresponding ``spack.lock`` file representing the full concretized state of the environment. +Up to this point, we've focused on the abstract environment configuration defined in the ``spack.yaml`` file. +Once that file is concretized, Spack automatically generates a corresponding ``spack.lock`` file. +This lockfile represents the fully concretized state of the environment. -This file is intended to be a machine-readable representation of the information needed to *reproduce* the build of an environment. -As such, it is written in ``json``, which is less readable than ``yaml``. +The ``spack.lock`` file is intended to serve as a machine-readable snapshot of everything needed to *reproduce* the environment's build. +Unlike the human-friendly ``yaml`` format used for ``spack.yaml``, the lockfile is written in ``json``, which is more suitable for automation—but less readable. -Let's look at the top 30 lines of our current environment: +Let's view the top 30 lines of the current environment's lockfile: .. literalinclude:: outputs/environments/lockfile-1.out :language: console - -While it is still readable, it consists of over 1900 lines of information representing the actual configurations for each of the environment's packages. +While still somewhat readable, the lockfile spans more than 1900 lines, detailing the exact configurations of all packages in the environment. ^^^^^^^^^^^^^^^^^^^^^^^^^^ Reproducing an environment ^^^^^^^^^^^^^^^^^^^^^^^^^^ -Now that we've described the contents of the environment files we can discuss how they can be used to reproduce environments. -You may want to do this yourself on a different machine, or use an environment built by someone else. -The process is the same in either case. +Now that we've explored the contents of environment files, let's discuss how they can be used to reproduce environments. +You may want to do this yourself on another machine, or use an environment defined by someone else—the process is the same in either case. -You can recreate an environment by passing either of the environment files to ``spack env create``. -The file you choose depends on whether you want to approximate the build using the abstract specs or an *exact* build based on the concrete specs. +You can recreate an environment by passing either ``spack.yaml`` or ``spack.lock`` to ``spack env create``. +The file you choose determines whether the environment is built approximately from abstract specs or exactly from concrete specs. """""""""""""""""""" Using ``spack.yaml`` """""""""""""""""""" -An approximate build is created using the ``spack.yaml`` file. -This approach is relevant when we want to build the same specs on a new platform, for example. -It allows you to reproduce the environment by preserving the abstract requirements in the file. -However, the software may actually build differently in part because the concretizer may choose different dependencies. +An approximate (re-)build is created using the ``spack.yaml`` file. +This approach is useful when replicating environments across different platforms or when aiming to match general software requirements rather than exact versions. + +Using ``spack.yaml`` preserves the *abstract* intent of the environment (e.g., package names and constraints), but the actual concretization may differ if: -Let's use ``spack env create`` to create an abstract environment from the file that we'll call ``abstract``: +- The platform has different compilers or external packages +- The Spack package recipes or configuration have changed +- The default versions or variants have been updated + +Let's create a new environment called ``abstract`` using the ``spack.yaml`` file: .. literalinclude:: outputs/environments/create-from-file-1.out :language: console +As shown, Spack created a new *managed* environment using the name we provided. -Here we see that Spack created a managed environment with the name we provided. - -And, since it is a newly created environment, it does not have any *installed* specs yet as we can see from calling ``spack find`` **after** activating the environment: +Since it's newly created, no specs have been installed yet. +We can confirm this by activating the environment and running ``spack find``: .. literalinclude:: outputs/environments/find-env-abstract-1.out :language: console -Notice that we have the same root specs as were listed in the ``spack.yaml`` file. +We see that the environment includes the same root specs listed in the original ``spack.yaml`` file, but they have not yet been installed. + """""""""""""""""""" Using ``spack.lock`` """""""""""""""""""" -The ``spack.lock`` file is used for an exact reproduction of the original build. -It can replicate the build because it contains the information for all the decisions made during concretization. +The ``spack.lock`` file enables *exact* reproduction of a previously concretized environment. +This is possible because it encodes all decisions made during the original concretization process, including package versions, variants, compilers, and dependency resolutions. -Now let's create a concrete environment, called ``concrete``, from the file: +Let's create a new environment called ``concrete`` using the ``spack.lock`` file: .. literalinclude:: outputs/environments/create-from-file-2.out :language: console -Here we see that Spack again created a managed environment with the provided name. +As shown, Spack created another *managed* environment using the provided name. -Since we created the environment from our ``spack.lock`` file, not only do we get the same root specs, all of the packages are installed in the environment as we can see from calling ``spack find`` **after** activating the environment: +Since we created this environment from the ``spack.lock`` file, it includes not just the root specs—but also all the concrete, previously installed packages. +We can confirm this by activating the environment and running ``spack find``: .. literalinclude:: outputs/environments/find-env-concrete-1.out :language: console .. note:: - Use of ``spack.lock`` to reproduce a build (currently) requires you - to be on the same type of machine. + Reproducing a build from ``spack.lock`` currently requires running on a + machine with the same platform (architecture, OS, etc.) as the one where + the lockfile was originally created. ------------------- More information ------------------- -This tutorial only scratches the surface of environments and what they can do. -For more information, take a look at the Spack resources below. +This tutorial only scratches the surface of what environments in Spack can do. +For more in-depth guidance, refer to the resources below. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Setting up and building environments ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -* `Environments `_: - reference docs -* `Configuration tutorial - `_: - for customizing your environment -* `Spack stacks tutorial - `_: - for configuring combinatorial environments (e.g., same packages across a - list of compilers) -* `Install-level parallel builds - `_: - for how to launch ``spack install`` to build your environment in parallel +* `Environments `_: reference docs +* `Configuration tutorial `_: customizing environment configuration +* `Spack stacks tutorial `_: managing combinatorial environments (e.g., same packages across multiple compilers) +* `Install-level parallel builds `_: using parallelism to speed up large environment builds ^^^^^^^^^^^^^^^^^^^^^^^ Using environments ^^^^^^^^^^^^^^^^^^^^^^^ -* `Developer workflows - `_: - for developing code in an environment -* `GitLab CI pipelines with Spack environments - `_: - for using environments to generate CI pipelines -* `Container Images `_: - for creating containers from environments -* `Spack stacks tutorial - `_: - for managing large deployments of software +* `Developer workflows `_: developing software in an active environment +* `GitLab CI pipelines with Spack environments `_: using environments to define and run CI/CD pipelines +* `Container Images `_: building container images directly from Spack environments +* `Spack stacks tutorial `_: deploying large-scale, multi-config environments ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Finding examples of environments ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -* `Spack Stack Catalog `_: for - discovering environments that you can explore on GitHub +* `Spack Stack Catalog `_: searchable catalog of example environments and configurations on GitHub From 399c0c4533f4f0c7a55b0ced7c364a47f2557205 Mon Sep 17 00:00:00 2001 From: Caetano Melone Date: Mon, 2 Jun 2025 17:39:57 -0700 Subject: [PATCH 5/5] fix lint --- tutorial_developer_workflows.rst | 80 +++++++++--------- tutorial_environments.rst | 139 +++++++++++++++---------------- tutorial_scripting.rst | 43 +++++----- tutorial_stacks.rst | 103 ++++++++++++----------- 4 files changed, 179 insertions(+), 186 deletions(-) diff --git a/tutorial_developer_workflows.rst b/tutorial_developer_workflows.rst index 3dea5f7954..c844e91ef4 100644 --- a/tutorial_developer_workflows.rst +++ b/tutorial_developer_workflows.rst @@ -11,30 +11,30 @@ Developer Workflows Tutorial ============================ -This tutorial will guide you through using the ``spack develop`` command to develop software from local source code within a Spack environment. +This tutorial will guide you through using the ``spack develop`` command to develop software from local source code within a Spack environment. With this command, Spack manages your dependencies while you focus on testing changes to your library and/or application. ----------------------------- Installing from Local Source ----------------------------- -The ``spack install`` command typically fetches source code from a mirror or the internet before building and installing your package. +The ``spack install`` command typically fetches source code from a mirror or the internet before building and installing your package. As developers, however, we often want to build from local source code that we continuously modify, build, and test. -Let's imagine we're working on ``scr``—a library used to implement scalable checkpointing in application codes. -It supports fast, efficient checkpoint read/write operations using MPI and high-bandwidth file I/O. +Let's imagine we're working on ``scr``—a library used to implement scalable checkpointing in application codes. +It supports fast, efficient checkpoint read/write operations using MPI and high-bandwidth file I/O. We want to test changes to ``scr`` within an actual application, so we'll use ``macsio``, a proxy application that mimics typical HPC I/O workloads. We've chosen ``scr`` and ``macsio`` because they are both quick to build. -We'll begin by creating a Spack environment for our development work. -We need to build ``macsio`` with ``scr`` support, and for now, we want everything built without Fortran support. +We'll begin by creating a Spack environment for our development work. +We need to build ``macsio`` with ``scr`` support, and for now, we want everything built without Fortran support. Let's set up this development workflow: .. literalinclude:: outputs/dev/setup-scr.out :language: console -Before making any changes, we verify that everything builds correctly. +Before making any changes, we verify that everything builds correctly. Spack builds the entire development tree as specified and links all components together for you. @@ -42,7 +42,7 @@ Spack builds the entire development tree as specified and links all components t :language: console -Before making any changes, we verify that everything builds correctly. +Before making any changes, we verify that everything builds correctly. Spack builds the entire development tree as specified and links all components together for you. @@ -100,17 +100,17 @@ Now we are ready to begin work on the actual application. Development Iteration Cycle ----------------------------- -Let's assume that ``scr`` has a bug, and we'd like to patch it to find out what the problem is. -First, we tell Spack that we want to check out the version of ``scr`` we intend to work on. +Let's assume that ``scr`` has a bug, and we'd like to patch it to find out what the problem is. +First, we tell Spack that we want to check out the version of ``scr`` we intend to work on. In this case, it's the 3.1.0 release that we want to patch: .. literalinclude:: outputs/dev/develop-1.out :language: console -The ``spack develop`` command marks the package as a "development" package in the ``spack.yaml``. -This adds a special ``dev_path=`` attribute to the spec for the package, so Spack remembers where the source code is located. -The ``develop`` command also downloads or checks out the source code for the package. -By default, the source code is downloaded into a subdirectory of the environment. +The ``spack develop`` command marks the package as a "development" package in the ``spack.yaml``. +This adds a special ``dev_path=`` attribute to the spec for the package, so Spack remembers where the source code is located. +The ``develop`` command also downloads or checks out the source code for the package. +By default, the source code is downloaded into a subdirectory of the environment. You can change the location of this source directory by modifying the ``path:`` attribute of the ``develop`` configuration in the environment. There are a few gotchas with the ``spack develop`` command: @@ -133,31 +133,31 @@ Now that this is done, we tell Spack to rebuild both ``scr`` and ``macsio`` by r .. literalinclude:: outputs/dev/develop-2.out :language: console -This rebuilds ``scr`` from the subdirectory we specified. -If your package uses CMake, Spack will build it in a directory corresponding to the hash of your package. +This rebuilds ``scr`` from the subdirectory we specified. +If your package uses CMake, Spack will build it in a directory corresponding to the hash of your package. From there, you can change into the appropriate directory and perform your own build/test cycles. -Now we can develop our code. -For the sake of this demo, we're going to intentionally introduce an error. +Now we can develop our code. +For the sake of this demo, we're going to intentionally introduce an error. Let’s edit a file and remove the first semicolon we find. .. literalinclude:: outputs/dev/edit-1.out :language: console -Once you have a development package, ``spack install`` works like ``make``. -Because Spack knows the source code directory of the package, it checks file timestamps to detect recent changes. +Once you have a development package, ``spack install`` works like ``make``. +Because Spack knows the source code directory of the package, it checks file timestamps to detect recent changes. If any files have been modified, Spack will rebuild ``scr`` and its dependents. .. literalinclude:: outputs/dev/develop-3.out :language: console -Here, the build failed as expected. -We can inspect the build output in ``scr/spack-build-out.txt``. +Here, the build failed as expected. +We can inspect the build output in ``scr/spack-build-out.txt``. Alternatively, to debug interactively, we can launch a shell within the build environment using: ``spack build-env scr@2.0 -- bash`` -If that’s too much to remember, sourcing ``scr/spack-build-env.txt`` will set the appropriate environment variables so you can diagnose the build manually. +If that’s too much to remember, sourcing ``scr/spack-build-env.txt`` will set the appropriate environment variables so you can diagnose the build manually. Now let's fix the issue and rebuild directly. .. literalinclude:: outputs/dev/develop-4.out @@ -165,8 +165,8 @@ Now let's fix the issue and rebuild directly. You'll notice that Spack rebuilt both ``scr`` and ``macsio``, as expected. -Taking advantage of iterative builds with Spack requires cooperation from your build system. -When Spack performs a rebuild on a development package, it reruns all the build stages without cleaning the source and build directories to a pristine state. +Taking advantage of iterative builds with Spack requires cooperation from your build system. +When Spack performs a rebuild on a development package, it reruns all the build stages without cleaning the source and build directories to a pristine state. If your build system can reuse previously compiled object files, you’ll benefit from an iterative build. - If your package uses ``make``, you should get iterative builds automatically when using ``spack develop``. @@ -174,14 +174,14 @@ If your build system can reuse previously compiled object files, you’ll benefi This is because CMake doesn’t modify the file time of ``CMakeCache.txt`` unless your CMake flags change. - If your package uses Autoconf, rerunning the typical ``autoreconf`` stage will usually modify the timestamp of ``config.h``, which may trigger a full rebuild. -Multiple packages can also be marked as development packages. +Multiple packages can also be marked as development packages. If we were co-developing ``macsio``, we could run: .. literalinclude:: outputs/dev/develop-5.out :language: console -Using development workflows also allows us to share our full development setup with other team members. -They can simply use our ``spack.yaml`` to create a new environment and replicate the entire build process. +Using development workflows also allows us to share our full development setup with other team members. +They can simply use our ``spack.yaml`` to create a new environment and replicate the entire build process. For example, here we create another development environment: .. literalinclude:: outputs/dev/otherdevel.out @@ -198,15 +198,15 @@ When we're done developing, we simply tell Spack that it no longer needs to keep Workflow Summary ------------------- -Use the ``spack develop`` command within an environment to create a reproducible build setup for your development workflow. -Spack will manage all dependencies and link your packages together automatically. +Use the ``spack develop`` command within an environment to create a reproducible build setup for your development workflow. +Spack will manage all dependencies and link your packages together automatically. Within a development environment, ``spack install`` behaves similarly to ``make``: it checks file timestamps and rebuilds only the minimum set of Spack packages required to reflect your changes. ------------------- Optional: Tips and Tricks ------------------- -This section covers additional features that complement the core tutorial above. +This section covers additional features that complement the core tutorial above. Many of these tips are especially useful for specific projects and developer workflows. A list of available options for the ``spack develop`` command can be viewed below: @@ -219,11 +219,11 @@ A list of available options for the ``spack develop`` command can be viewed belo Source Code Management ---------------------- -The ``spack develop`` command allows users to control the location of source code. +The ``spack develop`` command allows users to control the location of source code. By default, Spack manages source locations and handles cloning automatically, but software developers often prefer more control. -You can specify the source directory using the ``--path`` argument when invoking ``spack develop``. -If the specified directory already exists, Spack will not attempt to fetch the source code. +You can specify the source directory using the ``--path`` argument when invoking ``spack develop``. +If the specified directory already exists, Spack will not attempt to fetch the source code. This allows developers to pre-clone repositories or use preferred directory paths as needed. .. code-block:: console @@ -238,10 +238,10 @@ This allows developers to pre-clone repositories or use preferred directory path Navigation and the Build Environment ------------------------------------ -Diving into the build environment was introduced earlier in the packaging section using the ``spack build-env scr -- bash`` command. +Diving into the build environment was introduced earlier in the packaging section using the ``spack build-env scr -- bash`` command. This is a helpful feature because it allows you to run commands inside the package’s build environment. -In the packaging section, this was combined with ``spack cd`` to demonstrate a manual build process outside of Spack’s automated workflow. +In the packaging section, this was combined with ``spack cd`` to demonstrate a manual build process outside of Spack’s automated workflow. This approach is especially useful in developer environments, providing a streamlined workflow for iterating on a single package without the overhead of the ``spack install`` command. The extra features of the ``spack install`` command are often unnecessary when rapidly iterating between building and testing a specific package. @@ -272,9 +272,8 @@ Working directly within the build environment, combined with Spack's navigation Combinatorics ------------- -The final topic in this tutorial highlights the power of combinatorial development builds. -There are many cases where developers want to see how a single set of changes affects multiple build configurations—for example: -``+cuda`` vs ``~cuda``, ``%gcc`` vs ``%clang``, or ``build_type=Release`` vs ``build_type=Debug``. +The final topic in this tutorial highlights the power of combinatorial development builds. +There are many cases where developers want to see how a single set of changes affects multiple build configurations—for example: ``+cuda`` vs ``~cuda``, ``%gcc`` vs ``%clang``, or ``build_type=Release`` vs ``build_type=Debug``. Developers can build all of these configurations with a single ``spack install`` call, as long as the develop spec is generic enough to cover the spec variations of the packages. @@ -290,8 +289,7 @@ Developers can build all of these configurations with a single ``spack install`` While we won't build out this example, it illustrates how the ``dev_path`` for both ``build_type=Release`` and ``build_type=Debug`` points to the same source code. -If we want to do most of our incremental builds using the ``Release`` configuration and periodically check results using the ``Debug`` build, we can combine the workflow from the previous example: -First, enter the ``Release`` build environment using: +If we want to do most of our incremental builds using the ``Release`` configuration and periodically check results using the ``Debug`` build, we can combine the workflow from the previous example: First, enter the ``Release`` build environment using: ``spack build-env scr build_type=Release -- bash`` diff --git a/tutorial_environments.rst b/tutorial_environments.rst index a7dd191131..4d642f9035 100644 --- a/tutorial_environments.rst +++ b/tutorial_environments.rst @@ -20,10 +20,10 @@ We've covered how to install, remove, and list packages in Spack using the follo .. Customizing Spack's installation with configuration files such as `packages.yaml `_ was also discussed. -This section of the tutorial introduces **Spack Environments**, which let you manage independent groups of packages in a reproducible way. +This section of the tutorial introduces **Spack Environments**, which let you manage independent groups of packages in a reproducible way. In some ways, Spack environments are similar to *virtual environments* in other systems (e.g., `Python venv `_), but they are built around file formats (``spack.yaml`` and ``spack.lock``) that can be easily shared and reused across systems. -Managing software that involves many packages and varying configuration requirements (e.g., different ``mpi`` implementations) for multiple projects can quickly become overwhelming. +Managing software that involves many packages and varying configuration requirements (e.g., different ``mpi`` implementations) for multiple projects can quickly become overwhelming. Spack environments help by enabling you to: * define and standardize software requirements for your projects; @@ -33,7 +33,7 @@ Spack environments help by enabling you to: * reproduce builds—approximately or exactly—on other systems; and * much more. -This tutorial introduces the basics of creating and using environments, explains how to expand, configure, and build software within them, and distinguishes between Spack-managed and standalone environments. +This tutorial introduces the basics of creating and using environments, explains how to expand, configure, and build software within them, and distinguishes between Spack-managed and standalone environments. We'll begin with the command-line interface, then cover editing key environment files directly, and conclude with guidance on reproducible builds. ------------------- @@ -45,7 +45,7 @@ Let's take a look at the output of ``spack find`` at this point in the tutorial: .. literalinclude:: outputs/environments/find-no-env-1.out :language: console -This is a complete—but cluttered—list of installed packages and their dependencies. +This is a complete—but cluttered—list of installed packages and their dependencies. It includes packages built with both ``openmpi`` and ``mpich``, along with multiple variants of other packages such as ``hdf5`` and ``zlib-ng``. While the ``spack find`` query mechanisms we've covered can help navigate this, it would be even more useful to start from a clean slate—without losing what we've already installed. @@ -54,13 +54,13 @@ While the ``spack find`` query mechanisms we've covered can help navigate this, Creating and activating environments ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The ``spack env`` command is used to manage environments. +The ``spack env`` command is used to manage environments. Let's create a new environment called ``myproject``: .. literalinclude:: outputs/environments/env-create-1.out :language: console -An environment is like a virtualized Spack instance that aggregates package installations for a specific project or purpose. +An environment is like a virtualized Spack instance that aggregates package installations for a specific project or purpose. It has an associated *view*, which is a single directory where all packages from the environment are linked. You can list the environments you've created so far with the ``spack env list`` command: @@ -68,7 +68,7 @@ You can list the environments you've created so far with the ``spack env list`` .. literalinclude:: outputs/environments/env-list-1.out :language: console -Now, let's **activate** our environment. +Now, let's **activate** our environment. Use the ``spack env activate`` command: .. literalinclude:: outputs/environments/env-activate-1.out @@ -87,14 +87,13 @@ You can also use the shorter alias ``spacktivate`` for ``spack env activate``. This option is set by default in interactive shells, but not in non-interactive ones (e.g., when running a Bash script). To enable it in a script, run: ``shopt -s expand_aliases``. -Once you activate an environment, ``spack find`` shows only the packages in that environment. +Once you activate an environment, ``spack find`` shows only the packages in that environment. Since we just created this one, it does not contain any packages yet: .. literalinclude:: outputs/environments/find-env-1.out :language: console -The output from ``spack find`` is now *slightly* different: -It tells you that you're in the ``myproject`` environment—so don't panic if none of your previously installed packages are listed. +The output from ``spack find`` is now *slightly* different: It tells you that you're in the ``myproject`` environment—so don't panic if none of your previously installed packages are listed. It also notes that there are **no** *root specs* (we'll explain that term shortly). If you only want to check which environment is currently active, use: @@ -123,13 +122,14 @@ First, try the usual install command: .. literalinclude:: outputs/environments/env-fail-install-1.out :language: console -Environments are a bit different—before installing packages, you must first *add* them. -The ``spack add`` command queues up specs to be installed in the environment. Let's try it: +Environments are a bit different—before installing packages, you must first *add* them. +The ``spack add`` command queues up specs to be installed in the environment. +Let's try it: .. literalinclude:: outputs/environments/env-add-1.out :language: console -Now, ``tcl`` and ``trilinos`` have been registered as **root specs** in this environment. +Now, ``tcl`` and ``trilinos`` have been registered as **root specs** in this environment. Root specs are the packages you explicitly request for installation—they serve as the **roots** of the environment's dependency graph. Let's install them: @@ -137,7 +137,7 @@ Let's install them: .. literalinclude:: outputs/environments/env-install-1.out :language: console -We see that ``tcl`` and the dependencies of ``trilinos`` were already installed, while ``trilinos`` itself was newly built. +We see that ``tcl`` and the dependencies of ``trilinos`` were already installed, while ``trilinos`` itself was newly built. The environment's view was also updated to reflect the new installations. Now, confirm the environment contents using ``spack find``: @@ -167,10 +167,10 @@ You can also add and install specs incrementally, one at a time: $ spack install --add tcl $ spack install --add trilinos -When building environments incrementally, Spack ensures that already installed root specs are not re-concretized. +When building environments incrementally, Spack ensures that already installed root specs are not re-concretized. This means that adding new specs later will not cause previously installed packages to be rebuilt. -However, incrementally adding and installing specs may result in different package versions than installing them all at once. +However, incrementally adding and installing specs may result in different package versions than installing them all at once. We'll revisit this topic when we discuss different concretization strategies. Additionally, there are two key benefits to concretizing and installing an environment all at once: @@ -183,18 +183,18 @@ Additionally, there are two key benefits to concretizing and installing an envir Using packages ^^^^^^^^^^^^^^ -Environments provide a convenient way to use installed packages. +Environments provide a convenient way to use installed packages. Activating an environment with ``spack env activate`` automatically places everything in the environment on your ``PATH``. Without environments, you would need to manually use `spack load `_ or `module load `_ for each package to set up the appropriate environment for the package and its dependencies. -When you install packages into an environment, they are—by default—linked into a single directory prefix known as a *view*. +When you install packages into an environment, they are—by default—linked into a single directory prefix known as a *view*. Activating the environment with ``spack env activate`` adds relevant subdirectories from this view to ``PATH``, ``MANPATH``, ``CMAKE_PREFIX_PATH``, and other environment variables, making the environment easier to use. Let's try it out. -We just installed ``tcl`` in the ``myproject`` environment. -Tcl includes a shell-like executable called ``tclsh``. +We just installed ``tcl`` in the ``myproject`` environment. +Tcl includes a shell-like executable called ``tclsh``. You can check its location with ``which``: .. literalinclude:: outputs/environments/use-tcl-1.out @@ -215,10 +215,10 @@ You can now run ``tclsh`` like any other command available in your ``PATH``: Uninstalling packages ^^^^^^^^^^^^^^^^^^^^^ -Spack environments let you uninstall or remove packages *within* one environment without affecting other environments or globally installed packages. +Spack environments let you uninstall or remove packages *within* one environment without affecting other environments or globally installed packages. This is because environments primarily manage which packages are part of their configuration and view. -Let's demonstrate this by creating another environment. +Let's demonstrate this by creating another environment. Suppose ``myproject`` requires ``trilinos``, but we have another project that also installed it and no longer needs it. Start by creating a ``myproject2`` environment and adding the installed packages ``scr`` and ``trilinos``: @@ -236,13 +236,13 @@ Let's try uninstalling ``trilinos`` from ``myproject2`` and review the environme .. literalinclude:: outputs/environments/env-uninstall-1.out :language: console -We see that ``trilinos`` was not uninstalled because it is still referenced by another environment. +We see that ``trilinos`` was not uninstalled because it is still referenced by another environment. To remove it from the list of root specs in ``myproject2``, use ``spack remove``: .. literalinclude:: outputs/environments/env-remove-1.out :language: console -After removing it, you'll see that ``trilinos`` is no longer listed as a root spec—but it still appears in the environment until we re-concretize. +After removing it, you'll see that ``trilinos`` is no longer listed as a root spec—but it still appears in the environment until we re-concretize. Once we do, the spec is fully removed. To confirm ``trilinos`` is still available in the ``myproject`` environment, switch back: @@ -250,8 +250,7 @@ To confirm ``trilinos`` is still available in the ``myproject`` environment, swi .. literalinclude:: outputs/environments/env-swap-1.out :language: console -Phew! -``trilinos`` is still present as a root spec in ``myproject``. +Phew! ``trilinos`` is still present as a root spec in ``myproject``. Spack uses reference counting to ensure packages aren't uninstalled while they're still required elsewhere. .. note:: @@ -268,10 +267,10 @@ You can also uninstall a package and remove it from the environment in one step The ``spack.yaml`` file ----------------------- -An environment is more than just a list of root specs. +An environment is more than just a list of root specs. It also includes *configuration* settings that affect Spack's behavior when the environment is active. -So far, ``myproject`` has relied on configuration defaults, but these can be overridden. +So far, ``myproject`` has relied on configuration defaults, but these can be overridden. For example, we may want to ensure that all packages depending on ``mpi`` use ``mpich`` instead of another provider. We can customize provider selection using `concretization preferences `_. @@ -310,7 +309,7 @@ Editing environment configuration Let's edit ``spack.yaml`` to *require* ``mpich`` as the ``mpi`` provider using the ``spack config edit`` command. -This should open the file in your configured editor. +This should open the file in your configured editor. Modify it to include the ``packages:mpi:require`` entry as shown below: .. code-block:: yaml @@ -334,17 +333,17 @@ Modify it to include the ``packages:mpi:require`` entry as shown below: can affect concretization. Configuration is covered in more detail in the :ref:`configuration tutorial `. -Here, we've only scratched the surface by requiring a specific ``mpi`` provider for packages that depend on ``mpi``. +Here, we've only scratched the surface by requiring a specific ``mpi`` provider for packages that depend on ``mpi``. Spack environments support many additional customizations—refer to the links at the end of this section for further exploration. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Re-concretizing the environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -You may need to re-install packages in an environment after making significant configuration changes—such as switching virtual providers. +You may need to re-install packages in an environment after making significant configuration changes—such as switching virtual providers. This is done by forcing Spack to *re-concretize* the environment and re-install the affected specs. -For example, the packages currently installed in our ``myproject`` environment are now out of sync with the configuration, because we previously installed part of the environment with ``openmpi``. +For example, the packages currently installed in our ``myproject`` environment are now out of sync with the configuration, because we previously installed part of the environment with ``openmpi``. Suppose we now want everything in ``myproject`` to use ``mpich`` instead. We can run: @@ -360,17 +359,17 @@ We can run: All specs are now concrete and ready to be installed with ``mpich`` as the selected MPI provider. -Re-concretization is also necessary when building environments *incrementally* with unification enabled. +Re-concretization is also necessary when building environments *incrementally* with unification enabled. By default, Spack avoids modifying already concretized specs when new ones are added. -Incrementally adding and installing specs leads to *greedy concretization*. -For example, if you install ``python`` first, Spack may choose the most recent version. +Incrementally adding and installing specs leads to *greedy concretization*. +For example, if you install ``python`` first, Spack may choose the most recent version. Later, if you add ``py-numpy``, it might conflict with the previously selected ``python`` version and fail to concretize: .. literalinclude:: outputs/environments/incremental-1.out :language: console -To fix this, re-concretize the entire environment. +To fix this, re-concretize the entire environment. Spack will then select compatible versions for all specs—for example, downgrading ``python`` if needed: .. literalinclude:: outputs/environments/incremental-2.out @@ -380,17 +379,17 @@ Spack will then select compatible versions for all specs—for example, downgrad Building in environments ------------------------ -Activated environments allow you to use any installed programs as if they were installed system-wide. +Activated environments allow you to use any installed programs as if they were installed system-wide. In this section, we'll take advantage of that feature. -Suppose you want to compile an MPI program. -Since our ``myproject2`` environment includes an MPI implementation, ``mpicc`` is available on the ``PATH``. +Suppose you want to compile an MPI program. +Since our ``myproject2`` environment includes an MPI implementation, ``mpicc`` is available on the ``PATH``. We can confirm this using ``which``: .. literalinclude:: outputs/environments/show-mpicc-1.out :language: console -As mentioned earlier, activating an environment sets several key environment variables. +As mentioned earlier, activating an environment sets several key environment variables. These include ``PATH``, ``MANPATH``, and ``CMAKE_PREFIX_PATH``, which allow tools and compilers to easily locate the executables, headers, and libraries installed in the environment. Let's inspect the relevant environment variables using: @@ -430,7 +429,7 @@ Let's build and run the program: .. literalinclude:: outputs/environments/use-mpi-1.out :language: console -Notice that we only needed to pass the include path to the compiler—no additional setup was required. +Notice that we only needed to pass the include path to the compiler—no additional setup was required. The output confirms that: - Each MPI rank prints a “Hello world” message. @@ -447,11 +446,9 @@ As expected, the reported version matches the one installed in the environment. Reproducing builds ------------------ -Spack environments provide users with *virtual environments*—similar to -`Python venv `_ and -`Conda environments `_. +Spack environments provide users with *virtual environments*—similar to `Python venv `_ and `Conda environments `_. -These environments help ensure that packages in one project are isolated from those in another. +These environments help ensure that packages in one project are isolated from those in another. They can be either Spack-managed or standalone, but in both cases, their configuration files allow builds to be reproduced by other users or on different machines. Since these files are key to reproducibility, let's begin by reviewing them. @@ -482,10 +479,10 @@ Together, they allow developers and administrators to manage and reproduce softw Managed versus independent environments ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Spack environments can be either *managed* or *independent*. +Spack environments can be either *managed* or *independent*. Both types are defined by their environment files (``spack.yaml`` and optionally ``spack.lock``), but they differ in how they are created and referenced. -So far, we have only worked with **managed environments**. +So far, we have only worked with **managed environments**. This section explains the difference between the two types. **Managed environments** are created using: @@ -494,10 +491,10 @@ This section explains the difference between the two types. $ spack env create -These environments are automatically placed under the ``var/spack/environments`` directory in your Spack installation. +These environments are automatically placed under the ``var/spack/environments`` directory in your Spack installation. They are referenced by their given names. -**Independent environments** are created outside of Spack's internal directory structure. +**Independent environments** are created outside of Spack's internal directory structure. You can create one in two ways: 1. By placing a ``spack.yaml`` (and optionally ``spack.lock``) file in any directory *other than* ``var/spack/environments``. @@ -521,13 +518,13 @@ We created the ``myproject`` environment earlier using: So let's focus on reviewing its environment files. -When we modified the environment's configuration with ``spack config edit``, we were actually editing its ``spack.yaml`` file. +When we modified the environment's configuration with ``spack config edit``, we were actually editing its ``spack.yaml`` file. To navigate to the directory containing that file, use: .. literalinclude:: outputs/environments/filenames-1.out :language: console -As shown, ``myproject`` is a subdirectory under ``var/spack/environments`` in your Spack installation—making it a *managed* environment. +As shown, ``myproject`` is a subdirectory under ``var/spack/environments`` in your Spack installation—making it a *managed* environment. Because of this, it can be referenced by name and will appear in the list produced by: .. literalinclude:: outputs/environments/env-list-2.out @@ -535,7 +532,7 @@ Because of this, it can be referenced by name and will appear in the list produc The active environment will be highlighted in green in this listing. -We can also see from the listing above that the environment directory contains both ``spack.yaml`` and ``spack.lock``. +We can also see from the listing above that the environment directory contains both ``spack.yaml`` and ``spack.lock``. The ``spack.lock`` file was generated automatically when we concretized the environment. If we examine the contents of ``spack.yaml`` using ``cat``, we'll see the same specs and view options previously shown by ``spack config get``: @@ -548,8 +545,8 @@ If we examine the contents of ``spack.yaml`` using ``cat``, we'll see the same s Creating an independent environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Environments do not need to be created within or managed by a Spack instance. -Instead, their environment files (``spack.yaml`` and optionally ``spack.lock``) can be placed in any directory. +Environments do not need to be created within or managed by a Spack instance. +Instead, their environment files (``spack.yaml`` and optionally ``spack.lock``) can be placed in any directory. This flexibility is useful for workflows like software releases and CI/CD pipelines. Let's create an *independent* environment from scratch for a simple project: @@ -557,7 +554,7 @@ Let's create an *independent* environment from scratch for a simple project: .. literalinclude:: outputs/environments/independent-create-1.out :language: console -The output confirms that Spack created the environment, updated the view, and printed the command required to activate it. +The output confirms that Spack created the environment, updated the view, and printed the command required to activate it. Because the environment is independent, it must be referenced by its directory path when activating it. Let's inspect the directory contents and view the configuration file: @@ -565,7 +562,7 @@ Let's inspect the directory contents and view the configuration file: .. literalinclude:: outputs/environments/independent-create-2.out :language: console -As shown, Spack created a ``spack.yaml`` file in the ``code`` directory. +As shown, Spack created a ``spack.yaml`` file in the ``code`` directory. This file contains an empty spec list (``[]``), which is intended to hold only the *root specs* of the environment. To confirm this is not a managed environment, run: @@ -575,8 +572,8 @@ To confirm this is not a managed environment, run: You'll see that the environment path does not appear in the output—indicating it is not registered under Spack's ``var/spack/environments``. -Next, let's add some specs to the environment. -Suppose your project depends on ``trilinos`` and ``openmpi``. +Next, let's add some specs to the environment. +Suppose your project depends on ``trilinos`` and ``openmpi``. Edit the ``spack.yaml`` file using your preferred text editor, and add the following under ``specs:``: .. code-block:: yaml @@ -599,8 +596,8 @@ Now activate the environment and install the packages: .. literalinclude:: outputs/environments/install-independent-1.out :language: console -As shown, Spack concretizes the specs and installs them along with their dependencies. -It also updates the environment's view. +As shown, Spack concretizes the specs and installs them along with their dependencies. +It also updates the environment's view. In this case, since the packages were already installed elsewhere, Spack reused them and simply linked them into the environment. @@ -608,7 +605,7 @@ In this case, since the packages were already installed elsewhere, Spack reused Updating an installed environment ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Spack allows you to modify an environment even after it has been installed. +Spack allows you to modify an environment even after it has been installed. You can add or remove specs just as you would outside an environment—using the same command-line interface. For example, let's add ``hdf5`` and inspect the result: @@ -642,11 +639,11 @@ Reviewing ``spack.lock`` Now let's shift focus from the *abstract* configuration to the *concrete* environment state. -Up to this point, we've focused on the abstract environment configuration defined in the ``spack.yaml`` file. -Once that file is concretized, Spack automatically generates a corresponding ``spack.lock`` file. +Up to this point, we've focused on the abstract environment configuration defined in the ``spack.yaml`` file. +Once that file is concretized, Spack automatically generates a corresponding ``spack.lock`` file. This lockfile represents the fully concretized state of the environment. -The ``spack.lock`` file is intended to serve as a machine-readable snapshot of everything needed to *reproduce* the environment's build. +The ``spack.lock`` file is intended to serve as a machine-readable snapshot of everything needed to *reproduce* the environment's build. Unlike the human-friendly ``yaml`` format used for ``spack.yaml``, the lockfile is written in ``json``, which is more suitable for automation—but less readable. Let's view the top 30 lines of the current environment's lockfile: @@ -660,17 +657,17 @@ While still somewhat readable, the lockfile spans more than 1900 lines, detailin Reproducing an environment ^^^^^^^^^^^^^^^^^^^^^^^^^^ -Now that we've explored the contents of environment files, let's discuss how they can be used to reproduce environments. +Now that we've explored the contents of environment files, let's discuss how they can be used to reproduce environments. You may want to do this yourself on another machine, or use an environment defined by someone else—the process is the same in either case. -You can recreate an environment by passing either ``spack.yaml`` or ``spack.lock`` to ``spack env create``. +You can recreate an environment by passing either ``spack.yaml`` or ``spack.lock`` to ``spack env create``. The file you choose determines whether the environment is built approximately from abstract specs or exactly from concrete specs. """""""""""""""""""" Using ``spack.yaml`` """""""""""""""""""" -An approximate (re-)build is created using the ``spack.yaml`` file. +An approximate (re-)build is created using the ``spack.yaml`` file. This approach is useful when replicating environments across different platforms or when aiming to match general software requirements rather than exact versions. Using ``spack.yaml`` preserves the *abstract* intent of the environment (e.g., package names and constraints), but the actual concretization may differ if: @@ -686,7 +683,7 @@ Let's create a new environment called ``abstract`` using the ``spack.yaml`` file As shown, Spack created a new *managed* environment using the name we provided. -Since it's newly created, no specs have been installed yet. +Since it's newly created, no specs have been installed yet. We can confirm this by activating the environment and running ``spack find``: .. literalinclude:: outputs/environments/find-env-abstract-1.out @@ -699,7 +696,7 @@ We see that the environment includes the same root specs listed in the original Using ``spack.lock`` """""""""""""""""""" -The ``spack.lock`` file enables *exact* reproduction of a previously concretized environment. +The ``spack.lock`` file enables *exact* reproduction of a previously concretized environment. This is possible because it encodes all decisions made during the original concretization process, including package versions, variants, compilers, and dependency resolutions. Let's create a new environment called ``concrete`` using the ``spack.lock`` file: @@ -709,7 +706,7 @@ Let's create a new environment called ``concrete`` using the ``spack.lock`` file As shown, Spack created another *managed* environment using the provided name. -Since we created this environment from the ``spack.lock`` file, it includes not just the root specs—but also all the concrete, previously installed packages. +Since we created this environment from the ``spack.lock`` file, it includes not just the root specs—but also all the concrete, previously installed packages. We can confirm this by activating the environment and running ``spack find``: .. literalinclude:: outputs/environments/find-env-concrete-1.out @@ -725,7 +722,7 @@ We can confirm this by activating the environment and running ``spack find``: More information ------------------- -This tutorial only scratches the surface of what environments in Spack can do. +This tutorial only scratches the surface of what environments in Spack can do. For more in-depth guidance, refer to the resources below. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ diff --git a/tutorial_scripting.rst b/tutorial_scripting.rst index 12b3e98dc0..7a90e250ac 100644 --- a/tutorial_scripting.rst +++ b/tutorial_scripting.rst @@ -11,35 +11,35 @@ Scripting with Spack ==================== -This tutorial introduces advanced Spack features related to scripting. +This tutorial introduces advanced Spack features related to scripting. Specifically, we'll show how to write scripts using ``spack find`` and ``spack python``. -Earlier sections demonstrated using ``spack find`` to list and search installed packages. +Earlier sections demonstrated using ``spack find`` to list and search installed packages. The ``spack python`` command provides access to all of Spack's `internal APIs `_, allowing you to write more complex queries. -Since Spack has an extensive API, we'll only scratch the surface here. +Since Spack has an extensive API, we'll only scratch the surface here. Our goal is to give you enough information to start writing your own scripts and to help you discover what you need—with a little digging. ----------------------------- Scripting with ``spack find`` ----------------------------- -So far, the output we've seen from ``spack find`` has been intended for human consumption. +So far, the output we've seen from ``spack find`` has been intended for human consumption. However, the command also provides options for generating machine-readable output that can be piped into scripts. ^^^^^^^^^^^^^^^^^^^^^^^ ``spack find --format`` ^^^^^^^^^^^^^^^^^^^^^^^ -The main purpose of ``spack find`` is to display information about concrete specs corresponding to installed packages. +The main purpose of ``spack find`` is to display information about concrete specs corresponding to installed packages. By default, it shows these specs with a set of standard attributes, such as the familiar ``@version`` suffix. -The ``--format`` argument allows you to customize the output string for each found package. +The ``--format`` argument allows you to customize the output string for each found package. Format strings let you specify which *parts* of each spec you want to display. Let's see this option in action. -Suppose you only want to display the *name*, *version*, and the first ten (10) characters of the *hash* for every package installed in your Spack instance. +Suppose you only want to display the *name*, *version*, and the first ten (10) characters of the *hash* for every package installed in your Spack instance. You can generate that output with the following command: .. literalinclude:: outputs/scripting/find-format.out @@ -54,7 +54,7 @@ Using ``spack find --format`` allows you to retrieve only the information you ne ``spack find --json`` ^^^^^^^^^^^^^^^^^^^^^ -Alternatively, you can get a serialized version of ``Spec`` objects in `JSON` format using the ``--json`` option. +Alternatively, you can get a serialized version of ``Spec`` objects in `JSON` format using the ``--json`` option. For example, to retrieve attributes for all installations of ``zlib-ng``, you can run: .. literalinclude:: outputs/scripting/find-json.out @@ -72,7 +72,7 @@ Introducing the ``spack python`` command What if we need to perform more advanced queries? -Spack provides the ``spack python`` command, which launches a Python interpreter with Spack's modules available for import. +Spack provides the ``spack python`` command, which launches a Python interpreter with Spack's modules available for import. This interpreter uses the same underlying Python environment that Spack uses for its other commands. Using this interface, you can write scripts to: @@ -93,9 +93,9 @@ Since we are in a Python interpreter, use ``exit()`` to end the session and retu Accessing the ``Spec`` object ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Now let's take a look at the internal representation of the Spack ``Spec``. -As you may already know, specs can be either *abstract* or *concrete*. -The specs you've seen in ``package.py`` files (e.g., in the ``install()`` method) have been *concrete*—fully specified with resolved dependencies, compilers, and variants. +Now let's take a look at the internal representation of the Spack ``Spec``. +As you may already know, specs can be either *abstract* or *concrete*. +The specs you've seen in ``package.py`` files (e.g., in the ``install()`` method) have been *concrete*—fully specified with resolved dependencies, compilers, and variants. In contrast, the specs you've typed on the command line have been *abstract*. Understanding the difference between these two forms is key to effectively using Spack's internal API. @@ -134,8 +134,8 @@ It's not necessary to store the intermediate abstract spec—you can use the ``. Querying the Spack database ^^^^^^^^^^^^^^^^^^^^^^^^^^^ -More powerful queries become available when interacting with Spack's installation database. -The ``Database`` object is accessible via the ``spack.store.STORE.db`` variable. +More powerful queries become available when interacting with Spack's installation database. +The ``Database`` object is accessible via the ``spack.store.STORE.db`` variable. We'll primarily interact with it through the ``query()`` method. Let's view the documentation for ``query()`` using Python's built-in ``help()`` function: @@ -146,10 +146,10 @@ Let's view the documentation for ``query()`` using Python's built-in ``help()`` We will primarily make use of the ``query_spec`` argument. -Recall that queries using the ``spack find`` command are primarily intended to *match* packages based on specified criteria. +Recall that queries using the ``spack find`` command are primarily intended to *match* packages based on specified criteria. However, it's more difficult to use ``spack find`` for queries that involve *excluding* packages based on complex logic (e.g., “does *not* depend on X”). -Using the Python interface, we *can* write these more advanced queries. +Using the Python interface, we *can* write these more advanced queries. For example, let's find all packages that were compiled with ``gcc`` but do **not** depend on ``mpich``. We'll use ``spack.cmd.display_specs`` to print the results, replicating the display behavior of the ``spack find`` command: @@ -173,7 +173,7 @@ before generalizing the functionality for reuse. Using scripts ^^^^^^^^^^^^^ -Now let's parameterize our script to accept command-line arguments. +Now let's parameterize our script to accept command-line arguments. With a few generalizations, we can use the include and exclude specs as arguments to create a powerful, general-purpose query script. Open a file named ``find_exclude.py`` in your preferred editor and add the following code: @@ -197,7 +197,7 @@ Using the ``spack-python`` executable What if we want to make our script available for others to use—without requiring them to remember to run it with ``spack python``? -We can take advantage of a shebang line, typically included as the first line in executable Python scripts. +We can take advantage of a shebang line, typically included as the first line in executable Python scripts. However, there's a catch, as we'll see shortly. Open the ``find_exclude.py`` script you created earlier in your preferred editor, and add a shebang line that invokes ``spack python`` using ``env``: @@ -212,9 +212,8 @@ Then exit our editor and add execute permissions to the script before running it :language: console :emphasize-lines: 1-2 -If you're lucky, the script worked on your system—but there's no guarantee. -Some systems only support a single argument on the shebang line (see `here `_). -``spack-python``, a wrapper script for ``spack python``, solves this limitation. +If you're lucky, the script worked on your system—but there's no guarantee. +Some systems only support a single argument on the shebang line (see `here `_). ``spack-python``, a wrapper script for ``spack python``, solves this limitation. Open the script in your editor again and change the ``env`` argument to ``spack-python`` as shown below: @@ -230,7 +229,7 @@ Exit your editor, and let's run the script again: Congratulations! It will now work on any system with Spack installed. -You now have the basic tools to write your own custom Spack queries and prototype new ideas. +You now have the basic tools to write your own custom Spack queries and prototype new ideas. We hope you'll consider contributing them back to Spack in the future. .. LocalWords: LLC Spack's APIs hdf zlib literalinclude json uniq jq diff --git a/tutorial_stacks.rst b/tutorial_stacks.rst index 9ddf3aedc5..16b31f7b0b 100644 --- a/tutorial_stacks.rst +++ b/tutorial_stacks.rst @@ -11,17 +11,16 @@ Stacks Tutorial =============== -So far, we've discussed Spack environments in the context of unified user workflows. +So far, we've discussed Spack environments in the context of unified user workflows. However, Spack environments have much broader capabilities. In this tutorial, we'll explore how to use Spack environments to manage large software deployments. -The main difference between a typical environment for a single user and one used for large deployments is scope: -in the latter case, we often need to install a set of packages across a wide range of MPI implementations, LAPACK libraries, or compilers. +The main difference between a typical environment for a single user and one used for large deployments is scope: in the latter case, we often need to install a set of packages across a wide range of MPI implementations, LAPACK libraries, or compilers. In the following sections, we'll demonstrate how to create a software stack built as a cross-product of different LAPACK and MPI libraries, using a compiler that is newer than the one provided by the host system. -The first part focuses on how to properly configure and install the desired software. +The first part focuses on how to properly configure and install the desired software. We'll learn how to pin certain requirements and how to express a cross-product of specs in a compact and flexible way. Then, we'll consider how the installed software might be consumed by users, and examine the two main mechanisms that Spack provides for that: **views** and **module files**. @@ -39,13 +38,13 @@ Then, we'll consider how the installed software might be consumed by users, and Setup the compiler ------------------ -The first step in building our stack is to set up the compiler we want to use later. +The first step in building our stack is to set up the compiler we want to use later. This is currently an iterative process that can be done in one of two ways: 1. Install the compiler first, then register it in the environment 2. Use a separate environment dedicated to the compiler -In the following, we'll use the first approach. +In the following, we'll use the first approach. For those interested, an example of the second approach can be found `at this link `_. Let's start by creating an environment in a directory of our choice: @@ -53,7 +52,7 @@ Let's start by creating an environment in a directory of our choice: .. literalinclude:: outputs/stacks/setup-0.out :language: console -Now we can add a new compiler from the command line. +Now we can add a new compiler from the command line. We'll also disable view generation for now, as we'll return to this topic later in the tutorial: .. literalinclude:: outputs/stacks/setup-1.out @@ -80,7 +79,7 @@ The ``spack location -i`` command returns the installation prefix for the spec b .. literalinclude:: outputs/stacks/compiler-find-1.out :language: console -This is generally useful when scripting Spack commands, as the example above shows. +This is generally useful when scripting Spack commands, as the example above shows. Listing the compilers now confirms the presence of ``gcc@12.3.0``: .. literalinclude:: outputs/stacks/compiler-list-0.out @@ -97,7 +96,7 @@ We are now ready to build more software with our newly installed GCC! Install a software stack ------------------------ -Now that we have a compiler ready, the next objective is to build software with it. +Now that we have a compiler ready, the next objective is to build software with it. We'll start by adding different versions of ``netlib-scalapack``, each linked against a different MPI implementation: .. literalinclude:: outputs/stacks/unify-0.out @@ -120,34 +119,34 @@ Let's see what that means. Tuning concretizer options for a stack ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Whenever we concretize an environment with more than one root spec, we can configure Spack to be more or less strict about allowing duplicate nodes in the sub-DAG formed by following link and run dependencies from the roots. +Whenever we concretize an environment with more than one root spec, we can configure Spack to be more or less strict about allowing duplicate nodes in the sub-DAG formed by following link and run dependencies from the roots. This subgraph is commonly referred to as the *root unification set*. A diagram may help to better visualize the concept: .. image:: _static/images/stacks-unify.svg -The image above represents our current environment, with the three root specs highlighted using a thicker dashed outline. -Any node that can be reached via a link or run dependency from a root spec is part of the root unification set. +The image above represents our current environment, with the three root specs highlighted using a thicker dashed outline. +Any node that can be reached via a link or run dependency from a root spec is part of the root unification set. Pure build dependencies may fall outside of it. -The configuration option that controls unification behavior is ``concretizer:unify``. +The configuration option that controls unification behavior is ``concretizer:unify``. Let's check its current value: .. literalinclude:: outputs/stacks/unify-2.out :language: console -Setting ``concretizer:unify: true`` means that only one configuration of each package is allowed in the environment. +Setting ``concretizer:unify: true`` means that only one configuration of each package is allowed in the environment. This setting is ideal for single-project environments, since it ensures a unified view of all installed software, resembling a traditional Unix filesystem layout, without the risk of collisions between installations. -However, in our case, this strict unification is not feasible—our root specs already require two different configurations of ``netlib-scalapack``. +However, in our case, this strict unification is not feasible—our root specs already require two different configurations of ``netlib-scalapack``. Let's update the setting to ``false`` and try to re-concretize: .. literalinclude:: outputs/stacks/unify-3.out :language: console -This time, concretization succeeds. -Setting ``concretizer:unify: false`` effectively tells Spack to concretize each root spec independently, then merge the results into the environment. +This time, concretization succeeds. +Setting ``concretizer:unify: false`` effectively tells Spack to concretize each root spec independently, then merge the results into the environment. This allows us to retain multiple versions or configurations of the same package when necessary. .. note:: @@ -158,21 +157,21 @@ This allows us to retain multiple versions or configurations of the same package $ spack config add concretizer:unify:when_possible - With this option, Spack will attempt to unify the environment eagerly by solving it in multiple rounds. - The concretization at round ``n`` includes all specs that could not be unified at round ``n-1``, and considers previous rounds' results for reuse. +With this option, Spack will attempt to unify the environment eagerly by solving it in multiple rounds. +The concretization at round ``n`` includes all specs that could not be unified at round ``n-1``, and considers previous rounds' results for reuse. ^^^^^^^^^^^^^ Spec matrices ^^^^^^^^^^^^^ -Let's expand our stack further to also link against different LAPACK providers. +Let's expand our stack further to also link against different LAPACK providers. We could, of course, add new specs explicitly: .. literalinclude:: outputs/stacks/unify-4.out :language: console -However, this approach becomes tedious as soon as additional software is required. +However, this approach becomes tedious as soon as additional software is required. The best way to express a cross-product like this in Spack is by using a **matrix**: .. literalinclude:: outputs/stacks/examples/2.spack.stack.yaml @@ -214,7 +213,7 @@ As we can see, all four variations of ``netlib-scalapack`` have been successfull Reusable definitions ^^^^^^^^^^^^^^^^^^^^ -So far, we've seen how spec matrices can generate cross-product specs from rows containing lists of constraints. +So far, we've seen how spec matrices can generate cross-product specs from rows containing lists of constraints. In large deployments, it's common to include multiple matrices in the spec list—often sharing some of the same rows. To reduce duplication in the manifest file and lower the maintenance burden, Spack allows you to *define* reusable lists of constraints under the ``definitions`` attribute, and expand them later wherever needed. @@ -230,10 +229,10 @@ Next, let's verify that re-concretizing the environment results in no changes: .. literalinclude:: outputs/stacks/concretize-1.out :language: console -Now we can use those definitions to add, for example, serial packages built against the LAPACK libraries. +Now we can use those definitions to add, for example, serial packages built against the LAPACK libraries. Let's demonstrate this using ``py-scipy`` as an example. -Another useful feature is the ability to exclude specific entries from a cross-product matrix. +Another useful feature is the ability to exclude specific entries from a cross-product matrix. This can be done using the ``exclude`` keyword within the same item as the ``matrix``. Let's exclude ``py-scipy ^netlib-lapack`` from the matrix: @@ -247,7 +246,7 @@ Let's concretize the environment and install the specs again: .. literalinclude:: outputs/stacks/concretize-3.out :language: console -At this point, the environment should contain only ``py-scipy ^openblas``. +At this point, the environment should contain only ``py-scipy ^openblas``. Let's verify that: .. literalinclude:: outputs/stacks/concretize-4.out @@ -257,8 +256,8 @@ Let's verify that: Conditional definitions ^^^^^^^^^^^^^^^^^^^^^^^ -Spec list definitions can also be conditioned using a ``when`` clause. -The ``when`` clause is a Python conditional that is evaluated in a restricted context. +Spec list definitions can also be conditioned using a ``when`` clause. +The ``when`` clause is a Python conditional that is evaluated in a restricted context. The following variables are available for use in ``when`` clauses: ================= ============================================ @@ -274,14 +273,14 @@ Variable Name Value ``hostname`` The hostname of the current node ================= ============================================ -Let's say we want to restrict the MPI provider to just ``mpich``, unless the ``SPACK_STACK_USE_OPENMPI`` environment variable is set. +Let's say we want to restrict the MPI provider to just ``mpich``, unless the ``SPACK_STACK_USE_OPENMPI`` environment variable is set. To accomplish this, we could write the following ``spack.yaml``: .. literalinclude:: outputs/stacks/examples/5.spack.stack.yaml :language: yaml :emphasize-lines: 7-9 -Different definitions for the same list name are concatenated. +Different definitions for the same list name are concatenated. This allows you to define the base list unconditionally and then append additional values conditionally using separate ``when`` blocks. Let's first see what happens when we concretize without setting the environment variable: @@ -289,7 +288,7 @@ Let's first see what happens when we concretize without setting the environment .. literalinclude:: outputs/stacks/concretize-5.out :language: console -As expected, only ``mpich`` is used as the MPI provider. +As expected, only ``mpich`` is used as the MPI provider. To include ``openmpi``, we simply set the appropriate environment variable: .. literalinclude:: outputs/stacks/concretize-6.out @@ -302,21 +301,21 @@ There's no need to reinstall in this case, since all the specs are already prese Other useful features ^^^^^^^^^^^^^^^^^^^^^ -Sometimes it can be helpful to create a local source mirror for the specs installed in an environment. +Sometimes it can be helpful to create a local source mirror for the specs installed in an environment. If the environment is active, this is as simple as: .. code-block:: console $ spack mirror create --all -d ./stacks-mirror -This command fetches all the tarballs for the packages listed in the ``spack.lock`` file and places them in the specified directory. +This command fetches all the tarballs for the packages listed in the ``spack.lock`` file and places them in the specified directory. Later, you can move this mirror to an air-gapped machine and run: .. code-block:: console $ spack mirror add -This allows you to rebuild the specs from source. +This allows you to rebuild the specs from source. If instead you want to create a buildcache, you can do the following: .. code-block:: console @@ -324,10 +323,10 @@ If instead you want to create a buildcache, you can do the following: $ spack gpg create $ spack buildcache push ./mirror -In that case, don't forget to set an appropriate value for the padding of the install tree. +In that case, don't forget to set an appropriate value for the padding of the install tree. See the documentation on `how to set up relocation `_ for details. -By default, Spack installs one package at a time, using the ``-j`` option where possible. +By default, Spack installs one package at a time, using the ``-j`` option where possible. If you are installing a large environment and have access to a powerful build node, you might want to launch multiple builds in parallel to make better use of available resources. This can be done by generating a ``depfile`` while the environment is active: @@ -336,7 +335,7 @@ This can be done by generating a ``depfile`` while the environment is active: $ spack env depfile -o Makefile -This generates a Makefile that starts multiple Spack instances, sharing resources via the GNU jobserver. +This generates a Makefile that starts multiple Spack instances, sharing resources via the GNU jobserver. More details on this feature can be found in the `Spack documentation `_. Using this approach can significantly reduce build time, especially if you frequently build from source. @@ -345,19 +344,19 @@ Using this approach can significantly reduce build time, especially if you frequ Make the software stack easy to use ----------------------------------- -Now that the software stack has been installed, we need to focus on how it can be used by end users. -We'll first look at how to configure views to project a subset of installed specs onto a directory structure that resembles a typical Unix filesystem layout. -Then we'll discuss an alternative approach using module files. +Now that the software stack has been installed, we need to focus on how it can be used by end users. +We'll first look at how to configure views to project a subset of installed specs onto a directory structure that resembles a typical Unix filesystem layout. +Then we'll discuss an alternative approach using module files. Which of these methods is more appropriate depends heavily on the specific use case. ^^^^^^^^^^^^^^^^ View descriptors ^^^^^^^^^^^^^^^^ -At the beginning of this tutorial, we configured Spack not to create a view for this stack. +At the beginning of this tutorial, we configured Spack not to create a view for this stack. That's because simple views won't work well with software stacks: we've been concretizing multiple packages with the same name, and they would conflict if linked into a single view. -Instead, we can create *multiple views* using view descriptors. +Instead, we can create *multiple views* using view descriptors. This allows us to control exactly which packages are included in a view—and how they're laid out. Let's edit our ``spack.yaml`` file again: @@ -368,12 +367,12 @@ Let's edit our ``spack.yaml`` file again: In the configuration above, we define two views: ``default`` and ``full``. -The ``default`` view includes all packages built with ``gcc@12``, but excludes those that depend on ``mpich`` or ``netlib-lapack``. +The ``default`` view includes all packages built with ``gcc@12``, but excludes those that depend on ``mpich`` or ``netlib-lapack``. As shown, we can use both *include* and *exclude* spec constraints in our view descriptors. -The ``full`` view uses a more complex projection layout. -This places each spec in a subdirectory according to the first matching constraint. -The ``all`` pattern acts as a fallback and always has the lowest priority, regardless of where it appears. +The ``full`` view uses a more complex projection layout. +This places each spec in a subdirectory according to the first matching constraint. +The ``all`` pattern acts as a fallback and always has the lowest priority, regardless of where it appears. To avoid confusion, we recommend placing ``all`` last in the projection list. Let's re-concretize the environment to regenerate the views, and check their structure: @@ -381,8 +380,8 @@ Let's re-concretize the environment to regenerate the views, and check their str .. literalinclude:: outputs/stacks/view-0.out :language: console -View descriptors also support a ``link`` key. -By default, Spack links all packages—including implicit link and run dependencies—into the view. +View descriptors also support a ``link`` key. +By default, Spack links all packages—including implicit link and run dependencies—into the view. If we set this option to ``roots``, Spack links only the root packages into the view. .. literalinclude:: outputs/stacks/examples/7.spack.stack.yaml @@ -392,7 +391,7 @@ If we set this option to ``roots``, Spack links only the root packages into the .. literalinclude:: outputs/stacks/view-1.out :language: console -Now, only the root libraries appear in the ``default`` view. +Now, only the root libraries appear in the ``default`` view. All other packages are excluded from the view but remain available in the ``full`` view. You can find the complete documentation on views in the Spack manual `here `_. @@ -401,10 +400,10 @@ You can find the complete documentation on views in the Spack manual `here