diff --git a/tutorial_configuration.rst b/tutorial_configuration.rst index 1c6f786259..8f88cb0951 100644 --- a/tutorial_configuration.rst +++ b/tutorial_configuration.rst @@ -24,18 +24,18 @@ A partial list of some key configuration sections is provided below. * - config - General settings (install location, number of build jobs, etc) * - concretizer - - Specializaiton of the concretizer behavior (reuse, unification, etc) + - Specialization of the concretizer behavior (reuse, unification, etc) * - compilers - Define the compilers that Spack can use (required and system specific) - * - mirrors + * - Mirrors - Locations where spack can look for stashed source or binary distributions - * - packages + * - Packages - Specific settings and rules for packages - * - modules + * - Modules - Naming, location and additional configuration of Spack generated modules The full list of sections can be viewed with ``spack config list``. -For further education we encourage you to explore the spack `documentation on configuration files `_. +For further education, we encourage you to explore the spack `documentation on configuration files `_. The principle goals of this section of the tutorial are: @@ -43,7 +43,7 @@ The principle goals of this section of the tutorial are: 2. Demonstrate how to manipulate configurations 3. Show how to configure system assets with spack (compilers and packages) -As such we will primarily focus on the ``compilers`` and ``packages`` configuration sections in this portion of the tutorial. +As such, we will primarily focus on the ``compilers`` and ``packages`` configuration sections in this portion of the tutorial. We will explain this by first covering how to manipulate configurations from the command line and then show how this impacts the configuration file hierarchy. We will then move into compiler and package configurations to help you develop skills for getting the builds you want on your system. @@ -58,7 +58,7 @@ Configuration from the command line ----------------------------------- You can run ``spack config blame [section]`` at any point in time to see what your current configuration is. -If you omit the section then spack will dump all the configurations settings to your screen. +If you omit the section, then spack will dump all the configurations settings to your screen. Let's go ahead and run this for the ``concretizer`` section. .. code-block:: console @@ -80,10 +80,10 @@ If we rerun ``spack config blame concretizer`` we can see that the change was ap $ spack config blame concretizer -Notice that the reference file on for this option is now different. +Notice that the reference file for this option is now different. This indicates the scope where the configuration was set in, and we will discuss how spack chooses the default scope shortly. For now, it is important to note that the ``spack config`` command accepts an optional ``--scope`` flag so we can be more precise in the configuration process. -This will make more sense after the next section which provides the definition of spack's configuration scopes and their hierarchy. +This will make more sense after the next section which provides the definition of Spack's configuration scopes and their hierarchy. .. _configs-tutorial-scopes: @@ -210,7 +210,7 @@ Compiler Configuration For most tasks, we can use Spack with the compilers auto-detected the first time Spack runs on a system. As discussed in the basic installation tutorial, we can also tell Spack where compilers are located using the ``spack compiler add`` command. -However, in some circumstances we want even more fine-grained control over the compilers available. +However, in some circumstances, we want even more fine-grained control over the compilers available. This section will teach you how to exercise that control using the compilers configuration file. We will start by opening the compilers configuration file: @@ -243,8 +243,8 @@ We start with no active environment, so this will open a ``compilers.yaml`` file paths: cc: /usr/bin/gcc-10 cxx: /usr/bin/g++-10 - f77: usr/bin/gfortran-10 - fc: usr/bin/gfortran-10 + f77: /usr/bin/gfortran-10 + fc: /usr/bin/gfortran-10 flags: {} operating_system: ubuntu22.04 target: x86_64 @@ -308,7 +308,7 @@ We can verify that our new compiler works by invoking it now: This new compiler also works on Fortran codes. -We'll show it by compiling a small package using as a build dependency ``cmake%gcc@11.4.0`` since it is already available in our binary cache: +We'll show this by compiling a small package using ``cmake%gcc@11.4.0`` as a build dependency, since it is already available in our binary cache: .. code-block:: console @@ -521,7 +521,7 @@ The packages configuration file also controls when Spack will build against an e Spack has a ``spack external find`` command that can automatically discover and register externally installed packages. This works for many common build dependencies, but it's also important to know how to do this manually for packages that Spack cannot yet detect. -On these systems we have a pre-installed curl. +On these systems, we have a pre-installed curl. Let's tell Spack about this package and where it can be found: .. code-block:: yaml @@ -591,7 +591,7 @@ Now Spack will be forced to choose the external Curl. This gets slightly more complicated with virtual dependencies. Suppose we don't want to build our own MPI, but we now want a parallel version of HDF5. -Well, fortunately we have MPICH installed on these systems. +Well, fortunately, we have MPICH installed on these systems. Instead of manually configuring an external for MPICH like we did for Curl we will use the ``spack external find`` command. For packages that support this option, this is a useful way to avoid typos and get more accurate external specs. @@ -599,7 +599,7 @@ For packages that support this option, this is a useful way to avoid typos and g .. literalinclude:: outputs/config/0.external_find.out :language: console -To express that we don't want any other MPI installed we can use the virtual ``mpi`` package as a key. +To express that we don't want any other MPI installed, we can use the virtual ``mpi`` package as a key. While we're editing the ``spack.yaml`` file, make sure to configure HDF5 to be able to build with MPI again: .. code-block:: yaml @@ -793,7 +793,7 @@ If we uninstall and reinstall zlib-ng, we see that it now uses only 2 cores: Obviously, if you want to build everything in serial for whatever reason, you would set ``build_jobs`` to 1. -Last we'll unset ``concretizer:reuse:false`` since we'll want to enable concretizer reuse for the rest of this tutorial. +Last, we'll unset ``concretizer:reuse:false`` since we'll want to enable concretizer reuse for the rest of this tutorial. .. code-block:: yaml diff --git a/tutorial_developer_workflows.rst b/tutorial_developer_workflows.rst index 5611f3cc9c..e1196096e4 100644 --- a/tutorial_developer_workflows.rst +++ b/tutorial_developer_workflows.rst @@ -12,7 +12,7 @@ Developer Workflows Tutorial ============================ This tutorial will guide you through the process of using the ``spack develop`` command to develop software from local source code within a spack environment. -With this command spack will manage your dependencies while you focus on testing changes to your library and/or application. +With this command, Spack will manage your dependencies while you focus on testing changes to your library and/or application. ----------------------------- @@ -22,13 +22,13 @@ Installing from local source The ``spack install`` command, as you know, fetches source code from a mirror or the internet before building and installing your package. As developers, we want to build from local source, which we will constantly change, build, and test. -Let's imagine for a second we're working on ``scr``. ``scr`` is a library used to implement scalable checkpointing in application codes. +Let's imagine, for a second, we're working on ``scr``. ``scr`` is a library used to implement scalable checkpointing in application codes. It supports writing/reading checkpoints quickly and efficiently using MPI and high-bandwidth file I/O. -We'd like to test changes to scr within an actual application so we'll test with ``macsio``, a proxy application written to mimic typical HPC I/O workloads. +We'd like to test changes to ``scr`` within an actual application, so we'll test with ``macsio``, a proxy application written to mimic typical HPC I/O workloads. We've chosen ``scr`` and ``macsio`` because together they are quick to build. We'll start by making an environment for our development. -We need to build ``macsio`` with ``scr`` support, and we'd like everything to be built without fortran support for the time being. +We need to build ``macsio`` with ``scr`` support, and we'd like everything to be built without Fortran support for the time being. Let's set up that development workflow. .. literalinclude:: outputs/dev/setup-scr.out @@ -145,8 +145,8 @@ In this case, it will be the 3.1.0 release that we want to write a patch for: .. literalinclude:: outputs/dev/develop-1.out :language: console -The spack develop command marks the package as being a "development" package in the spack.yaml. -This adds a special ``dev_path=`` attribute to the spec for the package, so spack remembers where the source code for this package is located. +The spack develop command marks the package as being a "development" package in the ``spack.yaml``. +This adds a special ``dev_path=`` attribute to the spec for the package, so Spack remembers where the source code for this package is located. The develop command also downloads/checks out the source code for the package. By default, the source code is downloaded into a subdirectory of the environment. You can change the location of this source directory by modifying the ``path:`` attribute of the develop configuration in the environment. @@ -156,13 +156,13 @@ There are a few gotchas with the spack develop command * You often specify the package version manually when specifying a package as a dev package. Spack needs to know the version of the dev package so it can supply the correct flags for the package's build - system. If a version is not supplied then spack will take the maximum version - defined in the package where where `infinity versions `_ like ``develop`` and ``main`` + system. If a version is not supplied, then Spack will take the maximum version + defined in the package where `infinity versions `_ like ``develop`` and ``main`` have a higher value than the numeric versions. * You should ensure a spec for the package you are developing appears in the DAG of at least one of the roots of the environment with the same version that you are developing. ``spack add `` with the matching version you want to develop is a way to ensure - the develop spec is satisfied.the ``spack.yaml`` environments file. This is because - develop specs are not concretization constraints but rather a criteria for adding + the develop spec is satisfied in the ``spack.yaml`` environments file. This is because + develop specs are not concretization constraints but rather criteria for adding the ``dev_path=`` variant to existing spec. * You'll need to re-concretize the environment so that the version number and the ``dev_path=`` attributes are properly added to the @@ -171,13 +171,13 @@ There are a few gotchas with the spack develop command .. literalinclude:: outputs/dev/develop-conc.out :language: console -Now that we have this done, we tell spack to rebuild both ``scr`` and ``macsio`` by running ``spack install``. +Now that we have this done, we tell Spack to rebuild both ``scr`` and ``macsio`` by running ``spack install``. .. literalinclude:: outputs/dev/develop-2.out :language: console This rebuilds ``scr`` from the subdirectory we specified. -If your package uses cmake, spack will build the package in a build directory that matches the hash for your package. +If your package uses CMake, Spack will build the package in a build directory that matches the hash for your package. From here you can change into the appropriate directory and perform your own build/test cycles. Now, we can develop our code. @@ -188,7 +188,7 @@ Let's edit a file and remove the first semi-colon we find. :language: console Once you have a development package, ``spack install`` also works much like "make". -Since spack knows the source code directory of the package, it checks the filetimes on the source directory to see if we've made recent changes. +Since Spack knows the source code directory of the package, it checks the filetimes on the source directory to see if we've made recent changes. If the file times are newer, it will rebuild ``scr`` and any other package that depends on ``scr``. .. literalinclude:: outputs/dev/develop-3.out @@ -202,17 +202,17 @@ Now let's fix it and rebuild directly. .. literalinclude:: outputs/dev/develop-4.out :language: console -You'll notice here that spack rebuilt both ``scr`` and ``macsio``, as expected. +You'll notice here that Spack rebuilt both ``scr`` and ``macsio``, as expected. -Taking advantage of iterative builds with spack requires cooperation from your build system. -When spack performs a rebuild on a development package, it reruns all the build stages for your package without cleaning the source and build directories to a pristine state. +Taking advantage of iterative builds with Spack requires cooperation from your build system. +When Spack performs a rebuild on a development package, it reruns all the build stages for your package without cleaning the source and build directories to a pristine state. If your build system can take advantage of the previously compiled object files then you'll end up with an iterative build. - If your package just uses make, you also should get iterative builds for free when running ``spack develop``. -- If your package uses cmake with the typical ``cmake`` / ``build`` / +- If your package uses CMake with the typical ``cmake`` / ``build`` / ``install`` build stages, you'll get iterative builds for free with - spack because cmake doesn’t modify the filetime on the + Spack because CMake doesn’t modify the filetime on the ``CMakeCache.txt`` file if your cmake flags haven't changed. - If your package uses autoconf, then rerunning the typical ``autoreconf`` stage typically modifies the filetime of @@ -244,7 +244,7 @@ Workflow Summary Use the ``spack develop`` command with an environment to make a reproducible build environment for your development workflow. Spack will set up all the dependencies for you and link all your packages together. -Within a development environment, ``spack install`` works similar to ``make`` in that it will check file times to rebuild the minimum number of spack packages necessary to reflect the changes to your build. +Within a development environment, ``spack install`` works similarly to ``make`` in that it will check file times to rebuild the minimum number of Spack packages necessary to reflect the changes to your build. ------------------- Optional: Tips and Tricks @@ -261,7 +261,7 @@ A list of the options for the ``spack develop`` can be viewed below: Source Code Management ---------- -``spack develop`` allows users to manipulate the source code locations The default behavior is to let spack manage its location and cloning operations, but software developers often want more control over these. +``spack develop`` allows users to manipulate the source code locations The default behavior is to let Spack manage its location and cloning operations, but software developers often want more control over these. The source directory can be set with the ``--path`` argument when calling ``spack develop``. If this directory already exists then ``spack develop`` will not attempt to fetch the code for you. @@ -282,7 +282,7 @@ Navigation and the Build Environment Diving into the build environment was introduced previously in the packaging section with the ``spack build-env scr -- bash`` command. This is a helpful function because it allows you to run commands inside the build environment. In the packages section of the tutorial this was combined with ``spack cd`` to produce a manual build outside of Spack's automated Process. -This command is particularly useful in developer environments -- it allows developers a streamlined workflow when iterating on a single package without the overhead of the ``spack install`` command. +This command is particularly useful in developer environments—it allows developers a streamlined workflow when iterating on a single package without the overhead of the ``spack install`` command. The additional features of the install command are unnecessary when tightly iterating between building and testing a particular package. For example, the workflow modifying ``scr`` that we just went through can be simplified to: @@ -303,7 +303,7 @@ For example, the workflow modifying ``scr`` that we just went through can be sim $ make test $ exit -Working with the build environment and along with spack navigation features provides a nice way to iterate quickly and navigate through the hash heavy spack directory structures. +Working with the build environment and along with Spack navigation features provides a nice way to iterate quickly and navigate through the hash-heavy Spack directory structures. Combinatorics ------------ @@ -327,4 +327,4 @@ While we won't build out this example it illustrates how the ``dev_path`` for `` Now if we want to do most of our incremental builds using the ``Release`` build and periodically check the results using the ``Debug`` build we can combine the workflow from the previous example: dive into the ``Release`` versions build environment using ``spack build-env scr build_type=Release -- bash`` and navigate with ``spack cd -b scr build_type=Release``. Note that since there are two ``scr`` specs in the environment we must distinguish which one we want for these commands. -When we are ready to check our changes for the debug build we can exit out of the build environment subshell, rerun ``spack install`` to rebuild everything, and then inspect the debug build through our method of choice. +When we are ready to check our changes for the debug build, we can exit out of the build environment subshell, rerun ``spack install`` to rebuild everything, and then inspect the debug build through our method of choice. diff --git a/tutorial_environments.rst b/tutorial_environments.rst index c48d20277b..789e649682 100644 --- a/tutorial_environments.rst +++ b/tutorial_environments.rst @@ -34,7 +34,7 @@ Spack environments allow you to readily: * much more. This tutorial introduces the basics of creating and using environments, then explains how to expand, configure, and build software in them. -We will start with the command line interface, then cover editing key environment file directly. +We will start with the command line interface, then cover editing key environment files directly. We will describe the difference between Spack-managed and independent environments, then finish with a section on reproducible builds. ------------------- @@ -130,7 +130,7 @@ Try the usual install command first: .. literalinclude:: outputs/environments/env-fail-install-1.out :language: console -Environments are special in that you must *add* specs to them before installing. ``spack add`` allows us to do queue up several specs to be installed together. +Environments are special in that you must *add* specs to them before installing. ``spack add`` allows us to queue up several specs to be installed together. Let's try it: .. literalinclude:: outputs/environments/env-add-1.out @@ -177,7 +177,7 @@ You can also add and install specs to an environment incrementally. For example: If you create environments incrementally, Spack ensures that already installed roots are not re-concretized. So, adding specs to an environment at a later point in time will not cause existing packages to rebuild. -Do note however that incrementally creating an environment can give you different package versions from an environment created all at once. +Do note, however, that incrementally creating an environment can give you different package versions from an environment created all at once. We will cover this after we've discussed different concretization strategies. Further, there are two other advantages of concretizing and installing an environment all at once: @@ -244,8 +244,8 @@ Now let's try to uninstall ``trilinos`` from ``myproject2`` and review the conte :language: console -We can see that ``trilinos`` won't be uninstalled because it is still referenced in another environment managed by spack. -If we want to remove it from the roots list we need to use ``spack remove``: +We can see that ``trilinos`` won't be uninstalled because it is still referenced in another environment managed by Spack. +If we want to remove it from the roots list, we need to use ``spack remove``: .. literalinclude:: outputs/environments/env-remove-1.out :language: console @@ -299,7 +299,7 @@ There are several important parts of this file: The ``specs`` list should look familiar; these are the specs we've been modifying with ``spack add``. ``concretizer:unify:true``, the default, means that they are concretized *together*, so that there is only one version of each package in the environment. -Other options for ``unify`` are ``false`` and ``when_possible``. ``false`` means that the specs are concretized *independently*, so that there may be multiple versions of the same package in the environment. ``when_possible`` lies between those options. +Other options for ``unify`` are ``false`` and ``when_possible``. ``false`` means that the specs are concretized *independently*, so that there may be multiple versions of the same package in the environment. ``when_possible`` lies between these options. In this case, Spack will unify as many packages in the environment, but will not fail if it cannot unify all of them. @@ -382,7 +382,7 @@ Building in environments ------------------------ Activated environments allow you to invoke any programs installed in them as if they were installed on the system. -In this section we will take advantage of that feature. +In this section, we will take advantage of that feature. Suppose you want to compile some MPI programs. We have an MPI implementation installed in our ``myproject2`` environment, so ``mpicc`` is available in our path. @@ -484,7 +484,7 @@ Managed versus independent environments Environments are either Spack-managed or independent. Both types of environments are defined by their environment files. -So far we have only created managed environments. +So far, we have only created managed environments. This section describes their differences. *Managed environments* are created using ``spack env create ``. diff --git a/tutorial_modules.rst b/tutorial_modules.rst index 22caeab09d..b15b24a5c1 100644 --- a/tutorial_modules.rst +++ b/tutorial_modules.rst @@ -12,7 +12,7 @@ Module Files Tutorial ===================== This tutorial illustrates how Spack can be used to generate module files for the software that has been installed. -Both hierarchical and non-hierarchical deployments will be discussed in details and we will show how to customize the content and naming of each module file. +Both hierarchical and non-hierarchical deployments will be discussed in detail and we will show how to customize the content and naming of each module file. At the end of the tutorial readers should have a clear understanding of: @@ -48,7 +48,7 @@ Build a module tool ^^^^^^^^^^^^^^^^^^^ The first thing that we need is the module tool itself. -In the tutorial we will use ``lmod`` because it can work with both hierarchical and non-hierarchical layouts. +In this tutorial, we will use ``lmod`` because it can work with both hierarchical and non-hierarchical layouts. .. code-block:: console @@ -61,7 +61,7 @@ Installation directories in Spack's store are definitely not easy to remember, b $ . $(spack location -i lmod)/lmod/lmod/init/bash -Now we can re-source the setup file and Spack modules will be put in our module path. +Now we can re-source the setup file, and Spack modules will be put in our module path. .. code-block:: console @@ -70,7 +70,7 @@ Now we can re-source the setup file and Spack modules will be put in our module .. FIXME: this needs bootstrap support for ``lmod`` .. FIXME: check the docs here, update them if necessary -If you need to install Lmod or Environment module you can refer to the documentation `here `_. +If you need to install Lmod or Environment Modules, you can refer to the documentation `here `_. ^^^^^^^^^^^^^^^^^^ @@ -103,7 +103,7 @@ To check which compilers are available you can use ``spack compiler list``: .. literalinclude:: outputs/modules/list-compiler.out :language: console -Finally, when you confirmed ``gcc@12.3.0`` is properly registered, clean the environment with ``spack unload``: +Finally, when you have confirmed ``gcc@12.3.0`` is properly registered, clean the environment with ``spack unload``: .. code-block:: console @@ -196,7 +196,7 @@ There are also some unique `safety features `_ like ``^python^lapack``. +As you can see, it is possible to specify rules that apply only to a restricted set of packages using `anonymous specs `_ like ``^python^lapack``. Here we declare a conflict between any two modules with the same name, so they cannot be loaded together. We also format the names of modules according to compiler, compiler version, and MPI provider name using the `spec format syntax `_. This allows us to match specs by their dependencies, and format them based on their DAGs. @@ -567,7 +567,7 @@ Autoload dependencies ^^^^^^^^^^^^^^^^^^^^^ Spack can also generate module files that contain code to load the dependencies automatically. -You can, for instance generate python modules that load their dependencies by adding the ``autoload`` directive and assigning it the value ``direct``: +You can, for instance, generate python modules that load their dependencies by adding the ``autoload`` directive and assigning it the value ``direct``: .. code-block:: yaml :emphasize-lines: 4,32,33 @@ -622,7 +622,7 @@ In case messages are unwanted during the autoload procedure, it will be sufficie Hierarchical Module Files ------------------------- -So far we worked with non-hierarchical module files, i.e. with module files that are all generated in the same root directory and don't attempt to dynamically modify the ``MODULEPATH``. +So far, we have worked with non-hierarchical module files, i.e., with module files that are all generated in the same root directory and don't attempt to dynamically modify the ``MODULEPATH``. This results in a flat module structure where all the software is visible at the same time: .. literalinclude:: outputs/modules/lmod-intro-avail.out @@ -638,8 +638,8 @@ Even if ``conflicts`` directives are carefully placed in module files, they: - won't enforce a consistent environment, but will just report an error - need constant updates, for instance as soon as a new compiler or MPI library is installed -`Hierarchical module files `_ try to overcome these shortcomings by showing at start-up only a restricted view of what is available on the system: more specifically only the software that has been installed with OS provided compilers. -Among this software there will be other - usually more recent - compilers that, once loaded, will prepend new directories to ``MODULEPATH`` unlocking all the software that was compiled with them. +`Hierarchical module files `_ try to overcome these shortcomings by showing at start-up only a restricted view of what is available on the system: more specifically, only the software that has been installed with OS provided compilers. +Among this software, there will be other -- usually more recent -- compilers that, once loaded, will prepend new directories to ``MODULEPATH``, unlocking all the software that was compiled with them. This "unlocking" idea can then be extended arbitrarily to virtual dependencies, as we'll see in the following section. ^^^^^^^^^^^^^^^^^ @@ -695,7 +695,7 @@ After these modifications your configuration file should look like: .. note:: Double colon in configuration files - The double colon after ``enable`` is intentional and it serves the + The double colon after ``enable`` is intentional, and it serves the purpose of overriding the default list of enabled generators so that only ``lmod`` will be active (see `Overriding entire sections `_ for more details). @@ -728,7 +728,7 @@ Loading that we'll unlock the ``Compiler`` part of the hierarchy: .. literalinclude:: outputs/modules/module-avail-7.out :language: console -The same holds true also for the ``MPI`` part, that you can enable by loading either ``mpich`` or ``openmpi``. +The same holds true also for the ``MPI`` part, which you can enable by loading either ``mpich`` or ``openmpi``. Let's start by loading ``mpich``: .. literalinclude:: outputs/modules/module-avail-8.out @@ -771,7 +771,7 @@ to decide when to show it to the user. The situation becomes more involved as the number of virtual dependencies in the hierarchy increases. We can take advantage of the DAG that Spack maintains for the installed software and solve this combinatorial problem in a clean and automated way. -In some sense Spack's ability to manage this combinatorial complexity makes deeper hierarchies feasible. +In some sense, Spack's ability to manage this combinatorial complexity makes deeper hierarchies feasible. Coming back to our example, let's add ``lapack`` to the hierarchy and remove the remaining suffix projection for ``lapack``: @@ -866,7 +866,7 @@ and can be extended by users to employ custom templates, as we'll see next. Extend the default templates ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Let's assume one of our software is protected by group membership: allowed users belong to the same linux group, and access is granted at group level. +Let's assume one of our software is protected by group membership: allowed users belong to the same Linux group, and access is granted at the group level. Wouldn't it be nice if people that are not yet entitled to use it could receive a helpful message at module load time that tells them who to contact in your organization to be inserted in the group? To automate the generation of module files with such site-specific behavior we'll start by extending the list of locations where Spack looks for module files. @@ -973,7 +973,7 @@ we'll find the following at the end of each ``netlib-scalapack`` module file: ) end -and every user that doesn't have access to the software will now be redirected to the right e-mail address where to ask for it! +and every user that doesn't have access to the software will now be redirected to the right e-mail address where they can ask for it! ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Restore settings for future sections