diff --git a/tutorial_advanced_packaging.rst b/tutorial_advanced_packaging.rst index b6540136bf..5cf2772922 100644 --- a/tutorial_advanced_packaging.rst +++ b/tutorial_advanced_packaging.rst @@ -10,11 +10,11 @@ Advanced Topics in Packaging ============================ -Spack tries to automatically configure packages with information from dependencies such that all you need to do is to list the dependencies (i.e., with the ``depends_on`` directive) and the build system (for example by deriving from ``CmakePackage``). +Spack tries to automatically configure packages with information from dependencies such that all we will need to do is to list the dependencies (i.e., with the ``depends_on`` directive) and the build system (for example by deriving from ``CmakePackage``). However, there are many special cases. -Often you need to retrieve details about dependencies to set package-specific configuration options, or to define package-specific environment variables used by the package's build system. -This tutorial covers how to retrieve build information from dependencies, and how you can automatically provide important information to dependents in your package. +Often we will need to retrieve details about dependencies to set package-specific configuration options, or to define package-specific environment variables used by the package's build system. +This tutorial covers how to retrieve build information from dependencies, and how we can automatically provide important information to dependents in your package. ---------------------- Setup for the Tutorial @@ -33,7 +33,7 @@ These package definitions are stored in a separate package repository, which can $ spack repo add --scope=site var/spack/repos/tutorial This section of the tutorial may also require a newer version of gcc. -If you have not already installed gcc @7.2.0 and added it to your configuration, you can do so with: +If we have not already installed gcc @7.2.0 and added it to your configuration, we can do so with: .. code-block:: console @@ -41,7 +41,7 @@ If you have not already installed gcc @7.2.0 and added it to your configuration, $ spack compiler add --scope=site `spack location -i gcc@7.2.0 %gcc@5.4.0` If you are using the tutorial docker image, all dependency packages will have been installed. -Otherwise, to install these packages you can use the following commands: +Otherwise, to install these packages we can use the following commands: .. code-block:: console @@ -49,13 +49,13 @@ Otherwise, to install these packages you can use the following commands: $ spack install netlib-lapack $ spack install mpich -Now, you are ready to set your preferred ``EDITOR`` and continue with the rest of the tutorial. +Now, we are ready to set our preferred ``EDITOR`` and continue with the rest of the tutorial. .. note:: - Several of these packages depend on an MPI implementation. You can use - OpenMPI if you install it from scratch, but this is slow (>10 min.). - A binary cache of MPICH may be provided, in which case you can force + Several of these packages depend on an MPI implementation. We can use + OpenMPI if we install it from scratch, but this is slow (>10 min.). + A binary cache of MPICH may be provided, in which case we can force the package to use it and install quickly. All tutorial examples with packages that depend on MPICH include the spec syntax for building with it. @@ -97,7 +97,7 @@ We can practice by editing the ``mpich`` package to set the ``MPICC`` environmen root@advanced-packaging-tutorial:/# spack edit mpich -Once you're finished, the method should look like this: +Once we're finished, the method should look like this: .. code-block:: python @@ -113,7 +113,7 @@ Once you're finished, the method should look like this: spack_env.set('MPICH_F90', spack_fc) spack_env.set('MPICH_FC', spack_fc) -At this point we can, for instance, install ``netlib-scalapack`` with ``mpich``: +At this point, we will, for instance, install ``netlib-scalapack`` with ``mpich``: .. code-block:: console @@ -198,7 +198,7 @@ In the end your method should look like: spack_env.append_flags('LDFLAGS', spec['lapack'].libs.search_flags) spack_env.append_flags('LIBS', spec['lapack'].libs.link_flags) -At this point it's possible to proceed with the installation of ``elpa ^mpich`` +At this point, it's possible to proceed with the installation of ``elpa ^mpich`` ------------------------------ Retrieving Library Information @@ -213,7 +213,7 @@ This section covers how to retrieve library information from dependencies and ho Accessing dependency libraries ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -If you need to access the libraries of a dependency, you can do so via the ``libs`` property of the spec, for example in the ``arpack-ng`` package: +If we need to access the libraries of a dependency, we can do so via the ``libs`` property of the spec, for example in the ``arpack-ng`` package: .. code-block:: python @@ -229,13 +229,13 @@ If you need to access the libraries of a dependency, you can do so via the ``lib Note that ``arpack-ng`` is querying virtual dependencies, which Spack automatically resolves to the installed implementation (e.g. ``openblas`` for ``blas``). We've started work on a package for ``armadillo``. -You should open it, read through the comment that starts with ``# TUTORIAL:`` and complete the ``cmake_args`` section: +We should open it, read through the comment that starts with ``# TUTORIAL:`` and complete the ``cmake_args`` section: .. code-block:: console root@advanced-packaging-tutorial:/# spack edit armadillo -If you followed the instructions in the package, when you are finished your ``cmake_args`` method should look like: +If we followed the instructions in the package, when we are finished our ``cmake_args`` method should look like: .. code-block:: python @@ -256,10 +256,10 @@ If you followed the instructions in the package, when you are finished your ``cm '-DDETECT_HDF5={0}'.format('ON' if '+hdf5' in spec else 'OFF') ] -As you can see, getting the list of libraries that your dependencies provide is as easy as accessing the their ``libs`` attribute. -Furthermore, the interface remains the same whether you are querying regular or virtual dependencies. +As we can see, getting the list of libraries that our dependencies provide is as easy as accessing the their ``libs`` attribute. +Furthermore, the interface remains the same whether we are querying regular or virtual dependencies. -At this point you can complete the installation of ``armadillo`` using ``openblas`` as a LAPACK provider (``armadillo ^openblas ^mpich``): +At this point, we can complete the installation of ``armadillo`` using ``openblas`` as a LAPACK provider (``armadillo ^openblas ^mpich``): .. code-block:: console @@ -280,7 +280,7 @@ At this point you can complete the installation of ``armadillo`` using ``openbla Fetch: 0.01s. Build: 3.96s. Total: 3.98s. [+] /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/armadillo-8.100.1-n2eojtazxbku6g4l5izucwwgnpwz77r4 -Hopefully the installation went fine and the code we added expanded to the right list of semicolon separated libraries (you are encouraged to open ``armadillo``'s build logs to double check). +Hopefully the installation went fine and the code we added expanded to the right list of semicolon separated libraries (we are encouraged to open ``armadillo``'s build logs to double check). ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Providing libraries to dependents @@ -338,7 +338,7 @@ Let's edit it: root@advanced-packaging-tutorial:/# spack edit netlib-lapack and follow the instructions in the ``# TUTORIAL:`` comment as before. -What we need to implement is: +What we will implement is: .. code-block:: python @@ -353,7 +353,7 @@ i.e., a property that returns the correct list of libraries for the LAPACK inter We use the name ``lapack_libs`` rather than ``libs`` because ``netlib-lapack`` can also provide ``blas``, and when it does it is provided as a separate library file. Using this name ensures that when dependents ask for ``lapack`` libraries, ``netlib-lapack`` will retrieve only the libraries associated with the ``lapack`` interface. -Now we can finally install ``armadillo ^netlib-lapack ^mpich``: +Now we will finally install ``armadillo ^netlib-lapack ^mpich``: .. code-block:: console @@ -379,8 +379,8 @@ Attach attributes to other packages ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Build tools also usually provide a set of executables that can be used when another package is being installed. -Spack gives you the opportunity to monkey-patch dependent modules and attach attributes to them. -This helps make the packager's experience as similar as possible to what would have been the manual installation of the same package. +Spack gives us the opportunity to monkey-patch dependent modules and attach attributes to them. +This helps make our experience as similar as possible to what would have been the manual installation of the same package. An example here is the ``automake`` package, which overrides :py:func:`setup_dependent_package `: @@ -431,13 +431,13 @@ Let's look at an example and try to install ``netcdf ^mpich``: /usr/local/var/spack/stage/netcdf-4.4.1.1-gk2xxhbqijnrdwicawawcll4t3c7dvoj/netcdf-4.4.1.1/spack-build-out.txt We can see from the error that ``netcdf`` needs to know how to link the *high-level interface* of ``hdf5``, and thus passes the extra parameter ``hl`` after the request to retrieve it. -Clearly the implementation in the ``hdf5`` package is not complete, and we need to fix it: +Clearly the implementation in the ``hdf5`` package is not complete, and we will fix it: .. code-block:: console root@advanced-packaging-tutorial:/# spack edit hdf5 -If you followed the instructions correctly, the code added to the ``lib`` property should be similar to: +If we followed the instructions correctly, the code added to the ``lib`` property should be similar to: .. code-block:: python :emphasize-lines: 1 @@ -451,7 +451,7 @@ If you followed the instructions correctly, the code added to the ``lib`` proper ) where we highlighted the line retrieving the extra parameters. -Now we can successfully complete the installation of ``netcdf ^mpich``: +Now we will successfully complete the installation of ``netcdf ^mpich``: .. code-block:: console diff --git a/tutorial_basics.rst b/tutorial_basics.rst index 5c9aeb082e..aaa135cba3 100644 --- a/tutorial_basics.rst +++ b/tutorial_basics.rst @@ -8,8 +8,8 @@ Basic Installation Tutorial ========================================= -This tutorial will guide you through the process of installing software using Spack. -We will first cover the ``spack install`` command, focusing on the power of the spec syntax and the flexibility it gives to users. +This tutorial will guide us through the process of installing software using Spack. +First, we will cover the ``spack install`` command, focusing on the power of the spec syntax and the flexibility it gives to users. We will also cover the ``spack find`` command for viewing installed packages and the ``spack uninstall`` command for uninstalling them. Finally, we will touch on how Spack manages compilers, especially as it relates to using Spack-built compilers within Spack. We will include full output from all of the commands demonstrated, although we will frequently call attention to only small portions of that output (or merely to the fact that it succeeded). @@ -35,7 +35,7 @@ Spack has some nice command-line integration tools, so instead of simply prepend $ . share/spack/setup-env.sh -You're good to go! +We're good to go! ----------------- What is in Spack? @@ -72,20 +72,20 @@ Let's go ahead and install ``gmake``, .. literalinclude:: outputs/basics/gmake.out :language: console -You will see Spack installed ``gmake``, ``gcc-runtime``, and ``glibc``. +We will see Spack installed ``gmake``, ``gcc-runtime``, and ``glibc``. The ``glibc`` and ``gcc-runtime`` packages are automatically tracked by Spack to manage consistency requirements among compiler runtimes. They do not represent separate software builds from source, but are records of the system's compiler runtime components Spack used for the install. For the rest of this section, we will ignore these components and focus on the packages explicitly installed. Spack can install software either from source or from a binary cache. Packages in the binary cache are signed with GPG for security. -For the tutorial we have prepared a binary cache so you don't have to wait on slow compilation from source. +For the tutorial we have prepared a binary cache so we don't have to wait on slow compilation from source. To be able to install from the binary cache, we will need to configure Spack with the location of the binary cache and trust the GPG key that the binary cache was signed with. .. literalinclude:: outputs/basics/mirror.out :language: console -You'll learn more about configuring Spack later in the tutorial, but for now you will be able to install the rest of the packages in the tutorial from a binary cache using the same ``spack install`` command. +We'll learn more about configuring Spack later in the tutorial, but for now we will be able to install the rest of the packages in the tutorial from a binary cache using the same ``spack install`` command. By default this will install the binary cached version if it exists and fall back on installing from source if it does not. Spack's "spec" syntax is the interface by which we can request specific configurations of a package. @@ -97,7 +97,7 @@ The ``%`` sigil is used to specify compilers. Note that this installation is located separately from the previous one. We will discuss this in more detail later, but this is part of what allows Spack to support many versions of software packages. -You can check for particular versions before requesting them. +We can check for particular versions before requesting them. We will use the ``spack versions`` command to see the available versions, and then install a different version of ``zlib-ng``. .. literalinclude:: outputs/basics/versions-zlib.out @@ -149,8 +149,8 @@ Anything we could specify about the top-level package, we can also specify about Packages can also be referred to from the command line by their package hash. Using the ``spack find -lf`` command earlier we saw that the hash of our optimized installation of zlib-ng (``cflags="-O3"``) began with ``umrbkwv``. We can now explicitly build with that package without typing the entire spec, by using the ``/`` sigil to refer to it by hash. -As with other tools like Git, you do not need to specify an *entire* hash on the command line. -You can specify just enough digits to identify a hash uniquely. +As with other tools like Git, we do not need to specify an *entire* hash on the command line. +We can specify just enough digits to identify a hash uniquely. If a hash prefix is ambiguous (i.e., two or more installed packages share the prefix) then Spack will report an error. .. literalinclude:: outputs/basics/tcl-zlib-hash.out @@ -164,7 +164,7 @@ Note that each package has a top-level entry, even if it also appears as a depen Let's move on to slightly more complicated packages. HDF5 is a good example of a more complicated package, with an MPI dependency. -If we install it with default settings it will build with OpenMPI. +If we install it with default settings, it will build with OpenMPI. .. literalinclude:: outputs/basics/hdf5.out :language: console @@ -192,7 +192,7 @@ The partial spec ``^mpi@3`` can be satisfied by any of several MPI implementatio .. literalinclude:: outputs/basics/hdf5-hl-mpi.out :language: console -We'll do a quick check in on what we have installed so far. +Let's do a quick check in on what we have installed so far. .. literalinclude:: outputs/basics/find-ldf-2.out :language: console @@ -234,7 +234,7 @@ Again, the ``spack graph`` command shows the full DAG of the dependency informat .. literalinclude:: outputs/basics/graph-trilinos.out :language: console -You can control how the output is displayed with a number of options. +We can control how the output is displayed with a number of options. The ASCII output from ``spack graph`` can be difficult to parse for complicated packages. The output can be changed to the Graphviz ``.dot`` format using the ``--dot`` flag. @@ -250,7 +250,7 @@ Uninstalling Packages --------------------- Earlier we installed many configurations each of zlib-ng and Tcl. -Now we will go through and uninstall some of those packages that we didn't really need. +Now, let's go through and uninstall some of those packages that we didn't really need. .. literalinclude:: outputs/basics/find-d-tcl.out :language: console @@ -269,8 +269,8 @@ We can uninstall packages by spec using the same syntax as install. We can also uninstall packages by referring only to their hash. We can use either the ``--force`` (or ``-f``) flag or the ``--dependents`` (or ``-R``) flag to remove packages that are required by another installed package. -Use ``--force`` to remove just the specified package, leaving dependents broken. -Use ``--dependents`` to remove the specified package and all of its dependents. +Let's use ``--force`` to remove just the specified package, leaving dependents broken. +Let's use ``--dependents`` to remove the specified package and all of its dependents. .. literalinclude:: outputs/basics/uninstall-needed.out :language: console @@ -291,7 +291,7 @@ The ``--all`` (or ``-a``) flag can be used to uninstall all packages matching an Advanced ``spack find`` Usage ----------------------------- -We will go over some additional uses for the ``spack find`` command not already covered in the :ref:`basics-tutorial-install` and +Let's go over some additional uses for the ``spack find`` command not already covered in the :ref:`basics-tutorial-install` and :ref:`basics-tutorial-uninstall` sections. The ``spack find`` command can accept what we call "anonymous specs." These are expressions in spec syntax that do not contain a package name. @@ -321,7 +321,7 @@ The ``spack compilers`` command is an alias for ``spack compiler list``. :language: console The compilers are maintained in a YAML file (``compilers.yaml``). -Later in the tutorial, you will learn how to configure compilers by hand for special cases. +Later in the tutorial, we will learn how to configure compilers by hand for special cases. Spack also has tools to add compilers, and compilers built with Spack can be added to the configuration. .. literalinclude:: outputs/basics/install-gcc-12.1.0.out @@ -332,7 +332,7 @@ Spack also has tools to add compilers, and compilers built with Spack can be add We can add GCC to Spack as an available compiler using the ``spack compiler add`` command. This will allow future packages to build with ``gcc@12.3.0``. -To avoid having to copy and paste GCC's path, we can use ``spack location -i`` to get the installation prefix. +To avoid having to copy and paste GCC's path, we will use ``spack location -i`` to get the installation prefix. .. literalinclude:: outputs/basics/compiler-add-location.out :language: console diff --git a/tutorial_binary_cache.rst b/tutorial_binary_cache.rst index 3c7fe66311..9cc32c875c 100644 --- a/tutorial_binary_cache.rst +++ b/tutorial_binary_cache.rst @@ -10,7 +10,7 @@ Binary Caches Tutorial ================================== -In this section of the tutorial, you will learn how to share Spack-built binaries across machines and users using build caches. +In this section of the tutorial, we will learn how to share Spack-built binaries across machines and users using build caches. We will explore a few concepts that apply to all types of build caches, but the focus is primarily on **OCI container registries** (like Docker Hub or Github Packages) as a storage backend for binary caches. Spack supports a range of storage backends, such as an ordinary filesystem, Amazon S3, and Google Cloud Storage, but OCI build caches have a few interesting properties that make them worth exploring more in-depth. @@ -32,21 +32,21 @@ Let's run the ``julia`` REPL julia> 1 + 1 2 -Now we'd like to share these executables with other users. -First we will focus on sharing the binaries with other *Spack* users, and later we will see how users completely unfamiliar with Spack can easily use the applications too. +Now, let's share these executables with other users. +First, we will focus on sharing the binaries with other *Spack* users, and later we will see how users completely unfamiliar with Spack can easily use the applications too. ------------------------------------------------ Setting up an OCI build cache on GitHub Packages ------------------------------------------------ -For this tutorial we will be using GitHub Packages as an OCI registry, since most people have a GitHub account and it's easy to use. +For this tutorial, we will be using GitHub Packages as an OCI registry, since most people have a GitHub account and it's easy to use. -First, go to ``_ to generate a Personal Access Token (classic) with ``write:packages`` permissions. -Copy this token. +First, let's go to ``_ to generate a Personal Access Token (classic) with ``write:packages`` permissions. +Let's copy this token. Next, we will add this token to the mirror configuration section for the Spack environment. -Replace `` with your GitHub username and `` with your GitHub username or an organization where you have permission to create packages. -The build cache name, `buildcache-${USER}-${HOSTNAME}`, is a suggestion; you can choose your own. +Let's replace `` with our GitHub username and `` with our GitHub username or an organization where we have permission to create packages. +The build cache name, `buildcache-${USER}-${HOSTNAME}`, is a suggestion; we can choose our own. .. code-block:: console @@ -65,7 +65,7 @@ We talk about mirrors and build caches almost interchangeably, because every bui Source mirrors exist too, which we will not cover in this tutorial. -Your ``spack.yaml`` file should now contain the following: +Our ``spack.yaml`` file should now contain the following: .. code-block:: yaml @@ -112,8 +112,8 @@ look very similar to a container image --- we will get to that in a bit. .. note :: -Binaries pushed to GitHub packages are ``private`` by default, which means you need a token to download them. -You can change the visibility to ``public`` by going to GitHub Packages from your GitHub account, selecting the ``buildcache`` package, go to ``package settings``, and change the visibility to ``public`` in the ``Danger Zone`` section. +Binaries pushed to GitHub packages are ``private`` by default, which means we need a token to download them. +We can change the visibility to ``public`` by going to GitHub Packages from our GitHub account, selecting the ``buildcache`` package, go to ``package settings``, and change the visibility to ``public`` in the ``Danger Zone`` section. This page can also be directly accessed by going to .. code-block:: text @@ -125,7 +125,7 @@ This page can also be directly accessed by going to Installing from the build cache ------------------------------- -We will now verify that the build cache works by reinstalling ``julia``. +Now, let's verify that the build cache works by reinstalling ``julia``. Let's make sure that we *only* use the build cache that we just created, and not the builtin one that is configured for the tutorial. The easiest way to do this is to override the ``mirrors`` config section in the environment by using a double colon in the ``spack.yaml`` file: @@ -154,7 +154,7 @@ An "overwrite install" should be enough to show that the build cache is used (ou [+] /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl Two blobs are fetched for each spec: a metadata file and the actual binary package. -If you've used ``docker pull`` or other container runtimes before, these types of hashes may look familiar. +If we've used ``docker pull`` or other container runtimes before, these types of hashes may look familiar. OCI registries are content addressed, which means that we see hashes like these instead of human-readable file names. ------------------------------------ @@ -164,16 +164,16 @@ Reuse of binaries from a build cache Spack's concretizer optimizes for **reuse**. This means that it will avoid source builds if it can use specs for which binaries are readily available. -In the previous example we managed to install packages from our build cache, but we did not concretize our environment again. +In the previous example, we managed to install packages from our build cache, but we did not concretize our environment again. Users on other machines with different distributions will have to concretize, and therefore we should make sure that the build cache is indexed so that the concretizer can take it into account. -This can be done by running +We can do this by running .. code-block:: console $ spack -e . buildcache update-index my-mirror This operation can take a while for large build caches, since it fetches all metadata of available packages. -For convenience you can also run ``spack buildcache push --update-index ...`` to avoid a separate step. +For convenience, we can also run ``spack buildcache push --update-index ...`` to avoid a separate step. .. note:: @@ -194,7 +194,7 @@ Creating runnable container images ---------------------------------- The build cache we have created uses an OCI registry, which is the same technology that is used to store container images. -So far we have used this build cache as any other build cache: the concretizer can use it to avoid source builds, and ``spack install`` will fetch binaries from it. +So far, we have used this build cache as any other build cache: the concretizer can use it to avoid source builds, and ``spack install`` will fetch binaries from it. However, we can also use this build cache to share binaries directly as runnable container images. @@ -208,7 +208,7 @@ We can already attempt to run the image associated with the ``julia`` package th but immediately we see it fails. The reason is that one crucial part is missing, and that is ``glibc``, which Spack always treats as an external package. -To fix this, we force push to the registry again, but this time we specify a base image with a recent version of ``glibc``, for example from ``ubuntu:24.04``: +To fix this, let's force push to the registry again, but this time we will specify a base image with a recent version of ``glibc``, for example from ``ubuntu:24.04``: .. code-block:: console @@ -216,7 +216,7 @@ To fix this, we force push to the registry again, but this time we specify a bas ... ==> Pushed julia@1.9.3/dfzhutf to ghcr.io//buildcache:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack -Now let's pull this image again and run it: +Now, let's pull this image again and run it: .. code-block:: console @@ -229,7 +229,7 @@ Now let's pull this image again and run it: This time it works! The minimal ``ubuntu:24.04`` image provides us not only with ``glibc``, but also other utilities like a shell. -Notice that you can use any base image of choice, like ``fedora`` or ``rockylinux``. +Notice that we can use any base image of choice, like ``fedora`` or ``rockylinux``. The only constraint is that it has a ``libc`` compatible with the external ``libc`` Spack used to build the binaries. Spack does not validate this. @@ -237,8 +237,8 @@ Spack does not validate this. Spack environments as container images -------------------------------------- -The previous container image is a good start, but it would be nice to add some more utilities to the image. -If you've paid attention to the output of some of the commands we have run so far, you may have noticed that Spack generates exactly one image tag for each package it pushes to the registry. +The previous container image is a good start, but let's add some more utilities to the image. +If we've paid attention to the output of some of the commands we have run so far, we may have noticed that Spack generates exactly one image tag for each package it pushes to the registry. Every Spack package corresponds to a single layer in each image, and the layers are shared across the different image tags. Because Spack installs every package into a unique prefix, it is incredibly easy to compose multiple packages into a container image. @@ -248,14 +248,14 @@ Let's add a simple text editor like ``vim`` to our previous environment next to .. note:: - You may want to change ``mirrors::`` to ``mirrors:`` in the ``spack.yaml`` file to avoid + We may want to change ``mirrors::`` to ``mirrors:`` in the ``spack.yaml`` file to avoid a source build of ``vim`` --- but a source build should be quick. .. code-block:: console $ spack -e . install --add vim -This time, when we push to the OCI registry, we also pass ``--tag julia-and-vim`` to instruct Spack to create an additional image tag for the environment as a whole, with a more human-readable name: +This time, when we push to the OCI registry, let's also pass ``--tag julia-and-vim`` to instruct Spack to create an additional image tag for the environment as a whole, with a more human-readable name: .. code-block:: console @@ -263,7 +263,7 @@ This time, when we push to the OCI registry, we also pass ``--tag julia-and-vim` $ spack -e . buildcache push --base-image ubuntu:24.04 --tag julia-and-vim my-mirror ==> Tagged ghcr.io//buildcache:julia-and-vim -Now let's run a container from this image: +Now, let's run a container from this image: .. code-block:: console @@ -295,24 +295,24 @@ This approach is still valid, and the ``spack containerize`` command continues t * In certain CI environments, it is not possible to use ``docker build`` directly. For example, the CI script itself may already run in a Docker container, and running ``docker build`` *safely* inside a container (Docker-in-Docker) is tricky. The takeaway is that Spack decouples the steps that ``docker build`` combines: build isolation, running the build, and creating an image. -You can run ``spack install`` on your host machine or in a container, and run ``spack buildcache push`` separately to create an image. +We can run ``spack install`` on our host machine or in a container, and run ``spack buildcache push`` separately to create an image. ---------- Relocation ---------- -Spack is different from many package managers in that it lets users choose where to install packages. -This makes Spack very flexible, as users can install packages in their home directory and do not need root privileges. +Spack is different from many package managers in that it lets us choose where to install packages. +This makes Spack very flexible, as we can install packages in our home directory and do not need root privileges. The downside is that sharing binaries is more complicated, as binaries may contain hard-coded, absolute paths to machine specific locations, which have to be adjusted when these binaries are installed on a different machine or in a different path. Fortunately Spack handles this automatically upon install from a binary cache. -But when you build binaries that are intended to be shared, there is one thing you have to keep in mind: Spack can relocate hard-coded paths in binaries *provided that the target prefix is shorter than the prefix used during the build*. +But when we build binaries that are intended to be shared, there is one thing we have to keep in mind: Spack can relocate hard-coded paths in binaries *provided that the target prefix is shorter than the prefix used during the build*. The reason is that binaries typically embed these absolute paths in string tables, which is a list of null-terminated strings, to which the program stores offsets. That means we can only modify strings in-place, and if the new path is longer than the old one, we would overwrite the next string in the table. -To maximize the chances of successful relocation, you should build your binaries in a relatively long path. -Fortunately Spack can automatically pad paths to make them longer, using the following command: +To maximize the chances of successful relocation, we should build our binaries in a relatively long path. +Fortunately, Spack can automatically pad paths to make them longer, using the following command: .. code-block:: console @@ -323,9 +323,9 @@ Using build caches in CI ------------------------ Build caches are a great way to speed up CI pipelines. -Both GitHub Actions and GitLab CI support container registries, and this tutorial should give you a good starting point to leverage them. +Both GitHub Actions and GitLab CI support container registries, and this tutorial should give us a good starting point to leverage them. -Spack also provides a basic GitHub Action that already provides you with a binary cache: +Spack also provides a basic GitHub Action that already provides us with a binary cache: .. code-block:: yaml @@ -337,13 +337,13 @@ Spack also provides a basic GitHub Action that already provides you with a binar uses: spack/setup-spack@v2 - run: spack install python # uses a shared build cache -and the `setup-spack readme `_ shows you how to cache further binaries that are not in the shared build cache. +and the `setup-spack readme `_ shows us how to cache further binaries that are not in the shared build cache. ------- Summary ------- -In this tutorial we have created a build cache on top of an OCI registry, which can be used +In this tutorial, we have created a build cache on top of an OCI registry, which can be used * to run ``spack install julia vim`` on machines and have Spack fetch pre-built binaries instead of building from source. * to automatically create container images for individual packages when pushing to the cache. diff --git a/tutorial_buildsystems.rst b/tutorial_buildsystems.rst index d17bfbb95e..cfa5a8df0e 100644 --- a/tutorial_buildsystems.rst +++ b/tutorial_buildsystems.rst @@ -10,10 +10,10 @@ Spack Package Build Systems =========================== -You may begin to notice after writing a couple of package template files that a pattern emerges for some packages. -For example, you may find yourself writing an ``install()`` method that invokes: ``configure``, ``cmake``, ``make``, ``make install``. -You may also find yourself writing ``"prefix=" + prefix`` as an argument to ``configure`` or ``cmake``. -Rather than having you repeat these lines for all packages, Spack has classes that can take care of these patterns. +We may begin to notice after writing a couple of package template files that a pattern emerges for some packages. +For example, we may find ourselves writing an ``install()`` method that invokes: ``configure``, ``cmake``, ``make``, ``make install``. +We may also find ourselves writing ``"prefix=" + prefix`` as an argument to ``configure`` or ``cmake``. +Rather than have us repeat these lines for all packages, Spack has classes that can take care of these patterns. In addition, these package files allow for finer-grained control of these build systems. In this section, we will describe each build system and give examples on how these can be used to simplify packaging. @@ -49,16 +49,16 @@ In the following sections, we will go over examples of how to utilize each subcl Package ----------------- -We've already seen examples of using the generic ``Package`` class in our walkthrough for writing package files, so we won't be spending much time with it here. +We've already seen examples of using the generic ``Package`` class in our walkthrough for writing package files, so we won't spend much time with it here. Briefly, the Package class allows for arbitrary control over the build process, whereas subclasses rely on certain patterns (e.g. ``configure`` ``make`` ``make install``) to be useful. -The generic ``Package`` class is particularly useful for packages that have a non-conventional build process, as it allows the packager to use Spack's helper functions to customize the building and installing of a package fully. +The generic ``Package`` class is particularly useful for packages that have a non-conventional build process, as it allows us to use Spack's helper functions to customize the building and installing of a package fully. ------------------- Autotools ------------------- As we have seen earlier, packages using ``Autotools`` use ``configure``, ``make`` and ``make install`` commands to execute the build and install process. -In our ``Package`` class, your typical build incantation will consist of the following: +In our ``Package`` class, our typical build incantation will consist of the following: .. code-block:: python @@ -67,7 +67,7 @@ In our ``Package`` class, your typical build incantation will consist of the fol make() make("install") -You'll see that this looks similar to what we wrote in our packaging tutorial. +We'll see that this looks similar to what we wrote in our packaging tutorial. The ``AutotoolsPackage`` subclass aims to simplify writing package files for Autotools-based software and provides convenient methods to manipulate each of the different phases for an ``Autotools`` build system. @@ -87,7 +87,7 @@ Let's take a quick look at some of the internals of the ``Autotools`` class: $ spack edit --build-system autotools -This will open the ``AutotoolsPackage`` file in your text editor. +This will open the ``AutotoolsPackage`` file in our text editor. .. note:: The examples showing code for these classes are abridged to avoid having @@ -121,8 +121,8 @@ Here we see that the ``--prefix`` argument is already included since it is a com Therefore, we typically only need to override the ``configure_args()`` method to return a list of additional arguments. The ``configure()`` method will then append these to the standard arguments. -Packagers also have the option to run ``autoreconf`` in case a package needs to update the build system and generate a new ``configure``. -However, for the most part this will be unnecessary. +We also have the option to run ``autoreconf`` in case a package needs to update the build system and generate a new ``configure``. +However, for the most part, this will be unnecessary. Let's look at the ``mpileaks`` package.py file that we worked on earlier: @@ -131,14 +131,14 @@ Let's look at the ``mpileaks`` package.py file that we worked on earlier: $ spack edit mpileaks Notice that mpileaks was originally written as a generic ``Package`` but uses the ``Autotools`` build system. -Although this package is acceptable, let's covert it to an ``AutotoolsPackage`` to simplify it further. +Although this package is acceptable, let's convert it to an ``AutotoolsPackage`` to simplify it further. .. literalinclude:: tutorial/examples/Autotools/0.package.py :language: python :emphasize-lines: 9 :linenos: -We first inherit from the ``AutotoolsPackage`` class. +First, we will inherit from the ``AutotoolsPackage`` class. Although we could keep the ``install()`` method, most of it can be handled by the ``AutotoolsPackage`` base class. @@ -158,16 +158,16 @@ This version of the ``mpileaks`` package installs the same as the previous, but Makefile ----------------- -Packages that utilize ``Make`` or a ``Makefile`` usually require you to edit a ``Makefile`` to set up platform and compiler-specific variables. +Packages that utilize ``Make`` or a ``Makefile`` usually require us to edit a ``Makefile`` to set up platform and compiler-specific variables. These packages are handled by the ``MakefilePackage`` subclass which provides convenience methods to help write these types of packages. -A ``MakefilePackage`` build has three phases that can be overridden by the packager: +A ``MakefilePackage`` build has three phases that we can override: 1. ``edit()`` 2. ``build()`` 3. ``install()`` -Packagers then have the ability to control how a ``Makefile`` is edited, and what targets to include for the build phase or install phase. +We then have the ability to control how a ``Makefile`` is edited, and what targets to include for the build phase or install phase. Let's also take a look inside the ``MakefilePackage`` class: @@ -175,7 +175,7 @@ Let's also take a look inside the ``MakefilePackage`` class: $ spack edit --build-system makefile -Take note of the following: +Let's take note of the following: .. literalinclude:: _spack_root/lib/spack/spack/build_systems/makefile.py @@ -184,7 +184,7 @@ Take note of the following: :lines: 40-111 :linenos: -Similar to ``Autotools``, ``MakefilePackage`` class has properties that can be set by the packager. +Similar to ``Autotools``, the ``MakefilePackage`` class has properties that we can set. We can also override the different methods highlighted. @@ -210,7 +210,7 @@ Let's try to recreate the Bowtie_ package: ==> Created template for bowtie package ==> Created package file: /Users/mamelara/spack/var/spack/repos/builtin/packages/bowtie/package.py -Once the fetching is completed, Spack will open up your text editor in the usual fashion and create a template of a ``MakefilePackage`` package.py. +Once the fetching is completed, Spack will open up our text editor in the usual fashion and create a template of a ``MakefilePackage`` package.py. .. literalinclude:: tutorial/examples/Makefile/0.package.py :language: python @@ -225,7 +225,7 @@ Let's add in the rest of our details for our package: :linenos: As we mentioned earlier, most packages using a ``Makefile`` have hardcoded variables that must be edited. -These variables are fine if you happen to not care about setup or types of compilers used, but Spack is designed to work with any compiler. +These variables are fine if we happen to not care about setup or types of compilers used, but Spack is designed to work with any compiler. The ``MakefilePackage`` subclass makes it easy to edit these ``Makefiles`` by having an ``edit()`` method that can be overridden. Let's take a look at the default ``Makefile`` that ``Bowtie`` provides. @@ -236,7 +236,7 @@ If we look inside, we see that ``CC`` and ``CXX`` point to our GNU compiler: $ spack stage bowtie .. note:: - As usual make sure you have shell support activated with Spack: + As usual, let's make sure we have shell support activated with Spack: ``source /path/to/spack/share/spack/setup-env.sh`` .. code-block:: console @@ -254,7 +254,7 @@ If we look inside, we see that ``CC`` and ``CXX`` point to our GNU compiler: LIBS = $(LDFLAGS) -lz HEADERS = $(wildcard *.h) -To fix this, we need to use the ``edit()`` method to modify the ``Makefile``. +To fix this, we will use the ``edit()`` method to modify the ``Makefile``. .. literalinclude:: tutorial/examples/Makefile/2.package.py :language: python @@ -313,7 +313,7 @@ In our ``esmf`` example we set two environment variables in our ``edit()`` metho msg += "'{0}', is not supported by ESMF." raise InstallError(msg.format(self.compiler.name)) -As you may have noticed, we didn't really write anything to the ``Makefile`` but rather we set environment variables that will override variables set in the ``Makefile``. +As we may have noticed, we didn't really write anything to the ``Makefile`` but rather we set environment variables that will override variables set in the ``Makefile``. Some packages include a configuration file that sets certain compiler variables, platform specific variables, and the location of dependencies or libraries. If the file is simple and only requires a couple of changes, we can replace those entries with our ``FileFilter`` object. @@ -427,7 +427,7 @@ Typical build incantations look like this: make() make("install") -As you can see from the example above, it's very similar to invoking ``configure`` and ``make`` in an ``Autotools`` build system. +As we can see from the example above, it's very similar to invoking ``configure`` and ``make`` in an ``Autotools`` build system. However, the variable names and options differ. Most options in CMake are prefixed with a ``'-D'`` flag to indicate a configuration setting. @@ -459,8 +459,8 @@ Spack is able to support Unix-Makefile_ generators as well as Ninja_ generators. If no generator is specified, Spack will default to ``Unix Makefiles``. -Next we setup the build type. -In ``CMake`` you can specify the build type that you want. +Next, let's set up the build type. +In ``CMake``, we can specify the build type that we want. Options include: 1. ``empty`` @@ -469,16 +469,16 @@ Options include: 4. ``RelWithDebInfo`` 5. ``MinSizeRel`` -With these options you can specify whether you want your executable to have the debug version only, release version or the release with debug information. +With these options, we can specify whether we want our executable to have the debug version only, release version or the release with debug information. Release executables tend to be more optimized than Debug versions. In Spack, we set the default as `Release` unless otherwise specified through a variant (e.g., ``build_type=Debug``). Spack then automatically sets up the ``-DCMAKE_INSTALL_PREFIX`` path, appends the build type (defaulting to ``RelWithDebInfo``), and enables a verbose ``Makefile`` output by default. -Next we add the ``rpaths`` to ``-DCMAKE_INSTALL_RPATH:STRING``. +Next, let's add the ``rpaths`` to ``-DCMAKE_INSTALL_RPATH:STRING``. -Finally we add to ``-DCMAKE_PREFIX_PATH:STRING`` the locations of all our dependencies so that ``CMake`` can find them. +Finally, let's add to ``-DCMAKE_PREFIX_PATH:STRING`` the locations of all our dependencies so that ``CMake`` can find them. In the end our ``cmake`` line will look like this (example is ``xrootd``): @@ -519,7 +519,7 @@ which then produces the following template: :language: python :linenos: -Again we fill in the details: +Again, let's fill in the details: .. literalinclude:: tutorial/examples/Cmake/1.package.py :language: python @@ -540,7 +540,7 @@ Now we can control our build options using ``cmake_args()``. If defaults are sufficient enough for the package, we can leave this method out. ``CMakePackage`` classes allow for control of other features in the build system. -For example, you can specify the path to the "out of source" build directory and also point to the root of the ``CMakeLists.txt`` file if it is placed in a non-standard location. +For example, we can specify the path to the "out of source" build directory and also point to the root of the ``CMakeLists.txt`` file if it is placed in a non-standard location. A good example of a package that has its ``CMakeLists.txt`` file located at a different location is found in ``spades``. @@ -562,7 +562,7 @@ For example, let's take a look at ``sniffles``. $ spack edit sniffles -In the ``install()`` method, we have to manually install our targets so we override the ``install()`` method to do it for us: +In the ``install()`` method, we have to manually install our targets, so let's override the ``install()`` method to do it for us: .. code-block:: python @@ -588,7 +588,7 @@ These modules are usually installed using the following line: We can write package files for Python packages using the ``Package`` class, but the class brings with it a lot of methods that are useless for Python packages. -Instead, Spack has a ``PythonPackage`` subclass that allows packagers of Python modules to be able to invoke ``pip``. +Instead, Spack has a ``PythonPackage`` subclass that allows us to invoke ``pip``. We will write a package file for Pandas_: @@ -619,14 +619,14 @@ And we are left with the following template: :language: python :linenos: -As you can see this is not any different than any package template that we have written. +As we can see, this is not any different than any package template that we have written. We have the choice of providing build options or using the sensible defaults. Luckily for us, there is no need to provide build args. -Next we need to find the dependencies of a package. +Next, we will need to find the dependencies of a package. Dependencies are usually listed in ``setup.py``. -You can find the dependencies by searching for ``install_requires`` keyword in that file. +We can find the dependencies by searching for ``install_requires`` keyword in that file. Here it is for ``Pandas``: .. code-block:: python @@ -647,7 +647,7 @@ Here it is for ``Pandas``: # ... more code -You can find a more comprehensive list at the Pandas documentation_. +We can find a more comprehensive list at the Pandas documentation_. .. _documentation: https://pandas.pydata.org/pandas-docs/stable/install.html @@ -661,17 +661,17 @@ Here is the completed ``Pandas`` script: :linenos: It is quite important to declare all the dependencies of a Python package. -Spack can "activate" Python packages to prevent the user from having to load each dependency module explicitly. +Spack can "activate" Python packages to prevent us from having to load each dependency module explicitly. If a dependency is missed, Spack will be unable to properly activate the package and it will cause an issue. -To learn more about extensions go to `spack extensions `_. +To learn more about extensions, let's go to `spack extensions `_. -From this example, you can see that building Python modules is made easy through the ``PythonPackage`` class. +From this example, we can see that building Python modules is made easy through the ``PythonPackage`` class. ------------------- Other Build Systems ------------------- -Although we won't get in depth with any of the other build systems that Spack supports, it is worth mentioning that Spack does provide subclasses for the following build systems: +Although we won't get in depth with any of the other build systems that Spack supports, it is worth mentioning that Spack provides subclasses for the following build systems: 1. ``IntelPackage`` 2. ``SconsPackage`` @@ -682,7 +682,7 @@ Although we won't get in depth with any of the other build systems that Spack su Each of these classes have their own abstractions to help assist in writing package files. -For whatever doesn't fit nicely into the other build systems, you can use the ``Package`` class. +For whatever doesn't fit nicely into the other build systems, we can use the ``Package`` class. -Hopefully by now you can see how we aim to make packaging simple and robust through these classes. -If you want to learn more about these build systems, check out `Implementing the installation procedure `_ in the Packaging Guide. +Hopefully by now, we can see how Spack aims to make packaging simple and robust through these classes. +If we want to learn more about these build systems, let's check out `Implementing the installation procedure `_ in the Packaging Guide.