From a498a7b4e033822d9f620892b7c7ca34ee1e4924 Mon Sep 17 00:00:00 2001 From: Alec Scott Date: Mon, 2 Jun 2025 16:23:09 -0700 Subject: [PATCH 1/3] Update basics section --- tutorial_basics.rst | 44 +++++++++++++++++++++++--------------------- 1 file changed, 23 insertions(+), 21 deletions(-) diff --git a/tutorial_basics.rst b/tutorial_basics.rst index 77946640c9..1dff0c38a3 100644 --- a/tutorial_basics.rst +++ b/tutorial_basics.rst @@ -75,8 +75,8 @@ Let's go ahead and install ``gmake``, You will see Spack installed ``gmake``, ``gcc-runtime``, and ``glibc``. The ``glibc`` and ``gcc-runtime`` packages are automatically tracked by Spack to manage consistency requirements among compiler runtimes. -These do not represent separate installs from source, but represent aspects of the compiler Spack used for the install. -For the rest of this section, we will ignore these components and focus on the rest of the install. +They do not represent separate software builds from source, but are records of the system's compiler runtime components Spack used for the install. +For the rest of this section, we will ignore these components and focus on the packages explicitly installed. Spack can install software either from source or from a binary cache. Packages in the binary cache are signed with GPG for security. @@ -89,14 +89,14 @@ To be able to install from the binary cache, we will need to configure Spack wit You'll learn more about configuring Spack later in the tutorial, but for now you will be able to install the rest of the packages in the tutorial from a binary cache using the same ``spack install`` command. By default this will install the binary cached version if it exists and fall back on installing from source if it does not. -Spack's spec syntax is the interface by which we can request specific configurations of the package. +Spack's "spec" syntax is the interface by which we can request specific configurations of a package. The ``%`` sigil is used to specify compilers. .. literalinclude:: outputs/basics/zlib-clang.out :language: console Note that this installation is located separately from the previous one. -We will discuss this in more detail later, but this is part of what allows Spack to support arbitrarily versioned software. +We will discuss this in more detail later, but this is part of what allows Spack to support many versions of software packages. You can check for particular versions before requesting them. We will use the ``spack versions`` command to see the available versions, and then install a different version of ``zlib-ng``. @@ -115,7 +115,7 @@ The ``@`` sigil is used to specify versions, both of packages and of compilers. The spec syntax also includes compiler flags. Spack accepts ``cppflags``, ``cflags``, ``cxxflags``, ``fflags``, ``ldflags``, and ``ldlibs`` parameters. The values of these fields must be quoted on the command line if they include spaces. -These values are injected into the compile line automatically by the Spack compiler wrappers. +These values are injected into the compilation commands automatically by the Spack compiler wrappers. .. literalinclude:: outputs/basics/zlib-O3.out :language: console @@ -134,7 +134,7 @@ Spack generates a hash for each spec. This hash is a function of the full provenance of the package, so any change to the spec affects the hash. Spack uses this value to compare specs and to generate unique installation directories for every combinatorial version. As we move into more complicated packages with software dependencies, we can see that Spack reuses existing packages to satisfy a dependency. -By default, Spack tries hard to reuse existing installations as dependencies, either from a local store or from configured remote buildcaches. +By default, Spack tries hard to reuse existing installations as dependencies, either from a local store or from configured remote binary caches. This minimizes unwanted rebuilds of common dependencies, in particular if you update Spack frequently. .. literalinclude:: outputs/basics/tcl.out @@ -165,7 +165,7 @@ Note that each package has a top-level entry, even if it also appears as a depen Let's move on to slightly more complicated packages. HDF5 is a good example of a more complicated package, with an MPI dependency. -If we install it "out of the box," it will build with OpenMPI. +If we install it with default settings it will build with OpenMPI. .. literalinclude:: outputs/basics/hdf5.out :language: console @@ -181,14 +181,14 @@ Here we can install HDF5 without MPI support. :language: console We might also want to install HDF5 with a different MPI implementation. -While MPI is not a package itself, packages can depend on abstract interfaces like MPI. -Spack handles these through "virtual dependencies." A package, such as HDF5, can depend on the MPI interface. -Other packages (``openmpi``, ``mpich``, ``mvapich2``, etc.) provide the MPI interface. -Any of these providers can be requested for an MPI dependency. -For example, we can build HDF5 with MPI support provided by MPICH by specifying a dependency on ``mpich``. +While ``mpi`` itself is a virtual package representing an interface, other packages can depend on such abstract interfaces. +Spack handles these through "virtual dependencies." A package, such as HDF5, can depend on the ``mpi`` virtual package (the interface). +Actual MPI implementation packages (like ``openmpi``, ``mpich``, ``mvapich2``, etc.) provide the MPI interface. +Any of these providers can be requested to satisfy an MPI dependency. +For example, we can build HDF5 with MPI support provided by MPICH by specifying a dependency on ``mpich`` (e.g., ``hdf5 ^mpich``). Spack also supports versioning of virtual dependencies. -A package can depend on the MPI interface at version 3, and provider packages specify what version of the interface *they* provide. -The partial spec ``^mpi@3`` can be satisfied by any of several providers. +A package can depend on the MPI interface at version 3 (e.g., ``hdf5 ^mpi@3``), and provider packages specify what version of the interface *they* provide. +The partial spec ``^mpi@3`` can be satisfied by any of several MPI implementation packages that provide MPI version 3. .. literalinclude:: outputs/basics/hdf5-hl-mpi.out :language: console @@ -212,7 +212,7 @@ Now let's look at an even more complicated package. :language: console Now we're starting to see the power of Spack. -Trilinos in its default configuration has 23 top level dependencies, many of which have dependencies of their own. +Trilinos in its default configuration has 23 direct dependencies, many of which have dependencies of their own. Installing more complex packages can take days or weeks even for an experienced user. Although we've done a binary installation for the tutorial, a source installation of Trilinos using Spack takes about 3 hours (depending on the system), but only 20 seconds of programmer time. @@ -269,7 +269,9 @@ We can uninstall packages by spec using the same syntax as install. We can also uninstall packages by referring only to their hash. -We can use either ``-f`` (force) or ``-R`` (remove dependents as well) to remove packages that are required by another installed package. +We can use either the ``--force`` (or ``-f``) flag or the ``--dependents`` (or ``-R``) flag to remove packages that are required by another installed package. +Use ``--force`` to remove just the specified package, leaving dependents broken. +Use ``--dependents`` to remove the specified package and all of its dependents. .. literalinclude:: outputs/basics/uninstall-needed.out :language: console @@ -277,8 +279,8 @@ We can use either ``-f`` (force) or ``-R`` (remove dependents as well) to remove .. literalinclude:: outputs/basics/uninstall-r-needed.out :language: console -Spack will not uninstall packages that are not sufficiently specified. -The ``-a`` (all) flag can be used to uninstall multiple packages at once. +Spack will not uninstall packages that are not sufficiently specified specified (i.e., if the spec is ambiguous and matches multiple installed packages). +The ``--all`` (or ``-a``) flag can be used to uninstall all packages matching an ambiguous spec. .. literalinclude:: outputs/basics/uninstall-ambiguous.out :language: console @@ -314,13 +316,13 @@ Customizing Compilers --------------------- Spack manages a list of available compilers on the system, detected automatically from the user's ``PATH`` variable. -The ``spack compilers`` command is an alias for the command ``spack compiler list``. +The ``spack compilers`` command is an alias for ``spack compiler list``. .. literalinclude:: outputs/basics/compilers.out :language: console -The compilers are maintained in a YAML file. -Later in the tutorial you will learn how to configure compilers by hand for special cases. +The compilers are maintained in a YAML file (``compilers.yaml``). +Later in the tutorial, you will learn how to configure compilers by hand for special cases. Spack also has tools to add compilers, and compilers built with Spack can be added to the configuration. .. literalinclude:: outputs/basics/install-gcc-12.1.0.out From f5e2a0f8aed382483936f47d54a3ab9942179d67 Mon Sep 17 00:00:00 2001 From: Alec Scott Date: Mon, 2 Jun 2025 16:47:14 -0700 Subject: [PATCH 2/3] Fix typos and suggestions for binary cache --- tutorial_binary_cache.rst | 53 ++++++++++++++++++--------------------- 1 file changed, 25 insertions(+), 28 deletions(-) diff --git a/tutorial_binary_cache.rst b/tutorial_binary_cache.rst index 22fb666b36..f7d3d79260 100644 --- a/tutorial_binary_cache.rst +++ b/tutorial_binary_cache.rst @@ -11,10 +11,10 @@ Binary Caches Tutorial ================================== -In this section of the tutorial you will learn how to share Spack built binaries across machines and users using build caches. +In this section of the tutorial, you will learn how to share Spack-built binaries across machines and users using build caches. -We will explore a few concepts that apply to all types of build caches, but the focus is primarily on **OCI container registries** like Docker Hub or Github Packages as a storage backend for binary caches. -Spack supports a range of storage backends, like an ordinary filesystem, S3, and Google Cloud Storage, but OCI build caches have a few interesting properties that make them worth exploring more in depth. +We will explore a few concepts that apply to all types of build caches, but the focus is primarily on **OCI container registries** (like Docker Hub or Github Packages) as a storage backend for binary caches. +Spack supports a range of storage backends, such as an ordinary filesystem, Amazon S3, and Google Cloud Storage, but OCI build caches have a few interesting properties that make them worth exploring more in-depth. Before we configure a build cache, let's install the ``julia`` package, which is an interesting example because it has some non-trivial dependencies like ``llvm``, and features an interactive REPL that we can use to verify that the installation works. @@ -42,20 +42,22 @@ Setting up an OCI build cache on GitHub Packages For this tutorial we will be using GitHub Packages as an OCI registry, since most people have a GitHub account and it's easy to use. -First go to ``_ to generate a Personal access token with ``write:packages`` permissions. +First, go to ``_ to generate a Personal Access Token (classic) with ``write:packages`` permissions. Copy this token. -Next, we will add this token to the mirror config section of the Spack environment: +Next, we will add this token to the mirror configuration section for the Spack environment. +Replace `` with your GitHub username and `` with your GitHub username or an organization where you have permission to create packages. +The build cache name, `buildcache-${USER}-${HOSTNAME}`, is a suggestion; you can choose your own. .. code-block:: console - $ export MY_OCI_TOKEN= + $ export MY_OCI_TOKEN= $ spack -e . mirror add \ - --oci-username \ + --oci-username \ --oci-password-variable MY_OCI_TOKEN \ --unsigned \ my-mirror \ - oci://ghcr.io//buildcache-${USER}-${HOSTNAME} + oci://ghcr.io//buildcache-${USER}-${HOSTNAME} .. note :: @@ -101,7 +103,7 @@ which outputs ... ==> Pushed julia@1.9.3/dfzhutf to ghcr.io//buildcache--:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack -The location of the pushed package +The location of the pushed package, when referred to as a OCI image, will be: .. code-block:: text @@ -142,7 +144,7 @@ The easiest way to do this is to override the ``mirrors`` config section in the secret_variable: MY_OCI_TOKEN signed: false -An "overwrite install" should be enough to show that the build cache is used: +An "overwrite install" should be enough to show that the build cache is used (output will vary based on your specific configuration): .. code-block:: console @@ -177,10 +179,9 @@ For convenience you can also run ``spack buildcache push --update-index ...`` to .. note:: - As of Spack 0.22, build caches can be used across different Linux distros. The concretizer - will reuse specs that have a host compatible ``libc`` dependency (e.g. ``glibc`` or ``musl``). - For packages compiled with ``gcc`` (and a few others), users do not have to install compilers - first, as the build cache contains the compiler runtime libraries as a separate package. + As of Spack 0.22, build caches can be used across different Linux distros. + The concretizer will reuse specs that have a host-compatible ``libc`` dependency (e.g. ``glibc`` or ``musl``). + For packages compiled with ``gcc`` (and a few other compilers), users do not have to install compilers first, as the build cache contains the compiler runtime libraries as a separate package dependency. After an index is created, it's possible to list the available packages in the build cache: @@ -255,7 +256,7 @@ Let's add a simple text editor like ``vim`` to our previous environment next to $ spack -e . install --add vim -This time we push to the OCI registry, but also pass ``--tag julia-and-vim`` to instruct Spack to create an image for the environment as a whole, with a human-readable tag: +This time, when we push to the OCI registry, we also pass ``--tag julia-and-vim`` to instruct Spack to create an additional image tag for the environment as a whole, with a more human-readable name: .. code-block:: console @@ -291,12 +292,8 @@ For those familiar with ``Dockerfile`` syntax, it would structurally look like t This approach is still valid, and the ``spack containerize`` command continues to exist, but it has a few downsides: -* When ``RUN spack -e /root/env install`` fails, ``docker`` will not cache the layer, meaning - that all dependencies that did install successfully are lost. Troubleshooting the build - typically means starting from scratch in ``docker run`` or on the host system. -* In certain CI environments, it is not possible to use ``docker build``. For example, the - CI script itself may already run in a docker container, and running ``docker build`` *safely* - inside a container is tricky. +* When ``RUN spack -e /root/env install`` fails, ``docker`` will not cache the layer, meaning that all dependencies that did install successfully are lost. Troubleshooting the build typically means starting from scratch either within a ``docker run`` session or on the host system. +* In certain CI environments, it is not possible to use ``docker build`` directly. For example, the CI script itself may already run in a Docker container, and running ``docker build`` *safely* inside a container (Docker-in-Docker) is tricky. The takeaway is that Spack decouples the steps that ``docker build`` combines: build isolation, running the build, and creating an image. You can run ``spack install`` on your host machine or in a container, and run ``spack buildcache push`` separately to create an image. @@ -307,7 +304,7 @@ Relocation Spack is different from many package managers in that it lets users choose where to install packages. This makes Spack very flexible, as users can install packages in their home directory and do not need root privileges. -The downside is that sharing binaries is more complicated, as binaries may contain hard-coded, absolute paths to machine specific locations, which have to be adjusted when binaries are installed on a different machine. +The downside is that sharing binaries is more complicated, as binaries may contain hard-coded, absolute paths to machine specific locations, which have to be adjusted when these binaries are installed on a different machine or in a different path. Fortunately Spack handles this automatically upon install from a binary cache. But when you build binaries that are intended to be shared, there is one thing you have to keep in mind: Spack can relocate hard-coded paths in binaries *provided that the target prefix is shorter than the prefix used during the build*. @@ -315,7 +312,7 @@ But when you build binaries that are intended to be shared, there is one thing y The reason is that binaries typically embed these absolute paths in string tables, which is a list of null terminated strings, to which the program stores offsets. That means we can only modify strings in-place, and if the new path is longer than the old one, we would overwrite the next string in the table. -To maximize the chances of successful relocation, you should build your binaries in a relative long path. +To maximize the chances of successful relocation, you should build your binaries in a relatively long path. Fortunately Spack can automatically pad paths to make them longer, using the following command: .. code-block:: console @@ -327,9 +324,9 @@ Using build caches in CI ------------------------ Build caches are a great way to speed up CI pipelines. -Both GitHub Actions and Gitlab CI support container registries, and this tutorial should give you a good starting point to leverage them. +Both GitHub Actions and GitLab CI support container registries, and this tutorial should give you a good starting point to leverage them. -Spack also provides a basic GitHub Action to already provide you with a binary cache: +Spack also provides a basic GitHub Action that already provides you with a binary cache: .. code-block:: yaml @@ -349,6 +346,6 @@ Summary In this tutorial we have created a build cache on top of an OCI registry, which can be used -* to ``spack install julia vim`` on machines without source builds -* to automatically create container images for individual packages while pushing to the cache -* to create container images for multiple packages at once +* to run ``spack install julia vim`` on machines and have Spack fetch pre-built binaries instead of building from source. +* to automatically create container images for individual packages when pushing to the cache. +* to create container images for entire Spack environments (multiple packages) at once. From 8e105e88588ea73ce5a9b04cb8efb000cdb064b7 Mon Sep 17 00:00:00 2001 From: Alec Scott Date: Mon, 2 Jun 2025 17:52:15 -0700 Subject: [PATCH 3/3] Fix typos and add suggestions to buildsystems --- tutorial_buildsystems.rst | 264 ++++++++++++++++++-------------------- 1 file changed, 126 insertions(+), 138 deletions(-) diff --git a/tutorial_buildsystems.rst b/tutorial_buildsystems.rst index 62216f86f0..dff85f9116 100644 --- a/tutorial_buildsystems.rst +++ b/tutorial_buildsystems.rst @@ -11,13 +11,12 @@ Spack Package Build Systems =========================== -You may begin to notice after writing a couple of package template files a pattern emerge for some packages. -For example, you may find yourself writing an :code:`install()` method that invokes: :code:`configure`, :code:`cmake`, -:code:`make`, :code:`make install`. You may also find yourself writing -:code:`"prefix=" + prefix` as an argument to :code:`configure` or :code:`cmake`. +You may begin to notice after writing a couple of package template files that a pattern emerges for some packages. +For example, you may find yourself writing an ``install()`` method that invokes: ``configure``, ``cmake``, ``make``, ``make install``. +You may also find yourself writing ``"prefix=" + prefix`` as an argument to ``configure`` or ``cmake``. Rather than having you repeat these lines for all packages, Spack has classes that can take care of these patterns. -In addition, these package files allow for finer grained control of these build systems. -In this section, we will describe each build system and give examples on how these can be manipulated to install a package. +In addition, these package files allow for finer-grained control of these build systems. +In this section, we will describe each build system and give examples on how these can be used to simplify packaging. ----------------------- Package Class Hierarchy @@ -41,29 +40,26 @@ Package Class Hierarchy PackageBase -> PythonPackage [dir=back] } -The above diagram gives a high level view of the class hierarchy and how each package relates. -Each subclass inherits from the :code:`PackageBaseClass` super class. -The bulk of the work is done in this super class which includes fetching, extracting to a staging directory and installing. +The above diagram gives a high-level view of the class hierarchy and how each package relates. +Each build system specific class inherits from the ``PackageBaseClass`` superclass. +The bulk of the common work is done in this superclass which includes fetching, extracting to a staging directory and installing. Each subclass then adds additional build-system-specific functionality. -In the following sections, we will go over examples of how to utilize each subclass and to see how powerful these abstractions are when packaging. +In the following sections, we will go over examples of how to utilize each subclass and see how powerful these abstractions are when packaging. ----------------- Package ----------------- -We've already seen examples of a :code:`Package` class in our walkthrough for writing package files, so we won't be spending much time with them here. -Briefly, the Package class allows for arbitrary control over the build process, whereas subclasses rely on certain patterns (e.g. :code:`configure` :code:`make` -:code:`make install`) to be useful. :code:`Package` classes are particularly useful -for packages that have a non-conventional way of being built since the packager can utilize some of Spack's helper functions to customize the building and installing of a package. +We've already seen examples of using the generic ``Package`` class in our walkthrough for writing package files, so we won't be spending much time with it here. +Briefly, the Package class allows for arbitrary control over the build process, whereas subclasses rely on certain patterns (e.g. ``configure`` ``make`` ``make install``) to be useful. +The generic ``Package`` class is particularly useful for packages that have a non-conventional build process, as it allows the packager to use Spack's helper functions to customize the building and installing of a package fully. ------------------- Autotools ------------------- -As we have seen earlier, packages using :code:`Autotools` use :code:`configure`, -:code:`make` and :code:`make install` commands to execute the build and -install process. -In our :code:`Package` class, your typical build incantation will consist of the following: +As we have seen earlier, packages using ``Autotools`` use ``configure``, ``make`` and ``make install`` commands to execute the build and install process. +In our ``Package`` class, your typical build incantation will consist of the following: .. code-block:: python @@ -74,28 +70,28 @@ In our :code:`Package` class, your typical build incantation will consist of the You'll see that this looks similar to what we wrote in our packaging tutorial. -The :code:`Autotools` subclass aims to simplify writing package files and provides convenience methods to manipulate each of the different phases for a :code:`Autotools` build system. +The ``AutotoolsPackage`` subclass aims to simplify writing package files for Autotools-based software and provides convenient methods to manipulate each of the different phases for an ``Autotools`` build system. -:code:`Autotools` packages consist of four phases: +``AutotoolsPackage`` builds consist of four main phases, each cooresponding to a method that can be overridden: -1. :code:`autoreconf()` -2. :code:`configure()` -3. :code:`build()` -4. :code:`install()` +1. ``autoreconf()`` +2. ``configure()`` +3. ``build()`` +4. ``install()`` Each of these phases have sensible defaults. -Let's take a quick look at some the internals of the :code:`Autotools` class: +Let's take a quick look at some of the internals of the ``Autotools`` class: .. code-block:: console $ spack edit --build-system autotools -This will open the :code:`AutotoolsPackage` file in your text editor. +This will open the ``AutotoolsPackage`` file in your text editor. .. note:: - The examples showing code for these classes is abridged to avoid having + The examples showing code for these classes are abridged to avoid having long examples. We only show what is relevant to the packager. @@ -107,13 +103,13 @@ This will open the :code:`AutotoolsPackage` file in your text editor. Important to note are the highlighted lines. These properties allow the packager to set what build targets and install targets they want for their package. -If, for example, we wanted to add as our build target :code:`foo` then we can append to our :code:`build_targets` property: +If, for example, we wanted to add as our build target ``foo`` then we can append to our ``build_targets`` property: .. code-block:: python build_targets = ["foo"] -Which is similar to invoking make in our Package +Which is similar to invoking ``make`` in our Package .. code-block:: python @@ -121,61 +117,60 @@ Which is similar to invoking make in our Package This is useful if we have packages that ignore environment variables and need a command-line argument. -Another thing to take note of is in the :code:`configure()` method. -Here we see that the :code:`prefix` argument is already included since it is a common pattern amongst packages using :code:`Autotools`. -We then only have to override :code:`configure_args()`, which will then return it's output to to :code:`configure()`. -Then, :code:`configure()` will append the common arguments +Another thing to take note of is in the ``configure()`` method in ``AutotoolsPackage``. +Here we see that the ``--prefix`` argument is already included since it is a common pattern amongst packages using ``Autotools``. +Therefore, we typically only need to override the ``configure_args()`` method to return a list of additional arguments. +The ``configure()`` method will then append these to the standard arguments. -Packagers also have the option to run :code:`autoreconf` in case a package needs to update the build system and generate a new :code:`configure`. -Though, for the most part this will be unnecessary. +Packagers also have the option to run ``autoreconf`` in case a package needs to update the build system and generate a new ``configure``. +However, for the most part this will be unnecessary. -Let's look at the :code:`mpileaks` package.py file that we worked on earlier: +Let's look at the ``mpileaks`` package.py file that we worked on earlier: .. code-block:: console $ spack edit mpileaks -Notice that mpileaks is a :code:`Package` class but uses the :code:`Autotools` build system. -Although this package is acceptable let's make this into an -:code:`AutotoolsPackage` class and simplify it further. +Notice that mpileaks was originally written as a generic ``Package`` but uses the ``Autotools`` build system. +Although this package is acceptable, let's covert it to an ``AutotoolsPackage`` to simplify it further. .. literalinclude:: tutorial/examples/Autotools/0.package.py :language: python :emphasize-lines: 9 :linenos: -We first inherit from the :code:`AutotoolsPackage` class. +We first inherit from the ``AutotoolsPackage`` class. -Although we could keep the :code:`install()` method, most of it can be handled by the :code:`AutotoolsPackage` base class. -In fact, the only thing that needs to be overridden is :code:`configure_args()`. +Although we could keep the ``install()`` method, most of it can be handled by the ``AutotoolsPackage`` base class. +In fact, the only thing that needs to be overridden is ``configure_args()``. .. literalinclude:: tutorial/examples/Autotools/1.package.py :language: python :emphasize-lines: 25,26,27,28,29,30,31,32 :linenos: -Since Spack takes care of setting the prefix for us we can exclude that as an argument to :code:`configure`. -Our packages look simpler, and the packager does not need to worry about whether they have properly included :code:`configure` and :code:`make`. +Since Spack's ``AutotoolsPackage`` takes care of setting the prefix for us, we can exclude that as an argument to ``configure``. +Our package file looks simpler, and we don't not need to worry about whether we have properly included ``configure`` and ``make``. -This version of the :code:`mpileaks` package installs the same as the previous, but the :code:`AutotoolsPackage` class lets us do it with a cleaner looking package file. +This version of the ``mpileaks`` package installs the same as the previous, but the ``AutotoolsPackage`` class lets us do it with a cleaner looking package file. ----------------- Makefile ----------------- -Packages that utilize :code:`Make` or a :code:`Makefile` usually require you to edit a :code:`Makefile` to set up platform and compiler specific variables. -These packages are handled by the :code:`Makefile` subclass which provides convenience methods to help write these types of packages. +Packages that utilize ``Make`` or a ``Makefile`` usually require you to edit a ``Makefile`` to set up platform and compiler specific variables. +These packages are handled by the ``Makefile`` subclass which provides convenience methods to help write these types of packages. -A :code:`MakefilePackage` class has three phases that can be overridden. These include: +A ``MakefilePackage`` build has three phases that can be overridden by the packager: - 1. :code:`edit()` - 2. :code:`build()` - 3. :code:`install()` + 1. ``edit()`` + 2. ``build()`` + 3. ``install()`` -Packagers then have the ability to control how a :code:`Makefile` is edited, and what targets to include for the build phase or install phase. +Packagers then have the ability to control how a ``Makefile`` is edited, and what targets to include for the build phase or install phase. -Let's also take a look inside the :code:`MakefilePackage` class: +Let's also take a look inside the ``MakefilePackage`` class: .. code-block:: console @@ -190,7 +185,7 @@ Take note of the following: :lines: 40-111 :linenos: -Similar to :code:`Autotools`, :code:`MakefilePackage` class has properties that can be set by the packager. +Similar to ``Autotools``, ``MakefilePackage`` class has properties that can be set by the packager. We can also override the different methods highlighted. @@ -216,13 +211,13 @@ Let's try to recreate the Bowtie_ package: ==> Created template for bowtie package ==> Created package file: /Users/mamelara/spack/var/spack/repos/builtin/packages/bowtie/package.py -Once the fetching is completed, Spack will open up your text editor in the usual fashion and create a template of a :code:`MakefilePackage` package.py. +Once the fetching is completed, Spack will open up your text editor in the usual fashion and create a template of a ``MakefilePackage`` package.py. .. literalinclude:: tutorial/examples/Makefile/0.package.py :language: python :linenos: -Spack was successfully able to detect that :code:`Bowtie` uses :code:`Make`. +Spack was successfully able to detect that ``Bowtie`` uses ``Make``. Let's add in the rest of our details for our package: .. literalinclude:: tutorial/examples/Makefile/1.package.py @@ -230,20 +225,20 @@ Let's add in the rest of our details for our package: :emphasize-lines: 10,11,13,14,18,20 :linenos: -As we mentioned earlier, most packages using a :code:`Makefile` have hard-coded variables that must be edited. +As we mentioned earlier, most packages using a ``Makefile`` have hard-coded variables that must be edited. These variables are fine if you happen to not care about setup or types of compilers used but Spack is designed to work with any compiler. -The :code:`MakefilePackage` subclass makes it easy to edit these :code:`Makefiles` by having an :code:`edit()` method that can be overridden. +The ``MakefilePackage`` subclass makes it easy to edit these ``Makefiles`` by having an ``edit()`` method that can be overridden. -Let's take a look at the default :code:`Makefile` that :code:`Bowtie` provides. -If we look inside, we see that :code:`CC` and :code:`CXX` point to our GNU compiler: +Let's take a look at the default ``Makefile`` that ``Bowtie`` provides. +If we look inside, we see that ``CC`` and ``CXX`` point to our GNU compiler: .. code-block:: console $ spack stage bowtie .. note:: - As usual make sure you have shell support activated with spack: - :code:`source /path/to/spack_root/spack/share/spack/setup-env.sh` + As usual make sure you have shell support activated with Spack: + ``source /path/to/spack_root/spack/share/spack/setup-env.sh`` .. code-block:: console @@ -260,18 +255,16 @@ If we look inside, we see that :code:`CC` and :code:`CXX` point to our GNU compi LIBS = $(LDFLAGS) -lz HEADERS = $(wildcard *.h) -To fix this, we need to use the :code:`edit()` method to write our custom -:code:`Makefile`. +To fix this, we need to use the ``edit()`` method to modify the ``Makefile``. .. literalinclude:: tutorial/examples/Makefile/2.package.py :language: python :emphasize-lines: 23,24,25 :linenos: -Here we use a :code:`FileFilter` object to edit our :code:`Makefile`. -It takes in a regular expression and then replaces :code:`CC` and :code:`CXX` to whatever Spack sets :code:`CC` and :code:`CXX` environment variables to. -This allows us to build :code:`Bowtie` with whatever compiler we specify through Spack's -:code:`spec` syntax. +Here we use a ``FileFilter`` object (a Spack utility) to edit our ``Makefile``. +It takes a regular expression to find lines (e.g., assignments to ``CC`` and ``CXX``) and then replaces them with values derived from Spack's build environment (e.g., ``self.compiler.cc`` and ``self.compiler.cxx``). +This allows us to build ``Bowtie`` with whatever compiler we specify through Spack's spec syntax. Let's change the build and install phases of our package: @@ -280,13 +273,12 @@ Let's change the build and install phases of our package: :emphasize-lines: 28,29,30,31,32,35,36 :linenos: -Here demonstrate another strategy that we can use to manipulate our package We can provide command-line arguments to :code:`make()`. -Since :code:`Bowtie` can use :code:`tbb` we can either add :code:`NO_TBB=1` as a argument to prevent -:code:`tbb` support or we can just invoke :code:`make` with no arguments. +Here we demonstrate another strategy that we can use to manipulate our package's build. +We can provide command-line arguments to ``make()``. +Since ``Bowtie`` can use ``tbb`` we can either add ``NO_TBB=1`` as a argument to prevent ``tbb`` support, or we can invoke ``make`` with no arguments if EBB is desired and found by its build system. -:code:`Bowtie` requires our :code:`install_target` to provide a path to -the install directory. -We can do this by providing :code:`prefix=` as a command line argument to :code:`make()`. +``Bowtie`` requires our ``install_target`` to provide a path to the install directory. +We can do this by providing ``prefix=`` as a command line argument to ``make()``. Let's look at a couple of other examples and go through them: @@ -295,8 +287,8 @@ Let's look at a couple of other examples and go through them: $ spack edit esmf Some packages allow environment variables to be set and will honor them. -Packages that use :code:`?=` for assignment in their :code:`Makefile` can be set using environment variables. -In our :code:`esmf` example we set two environment variables in our :code:`edit()` method: +Packages that use ``?=`` for assignment in their ``Makefile`` can be set using environment variables. +In our ``esmf`` example we set two environment variables in our ``edit()`` method: .. code-block:: python @@ -322,13 +314,13 @@ In our :code:`esmf` example we set two environment variables in our :code:`edit( msg += "'{0}', is not supported by ESMF." raise InstallError(msg.format(self.compiler.name)) -As you may have noticed, we didn't really write anything to the :code:`Makefile` but rather we set environment variables that will override variables set in the :code:`Makefile`. +As you may have noticed, we didn't really write anything to the ``Makefile`` but rather we set environment variables that will override variables set in the ``Makefile``. Some packages include a configuration file that sets certain compiler variables, platform specific variables, and the location of dependencies or libraries. -If the file is simple and only requires a couple of changes, we can overwrite those entries with our :code:`FileFilter` object. -If the configuration involves complex changes, we can write a new configuration file from scratch. +If the file is simple and only requires a couple of changes, we can replace those entries with our ``FileFilter`` object. +If the configuration involves complex changes, we can write a new configuration file from scratch within the ``edit()`` method. -Let's look at an example of this in the :code:`elk` package: +Let's look at an example of this in the ``elk`` package: .. code-block:: console @@ -416,16 +408,15 @@ Let's look at an example of this in the :code:`elk` package: for key in config: inc.write('{0} = {1}\n'.format(key, config[key])) -:code:`config` is just a dictionary that we can add key-value pairs to. By the -end of the :code:`edit()` method we write the contents of our dictionary to -:code:`make.inc`. +``config`` is just a Python dictionary that we populate with key-value pairs. +By the end of the ``edit()`` method, we write the contents of our dictionary to the ``make.inc`` file, which the package's ``Makefile`` then includes. --------------- CMake --------------- CMake_ is another common build system that has been gaining popularity. -It works in a similar manner to :code:`Autotools` but with differences in variable names, the number of configuration options available, and the handling of shared libraries. +It works in a similar manner to ``Autotools`` but with differences in variable names, the number of configuration options available, and the handling of shared libraries. Typical build incantations look like this: .. _CMake: https://cmake.org @@ -437,20 +428,19 @@ Typical build incantations look like this: make() make("install") -As you can see from the example above, it's very similar to invoking -:code:`configure` and :code:`make` in an :code:`Autotools` build system. However, -the variable names and options differ. -Most options in CMake are prefixed with a :code:`'-D'` flag to indicate a configuration setting. +As you can see from the example above, it's very similar to invoking ``configure`` and ``make`` in an ``Autotools`` build system. +However, the variable names and options differ. +Most options in CMake are prefixed with a ``'-D'`` flag to indicate a configuration setting. -In the :code:`CMakePackage` class we can override the following phases: +In the ``CMakePackage`` class, we can override the following build phases: -1. :code:`cmake()` -2. :code:`build()` -3. :code:`install()` +1. ``cmake()`` +2. ``build()`` +3. ``install()`` -The :code:`CMakePackage` class also provides sensible defaults so we only need to override :code:`cmake_args()`. +The ``CMakePackage`` class also provides sensible defaults, so often we only need to override ``cmake_args()`` to pass package-specific options. -Let's look at these defaults in the :code:`CMakePackage` class in the :code:`_std_args()` method: +Let's look at these defaults in the ``CMakePackage`` class in the ``_std_args()`` method: .. code-block:: console @@ -462,43 +452,42 @@ Let's look at these defaults in the :code:`CMakePackage` class in the :code:`_st :emphasize-lines: 87,96 :linenos: -Some :code:`CMake` packages use different generators. +Some ``CMake`` packages use different generators. Spack is able to support Unix-Makefile_ generators as well as Ninja_ generators. .. _Unix-Makefile: https://cmake.org/cmake/help/v3.4/generator/Unix%20Makefiles.html .. _Ninja: https://cmake.org/cmake/help/v3.4/generator/Ninja.html -If no generator is specified Spack will default to :code:`Unix Makefiles`. +If no generator is specified, Spack will default to ``Unix Makefiles``. Next we setup the build type. -In :code:`CMake` you can specify the build type that you want. +In ``CMake`` you can specify the build type that you want. Options include: -1. :code:`empty` -2. :code:`Debug` -3. :code:`Release` -4. :code:`RelWithDebInfo` -5. :code:`MinSizeRel` +1. ``empty`` +2. ``Debug`` +3. ``Release`` +4. ``RelWithDebInfo`` +5. ``MinSizeRel`` With these options you can specify whether you want your executable to have the debug version only, release version or the release with debug information. -Release executables tend to be more optimized than Debug. -In Spack, we set the default as Release unless otherwise specified through a variant. +Release executables tend to be more optimized than Debug versions. +In Spack, we set the default as Release unless otherwise specified through a variant (e.g., ``build_type=Debug``). -Spack then automatically sets up the :code:`-DCMAKE_INSTALL_PREFIX` path, appends the build type (:code:`RelWithDebInfo` default), and then specifies a verbose -:code:`Makefile`. +Spack then automatically sets up the ``-DCMAKE_INSTALL_PREFIX`` path, appends the build type (defaulting to ``RelWithDebInfo``), and enables a verbose ``Makefile`` output by default. -Next we add the :code:`rpaths` to :code:`-DCMAKE_INSTALL_RPATH:STRING`. +Next we add the ``rpaths`` to ``-DCMAKE_INSTALL_RPATH:STRING``. -Finally we add to :code:`-DCMAKE_PREFIX_PATH:STRING` the locations of all our dependencies so that :code:`CMake` can find them. +Finally we add to ``-DCMAKE_PREFIX_PATH:STRING`` the locations of all our dependencies so that ``CMake`` can find them. -In the end our :code:`cmake` line will look like this (example is :code:`xrootd`): +In the end our ``cmake`` line will look like this (example is ``xrootd``): .. code-block:: console $ cmake $HOME/spack/var/spack/stage/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/xrootd-4.6.0 -G Unix Makefiles -DCMAKE_INSTALL_PREFIX:PATH=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCMAKE_FIND_FRAMEWORK:STRING=LAST -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=FALSE -DCMAKE_INSTALL_RPATH:STRING=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/lib:$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/lib64 -DCMAKE_PREFIX_PATH:STRING=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/cmake-3.9.4-hally3vnbzydiwl3skxcxcbzsscaasx5 -We can see now how :code:`CMake` takes care of a lot of the boilerplate code that would have to be otherwise typed in. +We can see now how ``CMake`` takes care of a lot of the boilerplate code that would have to be otherwise typed in. Let's try to recreate callpath_: @@ -538,24 +527,23 @@ Again we fill in the details: :linenos: :emphasize-lines: 9,13,14,18,19,20,21,22,23 -As mentioned earlier, Spack will use sensible defaults to prevent repeated code and to make writing :code:`CMake` package files simpler. +As mentioned earlier, Spack's ``CMakePackage`` uses sensible defaults to reduce biolerplate and simplify writing package files for ``CMake``-based software. -In callpath, we want to add options to :code:`CALLPATH_WALKER` as well as add compiler flags. -We add the following options like so: +In ``callpath``, we want to control options like ``CALLPATH_WALKER`` or add specific compiler flags. +We can return these options from ``cmake_args()`` like so: .. literalinclude:: tutorial/examples/Cmake/2.package.py :language: python :linenos: :emphasize-lines: 26,30,31 -Now we can control our build options using :code:`cmake_args()`. +Now we can control our build options using ``cmake_args()``. If defaults are sufficient enough for the package, we can leave this method out. -:code:`CMakePackage` classes allow for control of other features in the -build system. -For example, you can specify the path to the "out of source" build directory and also point to the root of the :code:`CMakeLists.txt` file if it is placed in a non-standard location. +``CMakePackage`` classes allow for control of other features in the build system. +For example, you can specify the path to the "out of source" build directory and also point to the root of the ``CMakeLists.txt`` file if it is placed in a non-standard location. -A good example of a package that has its :code:`CMakeLists.txt` file located at a different location is found in :code:`spades`. +A good example of a package that has its ``CMakeLists.txt`` file located at a different location is found in ``spades``. .. code-block:: console @@ -565,17 +553,17 @@ A good example of a package that has its :code:`CMakeLists.txt` file located at root_cmakelists_dir = "src" -Here :code:`root_cmakelists_dir` will tell Spack where to find the location of :code:`CMakeLists.txt`. -In this example, it is located a directory level below in the :code:`src` directory. +Here ``root_cmakelists_dir`` will tell Spack where to find the location of ``CMakeLists.txt``. +In this example, it is located a directory level below in the ``src`` directory. -Some :code:`CMake` packages also require the :code:`install` phase to be overridden. -For example, let's take a look at :code:`sniffles`. +Some ``CMake`` packages also require the ``install`` phase to be overridden. +For example, let's take a look at ``sniffles``. .. code-block:: console $ spack edit sniffles -In the :code:`install()` method, we have to manually install our targets so we override the :code:`install()` method to do it for us: +In the ``install()`` method, we have to manually install our targets so we override the ``install()`` method to do it for us: .. code-block:: python @@ -600,8 +588,8 @@ These modules are usually installed using the following line: $ pip install . -We can write package files for Python packages using the :code:`Package` class, but the class brings with it a lot of methods that are useless for Python packages. -Instead, Spack has a :code:`PythonPackage` subclass that allows packagers of Python modules to be able to invoke :code:`pip`. +We can write package files for Python packages using the ``Package`` class, but the class brings with it a lot of methods that are useless for Python packages. +Instead, Spack has a ``PythonPackage`` subclass that allows packagers of Python modules to be able to invoke ``pip``. We will write a package file for Pandas_: @@ -638,9 +626,9 @@ We have the choice of providing build options or using the sensible defaults Luckily for us, there is no need to provide build args. Next we need to find the dependencies of a package. -Dependencies are usually listed in :code:`setup.py`. -You can find the dependencies by searching for -:code:`install_requires` keyword in that file. Here it is for :code:`Pandas`: +Dependencies are usually listed in ``setup.py``. +You can find the dependencies by searching for ``install_requires`` keyword in that file. +Here it is for ``Pandas``: .. code-block:: python @@ -665,9 +653,9 @@ You can find a more comprehensive list at the Pandas documentation_. .. _documentation: https://pandas.pydata.org/pandas-docs/stable/install.html -By reading the documentation and :code:`setup.py` we found that :code:`Pandas` depends on :code:`python-dateutil`, :code:`pytz`, and :code:`numpy`, :code:`numexpr`, and finally :code:`bottleneck`. +By reading the documentation and ``setup.py`` we found that ``Pandas`` depends on ``python-dateutil``, ``pytz``, and ``numpy``, ``numexpr``, and finally ``bottleneck``. -Here is the completed :code:`Pandas` script: +Here is the completed ``Pandas`` script: .. literalinclude:: tutorial/examples/PyPackage/1.package.py :language: python @@ -678,7 +666,7 @@ Spack can "activate" Python packages to prevent the user from having to load eac If a dependency is missed, Spack will be unable to properly activate the package and it will cause an issue. To learn more about extensions go to `spack extensions `_. -From this example, you can see that building Python modules is made easy through the :code:`PythonPackage` class. +From this example, you can see that building Python modules is made easy through the ``PythonPackage`` class. ------------------- Other Build Systems @@ -686,16 +674,16 @@ Other Build Systems Although we won't get in depth with any of the other build systems that Spack supports, it is worth mentioning that Spack does provide subclasses for the following build systems: -1. :code:`IntelPackage` -2. :code:`SconsPackage` -3. :code:`WafPackage` -4. :code:`RPackage` -5. :code:`PerlPackage` -6. :code:`QMakePackage` +1. ``IntelPackage`` +2. ``SconsPackage`` +3. ``WafPackage`` +4. ``RPackage`` +5. ``PerlPackage`` +6. ``QMakePackage`` Each of these classes have their own abstractions to help assist in writing package files. -For whatever doesn't fit nicely into the other build-systems, you can use the :code:`Package` class. +For whatever doesn't fit nicely into the other build-systems, you can use the ``Package`` class. Hopefully by now you can see how we aim to make packaging simple and robust through these classes. If you want to learn more about these build systems, check out `Implementing the installation procedure `_ in the Packaging Guide.