Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions tutorial_advanced_packaging.rst
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ Now, you are ready to set your preferred ``EDITOR`` and continue with the rest o
Modifying a Package's Build Environment
---------------------------------------

Spack sets up several environment variables like ``PATH`` by default to aid in building a package, but many packages make use of environment variables which convey specific information about their dependencies (e.g., ``MPICC``).
Spack sets up several environment variables like ``PATH`` by default to aid in building a package, but many packages make use of environment variables that convey specific information about their dependencies (e.g., ``MPICC``).
This section covers how to update your Spack packages so that package-specific environment variables are defined at build-time.

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Expand All @@ -78,7 +78,7 @@ For example, when a package depends on a python extension like py-numpy, Spack's

To provide environment setup for a dependent, a package can implement the
:py:func:`setup_dependent_build_environment
<spack.package.PackageBase.setup_dependent_build_environment>` and or :py:func:`setup_dependent_run_environment <spack.package.PackageBase.setup_dependent_run_environment>` functions.
<spack.package.PackageBase.setup_dependent_build_environment>` and/or :py:func:`setup_dependent_run_environment <spack.package.PackageBase.setup_dependent_run_environment>` functions.
These functions take as a parameter a :py:class:`EnvironmentModifications <spack.util.environment.EnvironmentModifications>` object, which includes convenience methods to update the environment.
For example, an MPI implementation can set ``MPICC`` for build-time use for packages that depend on it:

Expand Down Expand Up @@ -379,9 +379,9 @@ Other Packaging Topics
Attach attributes to other packages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Build tools usually also provide a set of executables that can be used when another package is being installed.
Build tools also usually provide a set of executables that can be used when another package is being installed.
Spack gives you the opportunity to monkey-patch dependent modules and attach attributes to them.
This helps make the packager experience as similar as possible to what would have been the manual installation of the same package.
This helps make the packager's experience as similar as possible to what would have been the manual installation of the same package.

An example here is the ``automake`` package, which overrides
:py:func:`setup_dependent_package <spack.package.PackageBase.setup_dependent_package>`:
Expand Down
2 changes: 1 addition & 1 deletion tutorial_basics.rst
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,7 @@ Use ``--dependents`` to remove the specified package and all of its dependents.
.. literalinclude:: outputs/basics/uninstall-r-needed.out
:language: console

Spack will not uninstall packages that are not sufficiently specified specified (i.e., if the spec is ambiguous and matches multiple installed packages).
Spack will not uninstall packages that are not sufficiently specified (i.e., if the spec is ambiguous and matches multiple installed packages).
The ``--all`` (or ``-a``) flag can be used to uninstall all packages matching an ambiguous spec.

.. literalinclude:: outputs/basics/uninstall-ambiguous.out
Expand Down
12 changes: 6 additions & 6 deletions tutorial_binary_cache.rst
Original file line number Diff line number Diff line change
Expand Up @@ -103,13 +103,13 @@ which outputs
...
==> Pushed julia@1.9.3/dfzhutf to ghcr.io/<user>/buildcache-<user>-<host>:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack

The location of the pushed package, when referred to as a OCI image, will be:
The location of the pushed package, when referred to as an OCI image, will be:

.. code-block:: text

ghcr.io/<github_user>/buildcache-<user>-<host>:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack

looks very similar to a container image --- we will get to that in a bit.
look very similar to a container image --- we will get to that in a bit.

.. note ::

Expand Down Expand Up @@ -207,7 +207,7 @@ We can already attempt to run the image associated with the ``julia`` package th
exec /home/spack/spack/opt/spack/linux-ubuntu22.04-x86_64_v3/gcc-11.4.0/julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl/bin/julia: no such file or directory

but immediately we see it fails.
The reason is that one crucial part is missing, and that is a ``glibc``, which Spack always treats as an external package.
The reason is that one crucial part is missing, and that is ``glibc``, which Spack always treats as an external package.

To fix this, we force push to the registry again, but this time we specify a base image with a recent version of ``glibc``, for example from ``ubuntu:24.04``:

Expand All @@ -231,7 +231,7 @@ This time it works!
The minimal ``ubuntu:24.04`` image provides us not only with ``glibc``, but also other utilities like a shell.

Notice that you can use any base image of choice, like ``fedora`` or ``rockylinux``.
The only constraint is that it has a ``libc`` compatible with the external in the Spack built the binaries.
The only constraint is that it has a ``libc`` compatible with the external ``libc`` Spack used to build the binaries.
Spack does not validate this.

--------------------------------------
Expand All @@ -243,7 +243,7 @@ If you've paid attention to the output of some of the commands we have run so fa
Every Spack package corresponds to a single layer in each image, and the layers are shared across the different image tags.

Because Spack installs every package into a unique prefix, it is incredibly easy to compose multiple packages into a container image.
In contrast to Docker images built from commands in a ``Dockerfile`` where each command is run in order, Spack package layers are independent, and can in principle be combined in any order.
In contrast to Docker images built from commands in a ``Dockerfile`` where each command is run in sequence, Spack package layers are independent, and can in principle be combined in any order.

Let's add a simple text editor like ``vim`` to our previous environment next to ``julia``, so that we could both edit and run Julia code.

Expand Down Expand Up @@ -309,7 +309,7 @@ The downside is that sharing binaries is more complicated, as binaries may conta
Fortunately Spack handles this automatically upon install from a binary cache.
But when you build binaries that are intended to be shared, there is one thing you have to keep in mind: Spack can relocate hard-coded paths in binaries *provided that the target prefix is shorter than the prefix used during the build*.

The reason is that binaries typically embed these absolute paths in string tables, which is a list of null terminated strings, to which the program stores offsets.
The reason is that binaries typically embed these absolute paths in string tables, which is a list of null-terminated strings, to which the program stores offsets.
That means we can only modify strings in-place, and if the new path is longer than the old one, we would overwrite the next string in the table.

To maximize the chances of successful relocation, you should build your binaries in a relatively long path.
Expand Down
30 changes: 15 additions & 15 deletions tutorial_buildsystems.rst
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,8 @@ Package Class Hierarchy
}

The above diagram gives a high-level view of the class hierarchy and how each package relates.
Each build system specific class inherits from the ``PackageBaseClass`` superclass.
The bulk of the common work is done in this superclass which includes fetching, extracting to a staging directory and installing.
Each build system specific class inherits from the ``PackageBase`` superclass.
The bulk of the common work is done in this superclass which includes fetching, extracting to a staging directory and the install process.
Each subclass then adds additional build-system-specific functionality.
In the following sections, we will go over examples of how to utilize each subclass and see how powerful these abstractions are when packaging.

Expand Down Expand Up @@ -80,7 +80,7 @@ The ``AutotoolsPackage`` subclass aims to simplify writing package files for Aut
4. ``install()``


Each of these phases have sensible defaults.
Each of these phases has sensible defaults.
Let's take a quick look at some of the internals of the ``Autotools`` class:

.. code-block:: console
Expand All @@ -103,7 +103,7 @@ This will open the ``AutotoolsPackage`` file in your text editor.

Important to note are the highlighted lines.
These properties allow the packager to set what build targets and install targets they want for their package.
If, for example, we wanted to add as our build target ``foo`` then we can append to our ``build_targets`` property:
If, for example, we wanted to add as our build target ``foo`` then we can append to our ``build_targets`` list:

.. code-block:: python

Expand Down Expand Up @@ -151,16 +151,16 @@ In fact, the only thing that needs to be overridden is ``configure_args()``.
:linenos:

Since Spack's ``AutotoolsPackage`` takes care of setting the prefix for us, we can exclude that as an argument to ``configure``.
Our package file looks simpler, and we don't not need to worry about whether we have properly included ``configure`` and ``make``.
Our package file looks simpler, and we don't need to worry about whether we have properly included ``configure`` and ``make``.

This version of the ``mpileaks`` package installs the same as the previous, but the ``AutotoolsPackage`` class lets us do it with a cleaner looking package file.

-----------------
Makefile
-----------------

Packages that utilize ``Make`` or a ``Makefile`` usually require you to edit a ``Makefile`` to set up platform and compiler specific variables.
These packages are handled by the ``Makefile`` subclass which provides convenience methods to help write these types of packages.
Packages that utilize ``Make`` or a ``Makefile`` usually require you to edit a ``Makefile`` to set up platform and compiler-specific variables.
These packages are handled by the ``MakefilePackage`` subclass which provides convenience methods to help write these types of packages.

A ``MakefilePackage`` build has three phases that can be overridden by the packager:

Expand Down Expand Up @@ -217,16 +217,16 @@ Once the fetching is completed, Spack will open up your text editor in the usual
:language: python
:linenos:

Spack was successfully able to detect that ``Bowtie`` uses ``Make``.
Spack was successfully able to detect that ``Bowtie`` uses ``Makefiles``.
Let's add in the rest of our details for our package:

.. literalinclude:: tutorial/examples/Makefile/1.package.py
:language: python
:emphasize-lines: 10,11,13,14,18,20
:linenos:

As we mentioned earlier, most packages using a ``Makefile`` have hard-coded variables that must be edited.
These variables are fine if you happen to not care about setup or types of compilers used but Spack is designed to work with any compiler.
As we mentioned earlier, most packages using a ``Makefile`` have hardcoded variables that must be edited.
These variables are fine if you happen to not care about setup or types of compilers used, but Spack is designed to work with any compiler.
The ``MakefilePackage`` subclass makes it easy to edit these ``Makefiles`` by having an ``edit()`` method that can be overridden.

Let's take a look at the default ``Makefile`` that ``Bowtie`` provides.
Expand Down Expand Up @@ -275,7 +275,7 @@ Let's change the build and install phases of our package:

Here we demonstrate another strategy that we can use to manipulate our package's build.
We can provide command-line arguments to ``make()``.
Since ``Bowtie`` can use ``tbb`` we can either add ``NO_TBB=1`` as a argument to prevent ``tbb`` support, or we can invoke ``make`` with no arguments if EBB is desired and found by its build system.
Since ``Bowtie`` can use ``tbb`` we can either add ``NO_TBB=1`` as a argument to prevent ``tbb`` support, or we can invoke ``make`` with no arguments if TBB is desired and found by its build system.

``Bowtie`` requires our ``install_target`` to provide a path to the install directory.
We can do this by providing ``prefix=`` as a command line argument to ``make()``.
Expand Down Expand Up @@ -472,7 +472,7 @@ Options include:

With these options you can specify whether you want your executable to have the debug version only, release version or the release with debug information.
Release executables tend to be more optimized than Debug versions.
In Spack, we set the default as Release unless otherwise specified through a variant (e.g., ``build_type=Debug``).
In Spack, we set the default as `Release` unless otherwise specified through a variant (e.g., ``build_type=Debug``).

Spack then automatically sets up the ``-DCMAKE_INSTALL_PREFIX`` path, appends the build type (defaulting to ``RelWithDebInfo``), and enables a verbose ``Makefile`` output by default.

Expand Down Expand Up @@ -527,7 +527,7 @@ Again we fill in the details:
:linenos:
:emphasize-lines: 9,13,14,18,19,20,21,22,23

As mentioned earlier, Spack's ``CMakePackage`` uses sensible defaults to reduce biolerplate and simplify writing package files for ``CMake``-based software.
As mentioned earlier, Spack's ``CMakePackage`` uses sensible defaults to reduce boilerplate and simplify writing package files for ``CMake``-based software.

In ``callpath``, we want to control options like ``CALLPATH_WALKER`` or add specific compiler flags.
We can return these options from ``cmake_args()`` like so:
Expand Down Expand Up @@ -621,7 +621,7 @@ And we are left with the following template:
:linenos:

As you can see this is not any different than any package template that we have written.
We have the choice of providing build options or using the sensible defaults
We have the choice of providing build options or using the sensible defaults.

Luckily for us, there is no need to provide build args.

Expand Down Expand Up @@ -683,7 +683,7 @@ Although we won't get in depth with any of the other build systems that Spack su


Each of these classes have their own abstractions to help assist in writing package files.
For whatever doesn't fit nicely into the other build-systems, you can use the ``Package`` class.
For whatever doesn't fit nicely into the other build systems, you can use the ``Package`` class.

Hopefully by now you can see how we aim to make packaging simple and robust through these classes.
If you want to learn more about these build systems, check out `Implementing the installation procedure <https://spack.readthedocs.io/en/latest/packaging_guide.html#installation-procedure>`_ in the Packaging Guide.