Skip to content

Commit e1845b2

Browse files
authored
Fix typos and add suggestions (#433)
* Update basics section * Fix typos and suggestions for binary cache * Fix typos and add suggestions to buildsystems
1 parent 7af58f1 commit e1845b2

3 files changed

Lines changed: 174 additions & 187 deletions

File tree

tutorial_basics.rst

Lines changed: 23 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -75,8 +75,8 @@ Let's go ahead and install ``gmake``,
7575

7676
You will see Spack installed ``gmake``, ``gcc-runtime``, and ``glibc``.
7777
The ``glibc`` and ``gcc-runtime`` packages are automatically tracked by Spack to manage consistency requirements among compiler runtimes.
78-
These do not represent separate installs from source, but represent aspects of the compiler Spack used for the install.
79-
For the rest of this section, we will ignore these components and focus on the rest of the install.
78+
They do not represent separate software builds from source, but are records of the system's compiler runtime components Spack used for the install.
79+
For the rest of this section, we will ignore these components and focus on the packages explicitly installed.
8080

8181
Spack can install software either from source or from a binary cache.
8282
Packages in the binary cache are signed with GPG for security.
@@ -89,14 +89,14 @@ To be able to install from the binary cache, we will need to configure Spack wit
8989
You'll learn more about configuring Spack later in the tutorial, but for now you will be able to install the rest of the packages in the tutorial from a binary cache using the same ``spack install`` command.
9090
By default this will install the binary cached version if it exists and fall back on installing from source if it does not.
9191

92-
Spack's spec syntax is the interface by which we can request specific configurations of the package.
92+
Spack's "spec" syntax is the interface by which we can request specific configurations of a package.
9393
The ``%`` sigil is used to specify compilers.
9494

9595
.. literalinclude:: outputs/basics/zlib-clang.out
9696
:language: console
9797

9898
Note that this installation is located separately from the previous one.
99-
We will discuss this in more detail later, but this is part of what allows Spack to support arbitrarily versioned software.
99+
We will discuss this in more detail later, but this is part of what allows Spack to support many versions of software packages.
100100

101101
You can check for particular versions before requesting them.
102102
We will use the ``spack versions`` command to see the available versions, and then install a different version of ``zlib-ng``.
@@ -115,7 +115,7 @@ The ``@`` sigil is used to specify versions, both of packages and of compilers.
115115
The spec syntax also includes compiler flags.
116116
Spack accepts ``cppflags``, ``cflags``, ``cxxflags``, ``fflags``, ``ldflags``, and ``ldlibs`` parameters.
117117
The values of these fields must be quoted on the command line if they include spaces.
118-
These values are injected into the compile line automatically by the Spack compiler wrappers.
118+
These values are injected into the compilation commands automatically by the Spack compiler wrappers.
119119

120120
.. literalinclude:: outputs/basics/zlib-O3.out
121121
:language: console
@@ -134,7 +134,7 @@ Spack generates a hash for each spec.
134134
This hash is a function of the full provenance of the package, so any change to the spec affects the hash.
135135
Spack uses this value to compare specs and to generate unique installation directories for every combinatorial version.
136136
As we move into more complicated packages with software dependencies, we can see that Spack reuses existing packages to satisfy a dependency.
137-
By default, Spack tries hard to reuse existing installations as dependencies, either from a local store or from configured remote buildcaches.
137+
By default, Spack tries hard to reuse existing installations as dependencies, either from a local store or from configured remote binary caches.
138138
This minimizes unwanted rebuilds of common dependencies, in particular if you update Spack frequently.
139139

140140
.. literalinclude:: outputs/basics/tcl.out
@@ -165,7 +165,7 @@ Note that each package has a top-level entry, even if it also appears as a depen
165165

166166
Let's move on to slightly more complicated packages.
167167
HDF5 is a good example of a more complicated package, with an MPI dependency.
168-
If we install it "out of the box," it will build with OpenMPI.
168+
If we install it with default settings it will build with OpenMPI.
169169

170170
.. literalinclude:: outputs/basics/hdf5.out
171171
:language: console
@@ -181,14 +181,14 @@ Here we can install HDF5 without MPI support.
181181
:language: console
182182

183183
We might also want to install HDF5 with a different MPI implementation.
184-
While MPI is not a package itself, packages can depend on abstract interfaces like MPI.
185-
Spack handles these through "virtual dependencies." A package, such as HDF5, can depend on the MPI interface.
186-
Other packages (``openmpi``, ``mpich``, ``mvapich2``, etc.) provide the MPI interface.
187-
Any of these providers can be requested for an MPI dependency.
188-
For example, we can build HDF5 with MPI support provided by MPICH by specifying a dependency on ``mpich``.
184+
While ``mpi`` itself is a virtual package representing an interface, other packages can depend on such abstract interfaces.
185+
Spack handles these through "virtual dependencies." A package, such as HDF5, can depend on the ``mpi`` virtual package (the interface).
186+
Actual MPI implementation packages (like ``openmpi``, ``mpich``, ``mvapich2``, etc.) provide the MPI interface.
187+
Any of these providers can be requested to satisfy an MPI dependency.
188+
For example, we can build HDF5 with MPI support provided by MPICH by specifying a dependency on ``mpich`` (e.g., ``hdf5 ^mpich``).
189189
Spack also supports versioning of virtual dependencies.
190-
A package can depend on the MPI interface at version 3, and provider packages specify what version of the interface *they* provide.
191-
The partial spec ``^mpi@3`` can be satisfied by any of several providers.
190+
A package can depend on the MPI interface at version 3 (e.g., ``hdf5 ^mpi@3``), and provider packages specify what version of the interface *they* provide.
191+
The partial spec ``^mpi@3`` can be satisfied by any of several MPI implementation packages that provide MPI version 3.
192192

193193
.. literalinclude:: outputs/basics/hdf5-hl-mpi.out
194194
:language: console
@@ -212,7 +212,7 @@ Now let's look at an even more complicated package.
212212
:language: console
213213

214214
Now we're starting to see the power of Spack.
215-
Trilinos in its default configuration has 23 top level dependencies, many of which have dependencies of their own.
215+
Trilinos in its default configuration has 23 direct dependencies, many of which have dependencies of their own.
216216
Installing more complex packages can take days or weeks even for an experienced user.
217217
Although we've done a binary installation for the tutorial, a source installation of Trilinos using Spack takes about 3 hours (depending on the system), but only 20 seconds of programmer time.
218218

@@ -269,16 +269,18 @@ We can uninstall packages by spec using the same syntax as install.
269269

270270
We can also uninstall packages by referring only to their hash.
271271

272-
We can use either ``-f`` (force) or ``-R`` (remove dependents as well) to remove packages that are required by another installed package.
272+
We can use either the ``--force`` (or ``-f``) flag or the ``--dependents`` (or ``-R``) flag to remove packages that are required by another installed package.
273+
Use ``--force`` to remove just the specified package, leaving dependents broken.
274+
Use ``--dependents`` to remove the specified package and all of its dependents.
273275

274276
.. literalinclude:: outputs/basics/uninstall-needed.out
275277
:language: console
276278

277279
.. literalinclude:: outputs/basics/uninstall-r-needed.out
278280
:language: console
279281

280-
Spack will not uninstall packages that are not sufficiently specified.
281-
The ``-a`` (all) flag can be used to uninstall multiple packages at once.
282+
Spack will not uninstall packages that are not sufficiently specified specified (i.e., if the spec is ambiguous and matches multiple installed packages).
283+
The ``--all`` (or ``-a``) flag can be used to uninstall all packages matching an ambiguous spec.
282284

283285
.. literalinclude:: outputs/basics/uninstall-ambiguous.out
284286
:language: console
@@ -314,13 +316,13 @@ Customizing Compilers
314316
---------------------
315317

316318
Spack manages a list of available compilers on the system, detected automatically from the user's ``PATH`` variable.
317-
The ``spack compilers`` command is an alias for the command ``spack compiler list``.
319+
The ``spack compilers`` command is an alias for ``spack compiler list``.
318320

319321
.. literalinclude:: outputs/basics/compilers.out
320322
:language: console
321323

322-
The compilers are maintained in a YAML file.
323-
Later in the tutorial you will learn how to configure compilers by hand for special cases.
324+
The compilers are maintained in a YAML file (``compilers.yaml``).
325+
Later in the tutorial, you will learn how to configure compilers by hand for special cases.
324326
Spack also has tools to add compilers, and compilers built with Spack can be added to the configuration.
325327

326328
.. literalinclude:: outputs/basics/install-gcc-12.1.0.out

tutorial_binary_cache.rst

Lines changed: 25 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -11,10 +11,10 @@
1111
Binary Caches Tutorial
1212
==================================
1313

14-
In this section of the tutorial you will learn how to share Spack built binaries across machines and users using build caches.
14+
In this section of the tutorial, you will learn how to share Spack-built binaries across machines and users using build caches.
1515

16-
We will explore a few concepts that apply to all types of build caches, but the focus is primarily on **OCI container registries** like Docker Hub or Github Packages as a storage backend for binary caches.
17-
Spack supports a range of storage backends, like an ordinary filesystem, S3, and Google Cloud Storage, but OCI build caches have a few interesting properties that make them worth exploring more in depth.
16+
We will explore a few concepts that apply to all types of build caches, but the focus is primarily on **OCI container registries** (like Docker Hub or Github Packages) as a storage backend for binary caches.
17+
Spack supports a range of storage backends, such as an ordinary filesystem, Amazon S3, and Google Cloud Storage, but OCI build caches have a few interesting properties that make them worth exploring more in-depth.
1818

1919
Before we configure a build cache, let's install the ``julia`` package, which is an interesting example because it has some non-trivial dependencies like ``llvm``, and features an interactive REPL that we can use to verify that the installation works.
2020

@@ -42,20 +42,22 @@ Setting up an OCI build cache on GitHub Packages
4242

4343
For this tutorial we will be using GitHub Packages as an OCI registry, since most people have a GitHub account and it's easy to use.
4444

45-
First go to `<https://github.com/settings/tokens>`_ to generate a Personal access token with ``write:packages`` permissions.
45+
First, go to `<https://github.com/settings/tokens>`_ to generate a Personal Access Token (classic) with ``write:packages`` permissions.
4646
Copy this token.
4747

48-
Next, we will add this token to the mirror config section of the Spack environment:
48+
Next, we will add this token to the mirror configuration section for the Spack environment.
49+
Replace `<your-github-username>` with your GitHub username and `<your-github-username-or-org>` with your GitHub username or an organization where you have permission to create packages.
50+
The build cache name, `buildcache-${USER}-${HOSTNAME}`, is a suggestion; you can choose your own.
4951

5052
.. code-block:: console
5153
52-
$ export MY_OCI_TOKEN=<token>
54+
$ export MY_OCI_TOKEN=<paste-your-token-here>
5355
$ spack -e . mirror add \
54-
--oci-username <user> \
56+
--oci-username <your-github-username> \
5557
--oci-password-variable MY_OCI_TOKEN \
5658
--unsigned \
5759
my-mirror \
58-
oci://ghcr.io/<github_user>/buildcache-${USER}-${HOSTNAME}
60+
oci://ghcr.io/<your-github-username-or-org>/buildcache-${USER}-${HOSTNAME}
5961
6062
6163
.. note ::
@@ -101,7 +103,7 @@ which outputs
101103
...
102104
==> Pushed julia@1.9.3/dfzhutf to ghcr.io/<user>/buildcache-<user>-<host>:julia-1.9.3-dfzhutfh3s2ekaltdmujjn575eip5uhl.spack
103105
104-
The location of the pushed package
106+
The location of the pushed package, when referred to as a OCI image, will be:
105107

106108
.. code-block:: text
107109
@@ -142,7 +144,7 @@ The easiest way to do this is to override the ``mirrors`` config section in the
142144
secret_variable: MY_OCI_TOKEN
143145
signed: false
144146
145-
An "overwrite install" should be enough to show that the build cache is used:
147+
An "overwrite install" should be enough to show that the build cache is used (output will vary based on your specific configuration):
146148

147149
.. code-block:: console
148150
@@ -177,10 +179,9 @@ For convenience you can also run ``spack buildcache push --update-index ...`` to
177179

178180
.. note::
179181

180-
As of Spack 0.22, build caches can be used across different Linux distros. The concretizer
181-
will reuse specs that have a host compatible ``libc`` dependency (e.g. ``glibc`` or ``musl``).
182-
For packages compiled with ``gcc`` (and a few others), users do not have to install compilers
183-
first, as the build cache contains the compiler runtime libraries as a separate package.
182+
As of Spack 0.22, build caches can be used across different Linux distros.
183+
The concretizer will reuse specs that have a host-compatible ``libc`` dependency (e.g. ``glibc`` or ``musl``).
184+
For packages compiled with ``gcc`` (and a few other compilers), users do not have to install compilers first, as the build cache contains the compiler runtime libraries as a separate package dependency.
184185

185186
After an index is created, it's possible to list the available packages in the build cache:
186187

@@ -255,7 +256,7 @@ Let's add a simple text editor like ``vim`` to our previous environment next to
255256
256257
$ spack -e . install --add vim
257258
258-
This time we push to the OCI registry, but also pass ``--tag julia-and-vim`` to instruct Spack to create an image for the environment as a whole, with a human-readable tag:
259+
This time, when we push to the OCI registry, we also pass ``--tag julia-and-vim`` to instruct Spack to create an additional image tag for the environment as a whole, with a more human-readable name:
259260

260261

261262
.. code-block:: console
@@ -291,12 +292,8 @@ For those familiar with ``Dockerfile`` syntax, it would structurally look like t
291292
292293
This approach is still valid, and the ``spack containerize`` command continues to exist, but it has a few downsides:
293294

294-
* When ``RUN spack -e /root/env install`` fails, ``docker`` will not cache the layer, meaning
295-
that all dependencies that did install successfully are lost. Troubleshooting the build
296-
typically means starting from scratch in ``docker run`` or on the host system.
297-
* In certain CI environments, it is not possible to use ``docker build``. For example, the
298-
CI script itself may already run in a docker container, and running ``docker build`` *safely*
299-
inside a container is tricky.
295+
* When ``RUN spack -e /root/env install`` fails, ``docker`` will not cache the layer, meaning that all dependencies that did install successfully are lost. Troubleshooting the build typically means starting from scratch either within a ``docker run`` session or on the host system.
296+
* In certain CI environments, it is not possible to use ``docker build`` directly. For example, the CI script itself may already run in a Docker container, and running ``docker build`` *safely* inside a container (Docker-in-Docker) is tricky.
300297

301298
The takeaway is that Spack decouples the steps that ``docker build`` combines: build isolation, running the build, and creating an image.
302299
You can run ``spack install`` on your host machine or in a container, and run ``spack buildcache push`` separately to create an image.
@@ -307,15 +304,15 @@ Relocation
307304

308305
Spack is different from many package managers in that it lets users choose where to install packages.
309306
This makes Spack very flexible, as users can install packages in their home directory and do not need root privileges.
310-
The downside is that sharing binaries is more complicated, as binaries may contain hard-coded, absolute paths to machine specific locations, which have to be adjusted when binaries are installed on a different machine.
307+
The downside is that sharing binaries is more complicated, as binaries may contain hard-coded, absolute paths to machine specific locations, which have to be adjusted when these binaries are installed on a different machine or in a different path.
311308

312309
Fortunately Spack handles this automatically upon install from a binary cache.
313310
But when you build binaries that are intended to be shared, there is one thing you have to keep in mind: Spack can relocate hard-coded paths in binaries *provided that the target prefix is shorter than the prefix used during the build*.
314311

315312
The reason is that binaries typically embed these absolute paths in string tables, which is a list of null terminated strings, to which the program stores offsets.
316313
That means we can only modify strings in-place, and if the new path is longer than the old one, we would overwrite the next string in the table.
317314

318-
To maximize the chances of successful relocation, you should build your binaries in a relative long path.
315+
To maximize the chances of successful relocation, you should build your binaries in a relatively long path.
319316
Fortunately Spack can automatically pad paths to make them longer, using the following command:
320317

321318
.. code-block:: console
@@ -327,9 +324,9 @@ Using build caches in CI
327324
------------------------
328325

329326
Build caches are a great way to speed up CI pipelines.
330-
Both GitHub Actions and Gitlab CI support container registries, and this tutorial should give you a good starting point to leverage them.
327+
Both GitHub Actions and GitLab CI support container registries, and this tutorial should give you a good starting point to leverage them.
331328

332-
Spack also provides a basic GitHub Action to already provide you with a binary cache:
329+
Spack also provides a basic GitHub Action that already provides you with a binary cache:
333330

334331
.. code-block:: yaml
335332
@@ -349,6 +346,6 @@ Summary
349346

350347
In this tutorial we have created a build cache on top of an OCI registry, which can be used
351348

352-
* to ``spack install julia vim`` on machines without source builds
353-
* to automatically create container images for individual packages while pushing to the cache
354-
* to create container images for multiple packages at once
349+
* to run ``spack install julia vim`` on machines and have Spack fetch pre-built binaries instead of building from source.
350+
* to automatically create container images for individual packages when pushing to the cache.
351+
* to create container images for entire Spack environments (multiple packages) at once.

0 commit comments

Comments
 (0)