Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 17 additions & 17 deletions tutorial_configuration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,26 +24,26 @@ A partial list of some key configuration sections is provided below.
* - config
- General settings (install location, number of build jobs, etc)
* - concretizer
- Specializaiton of the concretizer behavior (reuse, unification, etc)
- Specialization of the concretizer behavior (reuse, unification, etc)
* - compilers
- Define the compilers that Spack can use (required and system specific)
* - mirrors
* - Mirrors
- Locations where spack can look for stashed source or binary distributions
* - packages
* - Packages
- Specific settings and rules for packages
* - modules
* - Modules
- Naming, location and additional configuration of Spack generated modules

The full list of sections can be viewed with ``spack config list``.
For further education we encourage you to explore the spack `documentation on configuration files <https://spack.readthedocs.io/en/latest/configuration.html#configuration-files>`_.
For further education, we encourage you to explore the spack `documentation on configuration files <https://spack.readthedocs.io/en/latest/configuration.html#configuration-files>`_.

The principle goals of this section of the tutorial are:

1. Introduce the configuration sections and scope hierarchy
2. Demonstrate how to manipulate configurations
3. Show how to configure system assets with spack (compilers and packages)

As such we will primarily focus on the ``compilers`` and ``packages`` configuration sections in this portion of the tutorial.
As such, we will primarily focus on the ``compilers`` and ``packages`` configuration sections in this portion of the tutorial.

We will explain this by first covering how to manipulate configurations from the command line and then show how this impacts the configuration file hierarchy.
We will then move into compiler and package configurations to help you develop skills for getting the builds you want on your system.
Expand All @@ -58,7 +58,7 @@ Configuration from the command line
-----------------------------------

You can run ``spack config blame [section]`` at any point in time to see what your current configuration is.
If you omit the section then spack will dump all the configurations settings to your screen.
If you omit the section, then spack will dump all the configurations settings to your screen.
Let's go ahead and run this for the ``concretizer`` section.

.. code-block:: console
Expand All @@ -80,10 +80,10 @@ If we rerun ``spack config blame concretizer`` we can see that the change was ap

$ spack config blame concretizer

Notice that the reference file on for this option is now different.
Notice that the reference file for this option is now different.
This indicates the scope where the configuration was set in, and we will discuss how spack chooses the default scope shortly.
For now, it is important to note that the ``spack config`` command accepts an optional ``--scope`` flag so we can be more precise in the configuration process.
This will make more sense after the next section which provides the definition of spack's configuration scopes and their hierarchy.
This will make more sense after the next section which provides the definition of Spack's configuration scopes and their hierarchy.

.. _configs-tutorial-scopes:

Expand Down Expand Up @@ -210,7 +210,7 @@ Compiler Configuration

For most tasks, we can use Spack with the compilers auto-detected the first time Spack runs on a system.
As discussed in the basic installation tutorial, we can also tell Spack where compilers are located using the ``spack compiler add`` command.
However, in some circumstances we want even more fine-grained control over the compilers available.
However, in some circumstances, we want even more fine-grained control over the compilers available.
This section will teach you how to exercise that control using the compilers configuration file.

We will start by opening the compilers configuration file:
Expand Down Expand Up @@ -243,8 +243,8 @@ We start with no active environment, so this will open a ``compilers.yaml`` file
paths:
cc: /usr/bin/gcc-10
cxx: /usr/bin/g++-10
f77: usr/bin/gfortran-10
fc: usr/bin/gfortran-10
f77: /usr/bin/gfortran-10
fc: /usr/bin/gfortran-10
flags: {}
operating_system: ubuntu22.04
target: x86_64
Expand Down Expand Up @@ -308,7 +308,7 @@ We can verify that our new compiler works by invoking it now:


This new compiler also works on Fortran codes.
We'll show it by compiling a small package using as a build dependency ``cmake%gcc@11.4.0`` since it is already available in our binary cache:
We'll show this by compiling a small package using ``cmake%gcc@11.4.0`` as a build dependency, since it is already available in our binary cache:

.. code-block:: console

Expand Down Expand Up @@ -521,7 +521,7 @@ The packages configuration file also controls when Spack will build against an e
Spack has a ``spack external find`` command that can automatically discover and register externally installed packages.
This works for many common build dependencies, but it's also important to know how to do this manually for packages that Spack cannot yet detect.

On these systems we have a pre-installed curl.
On these systems, we have a pre-installed curl.
Let's tell Spack about this package and where it can be found:

.. code-block:: yaml
Expand Down Expand Up @@ -591,15 +591,15 @@ Now Spack will be forced to choose the external Curl.

This gets slightly more complicated with virtual dependencies.
Suppose we don't want to build our own MPI, but we now want a parallel version of HDF5.
Well, fortunately we have MPICH installed on these systems.
Well, fortunately, we have MPICH installed on these systems.

Instead of manually configuring an external for MPICH like we did for Curl we will use the ``spack external find`` command.
For packages that support this option, this is a useful way to avoid typos and get more accurate external specs.

.. literalinclude:: outputs/config/0.external_find.out
:language: console

To express that we don't want any other MPI installed we can use the virtual ``mpi`` package as a key.
To express that we don't want any other MPI installed, we can use the virtual ``mpi`` package as a key.
While we're editing the ``spack.yaml`` file, make sure to configure HDF5 to be able to build with MPI again:

.. code-block:: yaml
Expand Down Expand Up @@ -793,7 +793,7 @@ If we uninstall and reinstall zlib-ng, we see that it now uses only 2 cores:

Obviously, if you want to build everything in serial for whatever reason, you would set ``build_jobs`` to 1.

Last we'll unset ``concretizer:reuse:false`` since we'll want to enable concretizer reuse for the rest of this tutorial.
Last, we'll unset ``concretizer:reuse:false`` since we'll want to enable concretizer reuse for the rest of this tutorial.

.. code-block:: yaml

Expand Down
46 changes: 23 additions & 23 deletions tutorial_developer_workflows.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Developer Workflows Tutorial
============================

This tutorial will guide you through the process of using the ``spack develop`` command to develop software from local source code within a spack environment.
With this command spack will manage your dependencies while you focus on testing changes to your library and/or application.
With this command, Spack will manage your dependencies while you focus on testing changes to your library and/or application.


-----------------------------
Expand All @@ -22,13 +22,13 @@ Installing from local source
The ``spack install`` command, as you know, fetches source code from a mirror or the internet before building and installing your package.
As developers, we want to build from local source, which we will constantly change, build, and test.

Let's imagine for a second we're working on ``scr``. ``scr`` is a library used to implement scalable checkpointing in application codes.
Let's imagine, for a second, we're working on ``scr``. ``scr`` is a library used to implement scalable checkpointing in application codes.
It supports writing/reading checkpoints quickly and efficiently using MPI and high-bandwidth file I/O.
We'd like to test changes to scr within an actual application so we'll test with ``macsio``, a proxy application written to mimic typical HPC I/O workloads.
We'd like to test changes to ``scr`` within an actual application, so we'll test with ``macsio``, a proxy application written to mimic typical HPC I/O workloads.
We've chosen ``scr`` and ``macsio`` because together they are quick to build.

We'll start by making an environment for our development.
We need to build ``macsio`` with ``scr`` support, and we'd like everything to be built without fortran support for the time being.
We need to build ``macsio`` with ``scr`` support, and we'd like everything to be built without Fortran support for the time being.
Let's set up that development workflow.

.. literalinclude:: outputs/dev/setup-scr.out
Expand Down Expand Up @@ -145,8 +145,8 @@ In this case, it will be the 3.1.0 release that we want to write a patch for:
.. literalinclude:: outputs/dev/develop-1.out
:language: console

The spack develop command marks the package as being a "development" package in the spack.yaml.
This adds a special ``dev_path=`` attribute to the spec for the package, so spack remembers where the source code for this package is located.
The spack develop command marks the package as being a "development" package in the ``spack.yaml``.
This adds a special ``dev_path=`` attribute to the spec for the package, so Spack remembers where the source code for this package is located.
The develop command also downloads/checks out the source code for the package.
By default, the source code is downloaded into a subdirectory of the environment.
You can change the location of this source directory by modifying the ``path:`` attribute of the develop configuration in the environment.
Expand All @@ -156,13 +156,13 @@ There are a few gotchas with the spack develop command
* You often specify the package version manually when specifying a
package as a dev package. Spack needs to know the version of the dev
package so it can supply the correct flags for the package's build
system. If a version is not supplied then spack will take the maximum version
defined in the package where where `infinity versions <https://spack.readthedocs.io/en/latest/packaging_guide.html#version-comparison>`_ like ``develop`` and ``main``
system. If a version is not supplied, then Spack will take the maximum version
defined in the package where `infinity versions <https://spack.readthedocs.io/en/latest/packaging_guide.html#version-comparison>`_ like ``develop`` and ``main``
have a higher value than the numeric versions.
* You should ensure a spec for the package you are developing appears in the DAG of at least one of the roots of the environment with the same version that you are developing.
``spack add <package>`` with the matching version you want to develop is a way to ensure
the develop spec is satisfied.the ``spack.yaml`` environments file. This is because
develop specs are not concretization constraints but rather a criteria for adding
the develop spec is satisfied in the ``spack.yaml`` environments file. This is because
develop specs are not concretization constraints but rather criteria for adding
the ``dev_path=`` variant to existing spec.
* You'll need to re-concretize the environment so that the version
number and the ``dev_path=`` attributes are properly added to the
Expand All @@ -171,13 +171,13 @@ There are a few gotchas with the spack develop command
.. literalinclude:: outputs/dev/develop-conc.out
:language: console

Now that we have this done, we tell spack to rebuild both ``scr`` and ``macsio`` by running ``spack install``.
Now that we have this done, we tell Spack to rebuild both ``scr`` and ``macsio`` by running ``spack install``.

.. literalinclude:: outputs/dev/develop-2.out
:language: console

This rebuilds ``scr`` from the subdirectory we specified.
If your package uses cmake, spack will build the package in a build directory that matches the hash for your package.
If your package uses CMake, Spack will build the package in a build directory that matches the hash for your package.
From here you can change into the appropriate directory and perform your own build/test cycles.

Now, we can develop our code.
Expand All @@ -188,7 +188,7 @@ Let's edit a file and remove the first semi-colon we find.
:language: console

Once you have a development package, ``spack install`` also works much like "make".
Since spack knows the source code directory of the package, it checks the filetimes on the source directory to see if we've made recent changes.
Since Spack knows the source code directory of the package, it checks the filetimes on the source directory to see if we've made recent changes.
If the file times are newer, it will rebuild ``scr`` and any other package that depends on ``scr``.

.. literalinclude:: outputs/dev/develop-3.out
Expand All @@ -202,17 +202,17 @@ Now let's fix it and rebuild directly.
.. literalinclude:: outputs/dev/develop-4.out
:language: console

You'll notice here that spack rebuilt both ``scr`` and ``macsio``, as expected.
You'll notice here that Spack rebuilt both ``scr`` and ``macsio``, as expected.

Taking advantage of iterative builds with spack requires cooperation from your build system.
When spack performs a rebuild on a development package, it reruns all the build stages for your package without cleaning the source and build directories to a pristine state.
Taking advantage of iterative builds with Spack requires cooperation from your build system.
When Spack performs a rebuild on a development package, it reruns all the build stages for your package without cleaning the source and build directories to a pristine state.
If your build system can take advantage of the previously compiled object files then you'll end up with an iterative build.

- If your package just uses make, you also should get iterative builds
for free when running ``spack develop``.
- If your package uses cmake with the typical ``cmake`` / ``build`` /
- If your package uses CMake with the typical ``cmake`` / ``build`` /
``install`` build stages, you'll get iterative builds for free with
spack because cmake doesn’t modify the filetime on the
Spack because CMake doesn’t modify the filetime on the
``CMakeCache.txt`` file if your cmake flags haven't changed.
- If your package uses autoconf, then rerunning the typical
``autoreconf`` stage typically modifies the filetime of
Expand Down Expand Up @@ -244,7 +244,7 @@ Workflow Summary

Use the ``spack develop`` command with an environment to make a reproducible build environment for your development workflow.
Spack will set up all the dependencies for you and link all your packages together.
Within a development environment, ``spack install`` works similar to ``make`` in that it will check file times to rebuild the minimum number of spack packages necessary to reflect the changes to your build.
Within a development environment, ``spack install`` works similarly to ``make`` in that it will check file times to rebuild the minimum number of Spack packages necessary to reflect the changes to your build.

-------------------
Optional: Tips and Tricks
Expand All @@ -261,7 +261,7 @@ A list of the options for the ``spack develop`` can be viewed below:
Source Code Management
----------

``spack develop`` allows users to manipulate the source code locations The default behavior is to let spack manage its location and cloning operations, but software developers often want more control over these.
``spack develop`` allows users to manipulate the source code locations The default behavior is to let Spack manage its location and cloning operations, but software developers often want more control over these.

The source directory can be set with the ``--path`` argument when calling ``spack develop``.
If this directory already exists then ``spack develop`` will not attempt to fetch the code for you.
Expand All @@ -282,7 +282,7 @@ Navigation and the Build Environment
Diving into the build environment was introduced previously in the packaging section with the ``spack build-env scr -- bash`` command.
This is a helpful function because it allows you to run commands inside the build environment.
In the packages section of the tutorial this was combined with ``spack cd`` to produce a manual build outside of Spack's automated Process.
This command is particularly useful in developer environments -- it allows developers a streamlined workflow when iterating on a single package without the overhead of the ``spack install`` command.
This command is particularly useful in developer environmentsit allows developers a streamlined workflow when iterating on a single package without the overhead of the ``spack install`` command.
The additional features of the install command are unnecessary when tightly iterating between building and testing a particular package.
For example, the workflow modifying ``scr`` that we just went through can be simplified to:

Expand All @@ -303,7 +303,7 @@ For example, the workflow modifying ``scr`` that we just went through can be sim
$ make test
$ exit

Working with the build environment and along with spack navigation features provides a nice way to iterate quickly and navigate through the hash heavy spack directory structures.
Working with the build environment and along with Spack navigation features provides a nice way to iterate quickly and navigate through the hash-heavy Spack directory structures.

Combinatorics
------------
Expand All @@ -327,4 +327,4 @@ While we won't build out this example it illustrates how the ``dev_path`` for ``

Now if we want to do most of our incremental builds using the ``Release`` build and periodically check the results using the ``Debug`` build we can combine the workflow from the previous example: dive into the ``Release`` versions build environment using ``spack build-env scr build_type=Release -- bash`` and navigate with ``spack cd -b scr build_type=Release``.
Note that since there are two ``scr`` specs in the environment we must distinguish which one we want for these commands.
When we are ready to check our changes for the debug build we can exit out of the build environment subshell, rerun ``spack install`` to rebuild everything, and then inspect the debug build through our method of choice.
When we are ready to check our changes for the debug build, we can exit out of the build environment subshell, rerun ``spack install`` to rebuild everything, and then inspect the debug build through our method of choice.
Loading