You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: tutorial_developer_workflows.rst
+39-41Lines changed: 39 additions & 41 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,38 +11,38 @@
11
11
Developer Workflows Tutorial
12
12
============================
13
13
14
-
This tutorial will guide you through using the ``spack develop`` command to develop software from local source code within a Spack environment.
14
+
This tutorial will guide you through using the ``spack develop`` command to develop software from local source code within a Spack environment.
15
15
With this command, Spack manages your dependencies while you focus on testing changes to your library and/or application.
16
16
17
17
-----------------------------
18
18
Installing from Local Source
19
19
-----------------------------
20
20
21
-
The ``spack install`` command typically fetches source code from a mirror or the internet before building and installing your package.
21
+
The ``spack install`` command typically fetches source code from a mirror or the internet before building and installing your package.
22
22
As developers, however, we often want to build from local source code that we continuously modify, build, and test.
23
23
24
-
Let's imagine we're working on ``scr``—a library used to implement scalable checkpointing in application codes.
25
-
It supports fast, efficient checkpoint read/write operations using MPI and high-bandwidth file I/O.
24
+
Let's imagine we're working on ``scr``—a library used to implement scalable checkpointing in application codes.
25
+
It supports fast, efficient checkpoint read/write operations using MPI and high-bandwidth file I/O.
26
26
We want to test changes to ``scr`` within an actual application, so we'll use ``macsio``, a proxy application that mimics typical HPC I/O workloads.
27
27
28
28
We've chosen ``scr`` and ``macsio`` because they are both quick to build.
29
29
30
-
We'll begin by creating a Spack environment for our development work.
31
-
We need to build ``macsio`` with ``scr`` support, and for now, we want everything built without Fortran support.
30
+
We'll begin by creating a Spack environment for our development work.
31
+
We need to build ``macsio`` with ``scr`` support, and for now, we want everything built without Fortran support.
32
32
Let's set up this development workflow:
33
33
34
34
.. literalinclude:: outputs/dev/setup-scr.out
35
35
:language: console
36
36
37
-
Before making any changes, we verify that everything builds correctly.
37
+
Before making any changes, we verify that everything builds correctly.
38
38
Spack builds the entire development tree as specified and links all components together for you.
39
39
40
40
41
41
.. literalinclude:: outputs/dev/setup-scr.out
42
42
:language: console
43
43
44
44
45
-
Before making any changes, we verify that everything builds correctly.
45
+
Before making any changes, we verify that everything builds correctly.
46
46
Spack builds the entire development tree as specified and links all components together for you.
47
47
48
48
@@ -100,17 +100,17 @@ Now we are ready to begin work on the actual application.
100
100
Development Iteration Cycle
101
101
-----------------------------
102
102
103
-
Let's assume that ``scr`` has a bug, and we'd like to patch it to find out what the problem is.
104
-
First, we tell Spack that we want to check out the version of ``scr`` we intend to work on.
103
+
Let's assume that ``scr`` has a bug, and we'd like to patch it to find out what the problem is.
104
+
First, we tell Spack that we want to check out the version of ``scr`` we intend to work on.
105
105
In this case, it's the 3.1.0 release that we want to patch:
106
106
107
107
.. literalinclude:: outputs/dev/develop-1.out
108
108
:language: console
109
109
110
-
The ``spack develop`` command marks the package as a "development" package in the ``spack.yaml``.
111
-
This adds a special ``dev_path=`` attribute to the spec for the package, so Spack remembers where the source code is located.
112
-
The ``develop`` command also downloads or checks out the source code for the package.
113
-
By default, the source code is downloaded into a subdirectory of the environment.
110
+
The ``spack develop`` command marks the package as a "development" package in the ``spack.yaml``.
111
+
This adds a special ``dev_path=`` attribute to the spec for the package, so Spack remembers where the source code is located.
112
+
The ``develop`` command also downloads or checks out the source code for the package.
113
+
By default, the source code is downloaded into a subdirectory of the environment.
114
114
You can change the location of this source directory by modifying the ``path:`` attribute of the ``develop`` configuration in the environment.
115
115
116
116
There are a few gotchas with the ``spack develop`` command:
@@ -133,55 +133,55 @@ Now that this is done, we tell Spack to rebuild both ``scr`` and ``macsio`` by r
133
133
.. literalinclude:: outputs/dev/develop-2.out
134
134
:language: console
135
135
136
-
This rebuilds ``scr`` from the subdirectory we specified.
137
-
If your package uses CMake, Spack will build it in a directory corresponding to the hash of your package.
136
+
This rebuilds ``scr`` from the subdirectory we specified.
137
+
If your package uses CMake, Spack will build it in a directory corresponding to the hash of your package.
138
138
From there, you can change into the appropriate directory and perform your own build/test cycles.
139
139
140
-
Now we can develop our code.
141
-
For the sake of this demo, we're going to intentionally introduce an error.
140
+
Now we can develop our code.
141
+
For the sake of this demo, we're going to intentionally introduce an error.
142
142
Let’s edit a file and remove the first semicolon we find.
143
143
144
144
.. literalinclude:: outputs/dev/edit-1.out
145
145
:language: console
146
146
147
-
Once you have a development package, ``spack install`` works like ``make``.
148
-
Because Spack knows the source code directory of the package, it checks file timestamps to detect recent changes.
147
+
Once you have a development package, ``spack install`` works like ``make``.
148
+
Because Spack knows the source code directory of the package, it checks file timestamps to detect recent changes.
149
149
If any files have been modified, Spack will rebuild ``scr`` and its dependents.
150
150
151
151
.. literalinclude:: outputs/dev/develop-3.out
152
152
:language: console
153
153
154
-
Here, the build failed as expected.
155
-
We can inspect the build output in ``scr/spack-build-out.txt``.
154
+
Here, the build failed as expected.
155
+
We can inspect the build output in ``scr/spack-build-out.txt``.
156
156
Alternatively, to debug interactively, we can launch a shell within the build environment using:
157
157
158
158
``spack build-env scr@2.0 -- bash``
159
159
160
-
If that’s too much to remember, sourcing ``scr/spack-build-env.txt`` will set the appropriate environment variables so you can diagnose the build manually.
160
+
If that’s too much to remember, sourcing ``scr/spack-build-env.txt`` will set the appropriate environment variables so you can diagnose the build manually.
161
161
Now let's fix the issue and rebuild directly.
162
162
163
163
.. literalinclude:: outputs/dev/develop-4.out
164
164
:language: console
165
165
166
166
You'll notice that Spack rebuilt both ``scr`` and ``macsio``, as expected.
167
167
168
-
Taking advantage of iterative builds with Spack requires cooperation from your build system.
169
-
When Spack performs a rebuild on a development package, it reruns all the build stages without cleaning the source and build directories to a pristine state.
168
+
Taking advantage of iterative builds with Spack requires cooperation from your build system.
169
+
When Spack performs a rebuild on a development package, it reruns all the build stages without cleaning the source and build directories to a pristine state.
170
170
If your build system can reuse previously compiled object files, you’ll benefit from an iterative build.
171
171
172
172
- If your package uses ``make``, you should get iterative builds automatically when using ``spack develop``.
173
173
- If your package uses CMake with the standard ``cmake`` / ``build`` / ``install`` stages, you'll also get iterative builds automatically.
174
174
This is because CMake doesn’t modify the file time of ``CMakeCache.txt`` unless your CMake flags change.
175
175
- If your package uses Autoconf, rerunning the typical ``autoreconf`` stage will usually modify the timestamp of ``config.h``, which may trigger a full rebuild.
176
176
177
-
Multiple packages can also be marked as development packages.
177
+
Multiple packages can also be marked as development packages.
178
178
If we were co-developing ``macsio``, we could run:
179
179
180
180
.. literalinclude:: outputs/dev/develop-5.out
181
181
:language: console
182
182
183
-
Using development workflows also allows us to share our full development setup with other team members.
184
-
They can simply use our ``spack.yaml`` to create a new environment and replicate the entire build process.
183
+
Using development workflows also allows us to share our full development setup with other team members.
184
+
They can simply use our ``spack.yaml`` to create a new environment and replicate the entire build process.
185
185
For example, here we create another development environment:
186
186
187
187
.. literalinclude:: outputs/dev/otherdevel.out
@@ -198,15 +198,15 @@ When we're done developing, we simply tell Spack that it no longer needs to keep
198
198
Workflow Summary
199
199
-------------------
200
200
201
-
Use the ``spack develop`` command within an environment to create a reproducible build setup for your development workflow.
202
-
Spack will manage all dependencies and link your packages together automatically.
201
+
Use the ``spack develop`` command within an environment to create a reproducible build setup for your development workflow.
202
+
Spack will manage all dependencies and link your packages together automatically.
203
203
Within a development environment, ``spack install`` behaves similarly to ``make``: it checks file timestamps and rebuilds only the minimum set of Spack packages required to reflect your changes.
204
204
205
205
-------------------
206
206
Optional: Tips and Tricks
207
207
-------------------
208
208
209
-
This section covers additional features that complement the core tutorial above.
209
+
This section covers additional features that complement the core tutorial above.
210
210
Many of these tips are especially useful for specific projects and developer workflows.
211
211
212
212
A list of available options for the ``spack develop`` command can be viewed below:
@@ -219,11 +219,11 @@ A list of available options for the ``spack develop`` command can be viewed belo
219
219
Source Code Management
220
220
----------------------
221
221
222
-
The ``spack develop`` command allows users to control the location of source code.
222
+
The ``spack develop`` command allows users to control the location of source code.
223
223
By default, Spack manages source locations and handles cloning automatically, but software developers often prefer more control.
224
224
225
-
You can specify the source directory using the ``--path`` argument when invoking ``spack develop``.
226
-
If the specified directory already exists, Spack will not attempt to fetch the source code.
225
+
You can specify the source directory using the ``--path`` argument when invoking ``spack develop``.
226
+
If the specified directory already exists, Spack will not attempt to fetch the source code.
227
227
This allows developers to pre-clone repositories or use preferred directory paths as needed.
228
228
229
229
.. code-block:: console
@@ -238,10 +238,10 @@ This allows developers to pre-clone repositories or use preferred directory path
238
238
Navigation and the Build Environment
239
239
------------------------------------
240
240
241
-
Diving into the build environment was introduced earlier in the packaging section using the ``spack build-env scr -- bash`` command.
241
+
Diving into the build environment was introduced earlier in the packaging section using the ``spack build-env scr -- bash`` command.
242
242
This is a helpful feature because it allows you to run commands inside the package’s build environment.
243
243
244
-
In the packaging section, this was combined with ``spack cd`` to demonstrate a manual build process outside of Spack’s automated workflow.
244
+
In the packaging section, this was combined with ``spack cd`` to demonstrate a manual build process outside of Spack’s automated workflow.
245
245
This approach is especially useful in developer environments, providing a streamlined workflow for iterating on a single package without the overhead of the ``spack install`` command.
246
246
247
247
The extra features of the ``spack install`` command are often unnecessary when rapidly iterating between building and testing a specific package.
@@ -272,9 +272,8 @@ Working directly within the build environment, combined with Spack's navigation
272
272
Combinatorics
273
273
-------------
274
274
275
-
The final topic in this tutorial highlights the power of combinatorial development builds.
276
-
There are many cases where developers want to see how a single set of changes affects multiple build configurations—for example:
277
-
``+cuda`` vs ``~cuda``, ``%gcc`` vs ``%clang``, or ``build_type=Release`` vs ``build_type=Debug``.
275
+
The final topic in this tutorial highlights the power of combinatorial development builds.
276
+
There are many cases where developers want to see how a single set of changes affects multiple build configurations—for example: ``+cuda`` vs ``~cuda``, ``%gcc`` vs ``%clang``, or ``build_type=Release`` vs ``build_type=Debug``.
278
277
279
278
Developers can build all of these configurations with a single ``spack install`` call, as long as the develop spec is generic enough to cover the spec variations of the packages.
280
279
@@ -290,8 +289,7 @@ Developers can build all of these configurations with a single ``spack install``
290
289
291
290
While we won't build out this example, it illustrates how the ``dev_path`` for both ``build_type=Release`` and ``build_type=Debug`` points to the same source code.
292
291
293
-
If we want to do most of our incremental builds using the ``Release`` configuration and periodically check results using the ``Debug`` build, we can combine the workflow from the previous example:
294
-
First, enter the ``Release`` build environment using:
292
+
If we want to do most of our incremental builds using the ``Release`` configuration and periodically check results using the ``Debug`` build, we can combine the workflow from the previous example: First, enter the ``Release`` build environment using:
0 commit comments