Skip to content

Commit 7f86746

Browse files
authored
Use dynamic CPU count for cmake --build -j in docs and test scripts (pytorch#20436)
### Summary Several build docs and `test/` scripts hardcode the `cmake --build -j` parallelism (`-j9`, `-j10`), which assumes a fixed machine. This replaces them with a portable expression that derives "core count + 1" at runtime — `nproc` on Linux, `sysctl -n hw.ncpu` on macOS: -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) "core count + 1" matches the guidance already documented in `docs/source/using-executorch-building-from-source.md`. The `nproc → sysctl` fallback keeps the commands working on both Linux and macOS, and the arithmetic degrades gracefully to `-j1` if neither tool is available. Partial fix for pytorch#10887. Scope is limited to general (non-vendor) docs and contributor-facing `test/` build scripts (9 files). Vendor-backend scripts (cadence, vulkan, coreml, qualcomm, mediatek, samsung, mps, nxp), CI scripts under `.ci/`, and non-cmake `-j` flags are intentionally left for follow-ups. ### Test plan - `lintrunner` passes on all changed files. - `bash -n` passes on the three modified shell scripts. - Verified the expression evaluates to a valid integer on Linux, e.g. `17` on a 16-core machine. cc @GregoryComer @digantdesai @cbilgin @JakeStevens @larryliu0820
1 parent 7628aa6 commit 7f86746

9 files changed

Lines changed: 16 additions & 15 deletions

File tree

CONTRIBUTING.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -325,7 +325,7 @@ To build C++ tests manually with CMake, run the following from the repository ro
325325

326326
```bash
327327
cmake . -Bcmake-out -DCMAKE_INSTALL_PREFIX=cmake-out -DEXECUTORCH_BUILD_TESTS=ON
328-
cmake --build cmake-out -j9 --target install
328+
cmake --build cmake-out -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target install
329329
```
330330

331331
You can then use `ctest` to list or run individual C++ tests directly:

backends/xnnpack/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,7 @@ cmake \
117117
Then you can build the runtime componenets with
118118

119119
```bash
120-
cmake --build cmake-out -j9 --target install --config Release
120+
cmake --build cmake-out -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target install --config Release
121121
```
122122

123123
Now you should be able to find the executable built at `./cmake-out/executor_runner` you can run the executable with the model you generated as such

docs/source/using-executorch-building-from-source.md

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -143,7 +143,7 @@ When building as a submodule as part of a user CMake build, ExecuTorch CMake opt
143143
CMake configuration for standalone runtime build:
144144
```bash
145145
cmake -B cmake-out --preset [preset] [options]
146-
cmake --build cmake-out -j10
146+
cmake --build cmake-out -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 ))
147147
```
148148
149149
#### Build Presets
@@ -265,8 +265,9 @@ cd executorch
265265
#
266266
# NOTE: The `-j` argument specifies how many jobs/processes to use when
267267
# building, and tends to speed up the build significantly. It's typical to use
268-
# "core count + 1" as the `-j` value.
269-
cmake --build cmake-out -j9
268+
# "core count + 1" as the `-j` value; the command below derives that
269+
# dynamically (`nproc` on Linux, `sysctl -n hw.ncpu` on macOS).
270+
cmake --build cmake-out -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 ))
270271
```
271272
272273
> **_TIP:_** For faster rebuilds, consider installing ccache (see [Compiler Cache section](#compiler-cache-ccache) below). On first builds, ccache populates its cache. Subsequent builds with the same compiler flags can be significantly faster.

examples/xnnpack/README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ cmake \
5151
Then you can build the runtime components with
5252

5353
```bash
54-
cmake --build cmake-out -j9 --target install --config Release
54+
cmake --build cmake-out -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target install --config Release
5555
```
5656

5757
Now finally you should be able to run this model with the following command
@@ -105,7 +105,7 @@ cmake \
105105
Then you can build the runtime componenets with
106106

107107
```bash
108-
cmake --build cmake-out -j9 --target install --config Release
108+
cmake --build cmake-out -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target install --config Release
109109
```
110110

111111
Now you should be able to find the executable built at `./cmake-out/executor_runner` you can run the executable with the model you generated as such

extension/training/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -247,7 +247,7 @@ cmake \
247247
Then you can build the runtime componenets with
248248

249249
```bash
250-
cmake --build cmake-out -j9 --target install --config Release
250+
cmake --build cmake-out -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target install --config Release
251251
```
252252

253253
Now you should be able to find the executable built at `./cmake-out/extension/training/train_xor` you can run the executable with the model you generated as such

kernels/README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -257,7 +257,7 @@ cmake -DCMAKE_INSTALL_PREFIX=cmake-out \
257257
-DCMAKE_BUILD_TYPE=Release \
258258
-DPYTHON_EXECUTABLE=python \
259259
-Bcmake-out .
260-
cmake --build cmake-out -j9 --target install --config Release
260+
cmake --build cmake-out -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target install --config Release
261261
```
262262
2. The generated `NativeFunctions.h` file is located in
263263
```
@@ -367,7 +367,7 @@ cmake . \
367367
-DEXECUTORCH_BUILD_TESTS=ON \
368368
-Bcmake-out
369369
370-
cmake --build cmake-out -j9 --target install
370+
cmake --build cmake-out -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target install
371371
```
372372
2. Run tests. You should see your test here.
373373
```

test/build_optimized_size_test.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -30,14 +30,14 @@ cmake_install_executorch_lib() {
3030
-DEXECUTORCH_OPTIMIZE_SIZE=ON \
3131
-DPYTHON_EXECUTABLE="$PYTHON_EXECUTABLE" \
3232
-Bcmake-out .
33-
cmake --build cmake-out -j9 --target install --config MinSizeRel
33+
cmake --build cmake-out -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target install --config MinSizeRel
3434
}
3535

3636
test_cmake_size_test() {
3737
CXXFLAGS="-g" retry cmake -DCMAKE_BUILD_TYPE=MinSizeRel -DEXECUTORCH_BUILD_KERNELS_OPTIMIZED=ON -DCMAKE_INSTALL_PREFIX=cmake-out -Bcmake-out/test test
3838

3939
echo "Build size test"
40-
cmake --build cmake-out/test -j9 --config MinSizeRel
40+
cmake --build cmake-out/test -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --config MinSizeRel
4141

4242
echo 'ExecuTorch with no ops binary size, unstripped:'
4343
ls -al cmake-out/test/size_test

test/build_size_test.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ cmake_install_executorch_lib() {
3333
-DPYTHON_EXECUTABLE="$PYTHON_EXECUTABLE" \
3434
${EXTRA_BUILD_ARGS} \
3535
-Bcmake-out .
36-
cmake --build cmake-out -j9 --target install --config Release
36+
cmake --build cmake-out -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target install --config Release
3737
}
3838

3939
test_cmake_size_test() {
@@ -43,7 +43,7 @@ test_cmake_size_test() {
4343
-Bcmake-out/test test
4444

4545
echo "Build size test"
46-
cmake --build cmake-out/test -j9 --config Release
46+
cmake --build cmake-out/test -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --config Release
4747

4848
echo 'ExecuTorch with no ops binary size, unstripped:'
4949
ls -al cmake-out/test/size_test

test/run_oss_cpp_tests.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ build_executorch() {
5959
-DEXECUTORCH_BUILD_XNNPACK=ON \
6060
-DEXECUTORCH_BUILD_TESTS=ON \
6161
-Bcmake-out
62-
cmake --build cmake-out -j9 --target install
62+
cmake --build cmake-out -j$(( $(nproc 2>/dev/null || sysctl -n hw.ncpu) + 1 )) --target install
6363
}
6464

6565
build_and_run_test() {

0 commit comments

Comments
 (0)