Skip to content

Commit 9e87d73

Browse files
committed
ci: build sanitizer jobs on /mnt to fix ASan disk exhaustion
The ASan+UBSan RelWithDebInfo build tree is ~27 GB. GitHub-hosted runners ship either a 72 GB root disk (only ~20 GB free) or a 145 GB one, so sanitizer-asan passed or failed purely on which runner it landed: small runners cannot hold the build and the linker/assembler abort with 'No space left on device'. The job's "Free disk space" step was a no-op - it removed host paths (/usr/local/lib/android, /usr/share/dotnet, /opt/ghc) that do not exist inside the ubuntu:noble container. Bind-mount the runner's large /mnt temp disk (~74 GB free) into both sanitizer containers and point the colcon build base plus the compiler's scratch files (TMPDIR) there. install/ stays on the overlay - with --symlink-install it is only symlinks and small setup files. Apply the same redirect to sanitizer-tsan, which shares the risk.
1 parent f7e25ad commit 9e87d73

1 file changed

Lines changed: 38 additions & 16 deletions

File tree

.github/workflows/quality.yml

Lines changed: 38 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,12 @@ jobs:
217217
runs-on: ubuntu-latest
218218
container:
219219
image: ubuntu:noble
220+
# Bind-mount the runner's large temp disk. The ASan+UBSan build tree is
221+
# ~27 GB; GitHub-hosted runners ship either a 72 GB / (only ~20 GB free,
222+
# too small) or a 145 GB / disk, so the job used to pass or fail purely
223+
# on which runner it landed. /mnt always has ~60+ GB free.
224+
volumes:
225+
- "/mnt:/mnt"
220226
# Cold-cache ASan builds are ~33 min; with the 12-min test budget this
221227
# left ~0 headroom and was already cancelling at thermal_protection on
222228
# PR branches that miss the ccache restore key. 60 min covers a full
@@ -227,14 +233,6 @@ jobs:
227233
shell: bash
228234

229235
steps:
230-
- name: Free disk space
231-
run: |
232-
# Host paths are no-ops inside the container - safe to keep
233-
# for the case where the workflow is run without `container:`.
234-
rm -rf /usr/local/lib/android /usr/share/dotnet /opt/ghc || true
235-
apt-get clean
236-
df -h /
237-
238236
- name: Install Git
239237
run: |
240238
apt-get update
@@ -267,25 +265,35 @@ jobs:
267265
rosdep update
268266
rosdep install --from-paths src --ignore-src -y
269267
268+
- name: Redirect heavy build output to /mnt
269+
run: |
270+
# The ASan+UBSan RelWithDebInfo build tree is ~27 GB and does not fit
271+
# the container overlay on small (72 GB) GitHub runners. Point the
272+
# colcon build base and the compiler's scratch files at the large
273+
# /mnt temp disk (bind-mounted above). install/ stays on the overlay:
274+
# with --symlink-install it is only symlinks plus small setup files.
275+
mkdir -p /mnt/asan/build /mnt/asan/tmp
276+
ln -sfn /mnt/asan/build build
277+
df -h / /mnt
278+
270279
- name: Build with ASan + UBSan
271280
env:
272281
CCACHE_DIR: /root/.cache/ccache
273282
CCACHE_MAXSIZE: 500M
274283
CCACHE_SLOPPINESS: pch_defines,time_macros
284+
TMPDIR: /mnt/asan/tmp
275285
run: |
276286
source /opt/ros/jazzy/setup.bash
277-
# RelWithDebInfo (not Debug): Debug uses -O0 which produces
278-
# binaries ~2x the size of -O1, and the ASan/UBSan instrumentation
279-
# already inflates them several times over. On github-hosted
280-
# runners (~14 GB /) the linker hits 'No space left on device' on
281-
# PR branches that miss the ccache warm-up. The sanitizer cmake
282-
# module overrides optimisation back to -O1 either way, so this
283-
# only changes debug info size.
287+
# RelWithDebInfo (not Debug): Debug's -O0 roughly doubles binary size
288+
# vs -O1, and ASan/UBSan instrumentation inflates it further. The
289+
# sanitizer cmake module forces -O1 regardless, so the build type only
290+
# affects debug-info size. The build tree lands on /mnt (redirect step
291+
# above) so the ~27 GB of instrumented objects + DWARF fit.
284292
colcon build --symlink-install \
285293
--cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo -DSANITIZER=asan,ubsan \
286294
--event-handlers console_direct+
287295
ccache -s
288-
df -h /
296+
df -h / /mnt
289297
290298
- name: Extend test timeouts for ASan overhead
291299
run: |
@@ -323,6 +331,10 @@ jobs:
323331
runs-on: ubuntu-latest
324332
container:
325333
image: ubuntu:noble
334+
# See sanitizer-asan: the instrumented RelWithDebInfo build tree does not
335+
# fit the container overlay on small GitHub runners, so build on /mnt.
336+
volumes:
337+
- "/mnt:/mnt"
326338
timeout-minutes: 45
327339
defaults:
328340
run:
@@ -361,17 +373,27 @@ jobs:
361373
rosdep update
362374
rosdep install --from-paths src --ignore-src -y
363375
376+
- name: Redirect heavy build output to /mnt
377+
run: |
378+
# TSan RelWithDebInfo build tree is large too; keep it off the cramped
379+
# container overlay on small GitHub runners (see sanitizer-asan).
380+
mkdir -p /mnt/tsan/build /mnt/tsan/tmp
381+
ln -sfn /mnt/tsan/build build
382+
df -h / /mnt
383+
364384
- name: Build with TSan
365385
env:
366386
CCACHE_DIR: /root/.cache/ccache
367387
CCACHE_MAXSIZE: 500M
368388
CCACHE_SLOPPINESS: pch_defines,time_macros
389+
TMPDIR: /mnt/tsan/tmp
369390
run: |
370391
source /opt/ros/jazzy/setup.bash
371392
colcon build --symlink-install \
372393
--cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo -DSANITIZER=tsan \
373394
--event-handlers console_direct+
374395
ccache -s
396+
df -h / /mnt
375397
376398
- name: Extend test timeouts for TSan overhead
377399
run: |

0 commit comments

Comments
 (0)