Skip to content

Commit 65e2642

Browse files
sbryngelsonclaude
andcommitted
ci: redirect Phoenix TMPDIR in test.sh to avoid MPI session dir failures
Phoenix compute nodes have a small /tmp. With 8 parallel test threads each spawning MPI processes over ~96 minutes, it fills up and ORTE fails to create its session directory, causing the last batch of tests to fail with 'No such file or directory'. Apply the same TMPDIR redirect to project storage that bench.sh already uses. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent defdd7c commit 65e2642

1 file changed

Lines changed: 13 additions & 0 deletions

File tree

.github/workflows/common/test.sh

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,19 @@ set -euo pipefail
88
source .github/scripts/gpu-opts.sh
99
build_opts="$gpu_opts"
1010

11+
# --- Phoenix TMPDIR setup ---
12+
# Phoenix compute nodes have a small /tmp. With 8 parallel test threads each
13+
# spawning MPI processes, it fills up and ORTE session dir creation fails.
14+
# Redirect TMPDIR to project storage, same as bench.sh.
15+
if [ "$job_cluster" = "phoenix" ]; then
16+
tmpbuild=/storage/project/r-sbryngelson3-0/sbryngelson3/mytmp_build
17+
currentdir=$tmpbuild/run-$(( RANDOM % 9000 ))
18+
mkdir -p $tmpbuild
19+
mkdir -p $currentdir
20+
export TMPDIR=$currentdir
21+
trap 'rm -rf "$currentdir" || true' EXIT
22+
fi
23+
1124
# --- Build (if not pre-built on login node) ---
1225
# Phoenix builds inside SLURM; Frontier pre-builds via build.sh on the login node.
1326
# Phoenix builds inside SLURM on heterogeneous compute nodes — always start fresh

0 commit comments

Comments
 (0)