Skip to content

Commit 060f752

Browse files
anandrdbzAnandsbryngelson
authored
Workaround elimination with latest amdflang drop and Removing array reshapes (#1433)
* merge with master * 94% of test suite passes * format fix * Fix mp_weno * Fix Surface tension * All cases except HLLD riemann solve pass (576/578) * Passing all test cases * Partial formatting * rename to resolve conflict * fix spacing * style: apply ffmt v0.4.0 formatting * fix: remove leftover dummy variable reference from rebase conflict resolution * Fix GPU cases, verified on frontier with cce-acc, cce-omp and amdflang * fix: disable PIE for post_process with LLVMFlang to fix R_X86_64_32 relocation error flang-23/LLD defaults to building PIE executables. SILO and LAPACK static libraries on Frontier are compiled without -fPIC, so their 32-bit absolute relocations (R_X86_64_32) are rejected by LLD when linking a PIE binary. Add -no-pie to post_process link options for LLVMFlang to allow non-PIC system libraries. simulation is unaffected (no SILO/LAPACK dependency). * ci: split frontier_amd case-opt into pre-build + run to avoid 2h wall limit AMD flang case-opt compilation takes close to the 2h hackathon wall limit, leaving no time for the run step. Split into two sequential hackathon GPU jobs: 1. Pre-Build: compiles all benchmarks via --dry-run (build only, no execution) 2. Run: skips build (binaries cached), runs and validates benchmarks Also preserve dependency dirs in prebuild for non-Phoenix clusters (deps are already built by the Fetch Dependencies step, so only clean staging dirs). --------- Co-authored-by: Anand <anand@Anands-MacBook-Pro.local> Co-authored-by: Spencer Bryngelson <sbryngelson@gmail.com>
1 parent e16c79c commit 060f752

26 files changed

Lines changed: 1414 additions & 1711 deletions

‎.github/scripts/prebuild-case-optimization.sh‎

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
#!/bin/bash
22

3-
# Pre-builds all benchmark cases with --case-optimization.
4-
# No GPU hardware needed — compilation only.
3+
# Pre-builds all benchmark cases with --case-optimization using --dry-run so
4+
# binaries are cached before the GPU run job. No simulation is executed.
55
# Can run in two modes:
66
# 1. Direct (Frontier login nodes): pass cluster/device/interface as args
7-
# 2. Inside SLURM (Phoenix): uses $job_device/$job_interface from submit-slurm-job.sh
7+
# 2. Inside SLURM (Phoenix/frontier_amd): uses $job_device/$job_interface
88
# Usage: bash prebuild-case-optimization.sh [<cluster> <device> <interface>]
99

1010
set -e
@@ -22,14 +22,18 @@ case "$cluster" in
2222
*) echo "ERROR: Unknown cluster '$cluster'"; exit 1 ;;
2323
esac
2424

25-
source .github/scripts/clean-build.sh
26-
clean_build
25+
# Phoenix starts fresh (no prior dep build); other clusters pre-build deps via
26+
# build.sh first, so we must preserve them and only clean MFC target staging.
27+
if [ "$cluster" = "phoenix" ]; then
28+
source .github/scripts/clean-build.sh
29+
clean_build
30+
else
31+
find build/staging -maxdepth 1 -regex '.*/[0-9a-f]+' -type d -exec rm -rf {} + 2>/dev/null || true
32+
find build/install -maxdepth 1 -regex '.*/[0-9a-f]+' -type d -exec rm -rf {} + 2>/dev/null || true
33+
fi
2734

2835
. ./mfc.sh load -c "$flag" -m g
2936

30-
# Set GPU build flags from interface — this is always a GPU build.
31-
# Don't use gpu-opts.sh since $job_device may be "cpu" when submitted
32-
# to a CPU SLURM partition (no GPU hardware needed for compilation).
3337
case "$job_interface" in
3438
acc) gpu_opts="--gpu acc" ;;
3539
omp) gpu_opts="--gpu mp" ;;
@@ -38,5 +42,5 @@ esac
3842

3943
for case in benchmarks/*/case.py; do
4044
echo "=== Pre-building: $case ==="
41-
./mfc.sh build -i "$case" --case-optimization $gpu_opts -j 8
45+
./mfc.sh run "$case" --case-optimization $gpu_opts -j 8 --dry-run
4246
done

‎.github/scripts/run_case_optimization.sh‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -23,13 +23,14 @@ benchmarks=(
2323

2424
# For Frontier/Frontier AMD: deps were fetched on the login node via --deps-only;
2525
# build case-optimized binaries here on the compute node before running.
26-
# For Phoenix: prebuild-case-optimization.sh already built everything in a prior SLURM job.
26+
# For Phoenix and frontier_amd: prebuild-case-optimization.sh already built
27+
# everything in a prior SLURM job (via --dry-run), so skip the build here.
2728
#
2829
# Clean stale MFC target staging before building. On self-hosted CI runners,
2930
# corrupted intermediate files from a prior failed build (e.g. CCE optcg crash)
3031
# can persist and poison subsequent builds. Each case-opt config gets its own
3132
# hash-named staging dir, but install dirs and other artifacts may be stale.
32-
if [ "$job_cluster" != "phoenix" ]; then
33+
if [ "$job_cluster" != "phoenix" ] && [ "$job_cluster" != "frontier_amd" ]; then
3334
# Clean stale MFC target dirs (hash-named) from prior builds, but
3435
# preserve dependency dirs (hipfort, fftw, etc.) since the compute
3536
# node has no internet to re-fetch them.

‎.github/workflows/test.yml‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -659,12 +659,16 @@ jobs:
659659
if: matrix.cluster == 'phoenix'
660660
run: bash .github/scripts/submit-slurm-job.sh .github/scripts/prebuild-case-optimization.sh cpu ${{ matrix.interface }} ${{ matrix.cluster }}
661661

662+
- name: Pre-Build (SLURM)
663+
if: matrix.cluster == 'frontier_amd'
664+
run: bash .github/scripts/submit-slurm-job.sh .github/scripts/prebuild-case-optimization.sh gpu ${{ matrix.interface }} ${{ matrix.cluster }}
665+
662666
- name: Build & Run Case-Optimization Tests
663-
if: matrix.cluster != 'phoenix'
667+
if: matrix.cluster != 'phoenix' && matrix.cluster != 'frontier_amd'
664668
run: bash .github/scripts/submit-slurm-job.sh .github/scripts/run_case_optimization.sh ${{ matrix.device }} ${{ matrix.interface }} ${{ matrix.cluster }}
665669

666670
- name: Run Case-Optimization Tests
667-
if: matrix.cluster == 'phoenix'
671+
if: matrix.cluster == 'phoenix' || matrix.cluster == 'frontier_amd'
668672
run: bash .github/scripts/submit-slurm-job.sh .github/scripts/run_case_optimization.sh ${{ matrix.device }} ${{ matrix.interface }} ${{ matrix.cluster }}
669673

670674
- name: Cancel SLURM Jobs

‎CMakeLists.txt‎

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -644,7 +644,7 @@ exit 0
644644
target_link_options(${a_target} PRIVATE -fopenmp)
645645
elseif(CMAKE_Fortran_COMPILER_ID STREQUAL "LLVMFlang")
646646
target_compile_options(${a_target} PRIVATE -fopenmp --offload-arch=gfx90a -fopenmp-target-fast -fopenmp-assume-threads-oversubscription -fopenmp-assume-teams-oversubscription)
647-
target_link_options(${a_target} PRIVATE -fopenmp --offload-arch=gfx90a)
647+
target_link_options(${a_target} PRIVATE -fopenmp --offload-arch=gfx90a -flto-partitions=${MFC_BUILD_JOBS})
648648
endif()
649649
endif()
650650

@@ -710,14 +710,15 @@ exit 0
710710
PRIVATE -DFRONTIER_UNIFIED)
711711
endif()
712712

713-
find_library(HIP_LIB amdhip64
713+
find_library(HIP_LIB amdhip64
714714
HINTS "$ENV{OLCF_AFAR_ROOT}/lib" REQUIRED)
715715
find_library(HIPFORT_AMDGCN_LIB hipfort-amdgcn
716716
HINTS "$ENV{OLCF_AFAR_ROOT}/lib" REQUIRED)
717717
target_include_directories(${a_target} PRIVATE
718718
"$ENV{OLCF_AFAR_ROOT}/include/hipfort/amdgcn")
719719
target_link_libraries(${a_target} PRIVATE
720-
${HIP_LIB} ${HIPFORT_AMDGCN_LIB} flang_rt.hostdevice)
720+
${HIP_LIB} ${HIPFORT_AMDGCN_LIB})
721+
721722
endif()
722723
elseif (CMAKE_Fortran_COMPILER_ID STREQUAL "Cray")
723724
target_compile_options(${a_target} PRIVATE "SHELL:-h noacc" "SHELL:-x acc")
@@ -790,6 +791,12 @@ if (MFC_POST_PROCESS)
790791

791792
# -O0 is in response to https://github.com/MFlowCode/MFC-develop/issues/95
792793
target_compile_options(post_process PRIVATE -O0)
794+
795+
# flang-23/LLD defaults to PIE; SILO and LAPACK static libs on Frontier are
796+
# non-PIC, producing R_X86_64_32 relocations that LLD rejects in PIE mode.
797+
if (CMAKE_Fortran_COMPILER_ID STREQUAL "LLVMFlang")
798+
target_link_options(post_process PRIVATE -no-pie)
799+
endif()
793800
endif()
794801

795802
if (MFC_SYSCHECK)

‎examples/3D_performance_test/case.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
json.dumps(
77
{
88
# Logistics
9-
"run_time_info": "T",
9+
"run_time_info": "F",
1010
# Computational Domain Parameters
1111
"x_domain%beg": 0.0e00,
1212
"x_domain%end": 4.0e-03 / 1.0e-03,

‎src/common/m_chemistry.fpp‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -193,7 +193,7 @@ contains
193193

194194
$:GPU_UPDATE(device='[isc1, isc2, isc3]')
195195

196-
if (chemistry .or. dummy) then
196+
if (chemistry) then
197197
! Set offsets based on direction using array indexing
198198
offsets = 0
199199
offsets(idir) = 1

‎src/post_process/m_global_parameters.fpp‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,6 @@ module m_global_parameters
186186
logical :: E_wrt
187187
logical, dimension(num_fluids_max) :: alpha_rho_e_wrt
188188
logical :: fft_wrt
189-
logical :: dummy !< AMDFlang workaround for case-optimization + GPU-kernel bug
190189
logical :: pres_wrt
191190
logical, dimension(num_fluids_max) :: alpha_wrt
192191
logical :: gamma_wrt
@@ -397,7 +396,6 @@ contains
397396
file_per_process = .false.
398397
E_wrt = .false.
399398
fft_wrt = .false.
400-
dummy = .false.
401399
pres_wrt = .false.
402400
alpha_wrt = .false.
403401
gamma_wrt = .false.

‎src/pre_process/m_global_parameters.fpp‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -186,7 +186,6 @@ module m_global_parameters
186186
real(wp) :: Bx0 !< Constant magnetic field in the x-direction (1D)
187187
integer :: buff_size !< Number of ghost cells for boundary condition storage
188188
logical :: fft_wrt
189-
logical :: dummy !< AMDFlang workaround for case-optimization + GPU-kernel bug
190189

191190
contains
192191

@@ -303,7 +302,6 @@ contains
303302
elliptic_smoothing = .false.
304303

305304
fft_wrt = .false.
306-
dummy = .false.
307305

308306
simplex_perturb = .false.
309307
simplex_params%perturb_vel(:) = .false.

‎src/simulation/m_acoustic_src.fpp‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -454,14 +454,16 @@ contains
454454
call s_mpi_abort('Fatal Error: Inconsistent allocation of source_spatials')
455455
end if
456456

457-
$:GPU_UPDATE(device='[source_spatials(ai)%coord]')
458-
$:GPU_UPDATE(device='[source_spatials(ai)%val]')
459-
if (support(ai) >= 5) then
460-
if (dim == 2) then
461-
$:GPU_UPDATE(device='[source_spatials(ai)%angle]')
462-
end if
463-
if (dim == 3) then
464-
$:GPU_UPDATE(device='[source_spatials(ai)%xyz_to_r_ratios]')
457+
if (count > 0) then
458+
$:GPU_UPDATE(device='[source_spatials(ai)%coord]')
459+
$:GPU_UPDATE(device='[source_spatials(ai)%val]')
460+
if (support(ai) >= 5) then
461+
if (dim == 2) then
462+
$:GPU_UPDATE(device='[source_spatials(ai)%angle]')
463+
end if
464+
if (dim == 3) then
465+
$:GPU_UPDATE(device='[source_spatials(ai)%xyz_to_r_ratios]')
466+
end if
465467
end if
466468
end if
467469
end do

‎src/simulation/m_cbc.fpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -529,7 +529,7 @@ contains
529529
#:for CBC_DIR, XYZ in [(1, 'x'), (2, 'y'), (3, 'z')]
530530
if (cbc_dir == ${CBC_DIR}$ .and. recon_type == WENO_TYPE) then
531531
! PI2 of flux_rs_vf and flux_src_rs_vf at j = 1/2
532-
if (weno_order == 3 .or. dummy) then
532+
if (weno_order == 3) then
533533
call s_convert_primitive_to_flux_variables(q_prim_rs${XYZ}$_vf, F_rs${XYZ}$_vf, F_src_rs${XYZ}$_vf, is1, is2, &
534534
& is3, idwbuff(2)%beg, idwbuff(3)%beg)
535535
@@ -557,7 +557,7 @@ contains
557557
end if
558558
559559
! PI4 of flux_rs_vf and flux_src_rs_vf at j = 1/2, 3/2
560-
if (weno_order == 5 .or. dummy) then
560+
if (weno_order == 5) then
561561
call s_convert_primitive_to_flux_variables(q_prim_rs${XYZ}$_vf, F_rs${XYZ}$_vf, F_src_rs${XYZ}$_vf, is1, is2, &
562562
& is3, idwbuff(2)%beg, idwbuff(3)%beg)
563563

0 commit comments

Comments
 (0)