`hse_fill` ignores HSE Newton non-convergence on GPU for implemented boundaries

[WeiqunZhang]

Summary

In all implemented HSE boundary integrations (-X, +X, -Y, +Y, -Z), the !converged_hse failure path is compiled only for non-GPU builds. On GPU, the code silently proceeds after exhausting iterations, using unconverged density/pressure states.

Location

• Source/problems/hse_fill.cpp:154
• Source/problems/hse_fill.cpp:345
• Source/problems/hse_fill.cpp:543
• Source/problems/hse_fill.cpp:735
• Source/problems/hse_fill.cpp:932

Problem Details

Each boundary block follows this pattern:

[[maybe_unused]] bool converged_hse = false;
for (int iter = 0; iter < hse::MAX_ITER; iter++) { ... }

#ifndef AMREX_USE_GPU
if (!converged_hse) {
    amrex::Error("... failure to converge ...");
}
#endif

So in GPU builds, convergence failure is neither reported nor handled, and fill continues with the last Newton iterate.

Impact

• Silent invalid ghost-state fills when Newton solve fails.
• Hard-to-diagnose boundary-driven instabilities on GPU runs.
• CPU/GPU behavior divergence for identical inputs.

Suggested Patch

Use a device-visible failure flag and check it on host after each kernel, instead of CPU-only error paths.

Example pattern:

diff --git a/Source/problems/hse_fill.cpp b/Source/problems/hse_fill.cpp
--- a/Source/problems/hse_fill.cpp
+++ b/Source/problems/hse_fill.cpp
@@
-            amrex::ParallelFor(gbx,
+            int hse_fail = 0;
+            amrex::ParallelFor(gbx,
             [=] AMREX_GPU_DEVICE (int i, int j, int k) noexcept
             {   
                 ...
                 if (!converged_hse) {
-#ifndef AMREX_USE_GPU
-                    amrex::Error("ERROR in bc_ext_fill_nd: failure to converge in -X BC");
-#endif
+                    amrex::Gpu::Atomic::Exch(&hse_fail, 1);
                 }
                 ...
             });
+            amrex::Gpu::streamSynchronize();
+            if (hse_fail != 0) {
+                amrex::Error("ERROR in bc_ext_fill_nd: failure to converge in -X BC");
+            }

Apply the same pattern to the other implemented HSE faces.

Prepared by Codex