Skip to content

Commit 6f518df

Browse files
committed
feat(amr): Lagrangian bubbles with the cloud excluded from fine blocks - regrid suppresses tags and clips candidate boxes around the cloud's padded bbox (positions + mapCells smearing + stencil + drift margin, recomputed collectively per regrid), the fine advance skips the EL hooks (a bubble would map to wrong indices on the swapped grid), and a per-stage guard aborts if the cloud reaches an active block. FIX: EL volume fractions sum to the local liquid fraction (beta), not 1 - the multi-fluid sum-to-one prolongation closure corrupted the EL state (found by the free-stream battery: O(1) momentum injection at block edges in quiescent flow; exactly 0 after the exemption); EL is correspondingly exempt from the mpp_lim gate. Validated: free-stream through block exactly 0, guard-trip named abort, 800-step wave + two-way bubble at 1e-9 field consistency with the bubble trajectory identical; goldens 4B08E9B7 (static) BCE1BBAE (dynamic regrid); multi-fluid AMR battery still green (closure fires for non-EL)
1 parent c3eac4c commit 6f518df

12 files changed

Lines changed: 619 additions & 22 deletions

File tree

docs/documentation/amr.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -215,7 +215,7 @@ a diagnostic message for unsupported combinations.
215215
| Hyperelasticity / MHD | **Not supported** | Gated (hyperelasticity has no upstream test coverage to validate against; MHD needs divergence-preserving B prolongation) |
216216
| QBMM bubbles (polytropic) | Supported | Bubble moments live in `q_cons`, injected piecewise-constant at prolongation to preserve CHyQMOM realizability |
217217
| QBMM bubbles (non-polytropic) | Supported (static block, no subcycle) | Each block carries its own `pb`/`mv` quadrature side-state: prolonged piecewise-constant (realizability), advanced with the block's own rhs scratch, restricted back with the moments; dynamic regrid and `amr_subcycle` remain gated |
218-
| Lagrangian bubbles | **Not supported** | Lagrangian tracking not advanced on the fine level |
218+
| Lagrangian bubbles (`bubbles_lagrange = T`) | Supported (cloud excluded from blocks) | Two-way coupling lives on the coarse grid: regrid suppresses tags and clips candidate boxes around the cloud's padded bbox (positions + `mapCells` smearing + stencil + drift margin, recomputed collectively each regrid), the fine advance skips the EL hooks, EL volume fractions prolong WITHOUT the sum-to-one closure (their sum is the local liquid fraction), and a per-stage guard aborts if the cloud reaches an active block |
219219
| Immersed boundaries (`ib = T`; one or more non-STL bodies, static or prescribed-motion `moving_ibm=1`) | Supported | Per-block fine-grid IB markers/ghost points, rebuilt each fine substage at the body's sub-time position for a moving body; non-conservative ghost-cell forcing at the body; with dynamic regrid candidate boxes expand to fully contain each body at its live position plus margin, the fine IB state rebuilds after every regrid, and a per-substage guard aborts if a moving body reaches its block boundary between regrids; force-driven (`moving_ibm=2`)/STL gated; a body spanning a rank seam is rejected at startup |
220220
| IGR solver | **Not supported** | Unvalidated with the fine-level advance |
221221
| 2D axisymmetric (`cyl_coord = T`, `p = 0`) | Supported | Geometric sources read the live (swapped) grid; the axis half-width cell's per-cell WENO coefficients are recomputed for each block on swap/restore; blocks stay `buff_size` off the axis (domain-edge clamp); the axis-singularity viscous treatment runs on the coarse pass only |

docs/documentation/case.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -850,7 +850,12 @@ boundary between regrids (reduce `amr_regrid_int` or increase `amr_buf`). Force-
850850
subdomain is bit-exact across decompositions; a body spanning a rank seam is rejected at startup
851851
(the fine-IB image-point stencil across the seam is not yet decomposition-exact), so keep the body
852852
inside a single rank's subdomain (use fewer ranks or reposition it).
853-
AMR is incompatible with surface tension, Lagrangian bubbles, IGR, 3D cylindrical
853+
Lagrangian bubbles are supported with the cloud excluded from fine blocks: two-way coupling
854+
lives on the coarse grid, regrid suppresses tags and clips boxes around the cloud's padded
855+
bounding box, EL volume fractions prolong without the sum-to-one closure (their sum is the
856+
local liquid fraction, not 1), and a per-stage guard aborts if the cloud reaches an active
857+
block (reduce `amr_regrid_int` or increase `amr_buf`).
858+
AMR is incompatible with surface tension, IGR, 3D cylindrical
854859
coordinates (2D axisymmetric IS supported), MHD, hyperelasticity, Riemann-extrapolation
855860
boundaries (bc = -4), and `active_box`. `hybrid_weno`/`hybrid_riemann` are supported: each
856861
level recomputes the smoothness sensor over its own (swapped) bounds every RHS call.

src/simulation/m_amr.fpp

Lines changed: 139 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,11 @@ module m_amr
1515

1616
use m_derived_types ! scalar_field, t_box, int_bounds_info
1717
use m_global_parameters
18-
use m_constants, only: num_fluids_max, model_eqns_6eq
18+
use m_constants, only: num_fluids_max, model_eqns_6eq, mapCells
1919
use m_pressure_relaxation, only: s_pressure_relaxation_procedure
2020
use m_mpi_proxy, only: s_mpi_abort, s_initialize_amr_mpi_buffers, s_mpi_sendrecv_amr_fine_halo
21-
use m_mpi_common, only: s_mpi_allreduce_integer_min, s_mpi_allreduce_integer_max, s_mpi_allreduce_sum, &
22-
& s_mpi_allreduce_integer_sum, s_mpi_sendrecv_variables_buffers
21+
use m_mpi_common, only: s_mpi_allreduce_integer_min, s_mpi_allreduce_integer_max, s_mpi_allreduce_sum, s_mpi_allreduce_min, &
22+
& s_mpi_allreduce_max, s_mpi_allreduce_integer_sum, s_mpi_sendrecv_variables_buffers
2323
use m_rhs, only: s_compute_rhs
2424
use m_phase_change, only: s_infinite_relaxation_k
2525
use m_amr_registers, only: s_amr_zero_fine_registers
@@ -29,6 +29,7 @@ module m_amr
2929
use m_hypoelastic, only: s_hypoelastic_update_fd_coeffs
3030
use m_weno, only: s_compute_weno_coefficients
3131
use m_acoustic_src, only: acoustic_supp_lo, acoustic_supp_hi
32+
use m_bubbles_EL, only: s_lag_cloud_bbox_local
3233

3334
implicit none
3435

@@ -91,7 +92,14 @@ module m_amr
9192
integer :: sw_m, sw_n, sw_p
9293
type(int_bounds_info) :: sw_idwint(3), sw_idwbuff(3)
9394
logical :: sw_acoustic_source
94-
logical :: amr_swapped = .false. !< paired-swap guard: a nested swap would corrupt the bounce buffers silently
95+
96+
!> Lagrangian bubble-cloud exclusion support: padded global coarse-index bbox of the cloud (positions + mapCells smearing +
97+
!! stencil headroom [+ drift margin at regrid]). Blocks and regrid boxes stay clear of it: a bubble inside a block loses two-way
98+
!! coupling (the fine advance skips the EL hooks and the coarse result under the block is discarded by restriction). Recomputed
99+
!! collectively at each regrid; guarded rank-locally per stage.
100+
integer :: lag_supp_lo(3), lag_supp_hi(3)
101+
logical :: lag_supp_on = .false.
102+
logical :: amr_swapped = .false. !< paired-swap guard: a nested swap would corrupt the bounce buffers silently
95103
!> True when the coarse grid is nonuniform in an allowed way (the 2D-axisymmetric half-width axis cell): the WENO reconstruction
96104
!! coefficients are then per-cell, so the fine advance must recompute them for the block's own (uniform) grid on every swap and
97105
!! restore the coarse ones after. False on fully uniform grids - the recompute is skipped and behavior is bit-identical.
@@ -488,7 +496,9 @@ contains
488496
bstride = 1
489497
if (bubbles_euler) bstride = (eqn_idx%bub%end - eqn_idx%bub%beg + 1)/nb
490498
do i = 1, sys_size
491-
if (num_fluids > 1 .and. i >= eqn_idx%adv%beg .and. i <= eqn_idx%adv%end) cycle
499+
! Lagrangian bubbles: alphas sum to the LOCAL liquid fraction beta (not 1), so the
500+
! sum-to-one closure would corrupt the EL state; each alpha prolongs plainly instead
501+
if (num_fluids > 1 .and. (.not. bubbles_lagrange) .and. i >= eqn_idx%adv%beg .and. i <= eqn_idx%adv%end) cycle
492502
if (chemistry .and. i >= eqn_idx%species%beg .and. i <= eqn_idx%species%end) cycle ! sum/positivity closure below
493503
! QBMM carries a bivariate 6-moment set per R0 bin whose CHyQMOM inversion requires realizability
494504
! (variance c20 = m20/m00 - (m10/m00)^2 > 0); per-component minmod prolongation can break that joint
@@ -501,7 +511,7 @@ contains
501511
& .and. mod(i - eqn_idx%bub%beg, bstride) /= 1, &
502512
& inject=qbmm .and. i >= eqn_idx%bub%beg .and. i <= eqn_idx%bub%end)
503513
end do
504-
if (num_fluids > 1) call s_prolong_alphas_closure(q_cons_base, amr_slots(amr_cur)%q_cons)
514+
if (num_fluids > 1 .and. (.not. bubbles_lagrange)) call s_prolong_alphas_closure(q_cons_base, amr_slots(amr_cur)%q_cons)
505515
if (chemistry) call s_prolong_species_closure(q_cons_base, amr_slots(amr_cur)%q_cons)
506516

507517
end subroutine s_interpolate_coarse_to_fine
@@ -1319,7 +1329,7 @@ contains
13191329
b1 = amr_slots(amr_cur)%idwbuff(1)%beg; e1 = amr_slots(amr_cur)%idwbuff(1)%end
13201330
b2 = amr_slots(amr_cur)%idwbuff(2)%beg; e2 = amr_slots(amr_cur)%idwbuff(2)%end
13211331
b3 = amr_slots(amr_cur)%idwbuff(3)%beg; e3 = amr_slots(amr_cur)%idwbuff(3)%end
1322-
multi = num_fluids > 1
1332+
multi = num_fluids > 1 .and. (.not. bubbles_lagrange) ! EL alphas sum to beta, not 1: no sum-to-one closure
13231333
advb = eqn_idx%adv%beg; adve = eqn_idx%adv%end
13241334
bubEE = bubbles_euler; bbeg = eqn_idx%bub%beg; bend = eqn_idx%bub%end
13251335
bstride = 1; if (bubEE) bstride = (bend - bbeg + 1)/nb
@@ -1586,6 +1596,7 @@ contains
15861596
! valid coarse CONS ghosts for the ghost prolongation below (ALL ranks call: pairwise halo)
15871597
if (amr_xchg_coarse_ghosts) call s_amr_exchange_coarse_cons_halo(q_cons_coarse)
15881598
if (.not. amr_rank_owns_block) return
1599+
call s_amr_check_lag_clear()
15891600

15901601
! ghost prolongation from the coarse stage-entry conservative state (device kernel reads the
15911602
! device-current coarse stage-entry state directly); rank_time brackets cover the fine-advance
@@ -1672,6 +1683,7 @@ contains
16721683
call s_amr_exchange_coarse_cons_halo(q_old)
16731684
call s_amr_exchange_coarse_cons_halo(q_new)
16741685
end if
1686+
if (amr_rank_owns_block) call s_amr_check_lag_clear()
16751687
if (.not. amr_rank_owns_block) return
16761688

16771689
! fill both lerp sources once: ghost shells prolonged from coarse t^n and t^{n+1} (device kernels
@@ -1903,6 +1915,113 @@ contains
19031915
19041916
end subroutine s_amr_clip_box_from_sources
19051917
1918+
!> Convert a physical-space bbox to a global coarse-index bbox padded by pad_cells.
1919+
pure subroutine s_lag_phys_to_cells(pmin, pmax, pad_cells, blo, bhi)
1920+
1921+
real(wp), dimension(3), intent(in) :: pmin, pmax
1922+
integer, intent(in) :: pad_cells
1923+
integer, intent(out) :: blo(3), bhi(3)
1924+
1925+
blo(1) = int((pmin(1) - x_domain%beg)/dx(0)) - pad_cells
1926+
bhi(1) = int((pmax(1) - x_domain%beg)/dx(0)) + pad_cells
1927+
blo(2) = 0; bhi(2) = 0; blo(3) = 0; bhi(3) = 0
1928+
if (n_glb > 0) then
1929+
blo(2) = int((pmin(2) - y_domain%beg)/dy(min(1, n))) - pad_cells
1930+
bhi(2) = int((pmax(2) - y_domain%beg)/dy(min(1, n))) + pad_cells
1931+
end if
1932+
if (p_glb > 0) then
1933+
blo(3) = int((pmin(3) - z_domain%beg)/dz(0)) - pad_cells
1934+
bhi(3) = int((pmax(3) - z_domain%beg)/dz(0)) + pad_cells
1935+
end if
1936+
1937+
end subroutine s_lag_phys_to_cells
1938+
1939+
!> Recompute the global Lagrangian-cloud exclusion bbox (collective: allreduces the rank-local position extrema). pad_cells
1940+
!! covers smearing + stencil (+ drift until the next recompute). No-op (lag_supp_on = false) when no rank holds a bubble.
1941+
impure subroutine s_amr_compute_lag_supp(pad_cells)
1942+
1943+
integer, intent(in) :: pad_cells
1944+
real(wp), dimension(3) :: pmin_loc, pmax_loc, pmin_glb, pmax_glb
1945+
integer :: d
1946+
1947+
call s_lag_cloud_bbox_local(pmin_loc, pmax_loc)
1948+
do d = 1, 3
1949+
call s_mpi_allreduce_min(pmin_loc(d), pmin_glb(d))
1950+
call s_mpi_allreduce_max(pmax_loc(d), pmax_glb(d))
1951+
end do
1952+
lag_supp_on = pmin_glb(1) <= pmax_glb(1)
1953+
if (.not. lag_supp_on) return
1954+
call s_lag_phys_to_cells(pmin_glb, pmax_glb, pad_cells, lag_supp_lo, lag_supp_hi)
1955+
1956+
end subroutine s_amr_compute_lag_supp
1957+
1958+
!> True iff global level-0 cell (gi, gj, gk) lies inside the Lagrangian-cloud exclusion bbox.
1959+
pure logical function f_in_lag_support(gi, gj, gk) result(insup)
1960+
1961+
integer, intent(in) :: gi, gj, gk
1962+
1963+
insup = .false.
1964+
if (.not. lag_supp_on) return
1965+
insup = gi >= lag_supp_lo(1) .and. gi <= lag_supp_hi(1) .and. (n_glb == 0 .or. (gj >= lag_supp_lo(2) &
1966+
& .and. gj <= lag_supp_hi(2))) .and. (p_glb == 0 .or. (gk >= lag_supp_lo(3) &
1967+
& .and. gk <= lag_supp_hi(3)))
1968+
1969+
end function f_in_lag_support
1970+
1971+
!> Clip a candidate regrid box (global indices) clear of one support bbox: remove the overlap along the single axis/side that
1972+
!! keeps the largest remaining extent (deterministic: lower axis, then begin side, wins ties). Only shrinks; may empty the box
1973+
!! (hi < lo).
1974+
pure subroutine s_amr_clip_box_from_supp(lo, hi, slo, shi)
1975+
1976+
integer, intent(inout) :: lo(3), hi(3)
1977+
integer, intent(in) :: slo(3), shi(3)
1978+
integer :: d, best_d, best_side, best_ext, ext_l, ext_r
1979+
logical :: ovl
1980+
1981+
if (hi(1) < lo(1) .or. hi(2) < lo(2) .or. hi(3) < lo(3)) return
1982+
ovl = lo(1) <= shi(1) .and. hi(1) >= slo(1)
1983+
if (n_glb > 0) ovl = ovl .and. lo(2) <= shi(2) .and. hi(2) >= slo(2)
1984+
if (p_glb > 0) ovl = ovl .and. lo(3) <= shi(3) .and. hi(3) >= slo(3)
1985+
if (.not. ovl) return
1986+
best_d = 1; best_side = 1; best_ext = -1
1987+
do d = 1, num_dims
1988+
ext_l = slo(d) - lo(d)
1989+
ext_r = hi(d) - shi(d)
1990+
if (ext_l > best_ext) then; best_ext = ext_l; best_d = d; best_side = 1; end if
1991+
if (ext_r > best_ext) then; best_ext = ext_r; best_d = d; best_side = 2; end if
1992+
end do
1993+
if (best_side == 1) then
1994+
hi(best_d) = slo(best_d) - 1
1995+
else
1996+
lo(best_d) = shi(best_d) + 1
1997+
end if
1998+
1999+
end subroutine s_amr_clip_box_from_supp
2000+
2001+
!> Rank-local per-stage guard: the local bubbles' padded bbox must stay clear of the current block. Catches an overlapping
2002+
!! initial placement on the first stage and drift that outran the regrid margin afterwards.
2003+
impure subroutine s_amr_check_lag_clear()
2004+
2005+
real(wp), dimension(3) :: pmin_loc, pmax_loc
2006+
integer :: blo(3), bhi(3)
2007+
logical :: ovl
2008+
2009+
if (.not. bubbles_lagrange) return
2010+
call s_lag_cloud_bbox_local(pmin_loc, pmax_loc)
2011+
if (pmin_loc(1) > pmax_loc(1)) return ! no bubbles on this rank
2012+
call s_lag_phys_to_cells(pmin_loc, pmax_loc, mapCells + 2, blo, bhi)
2013+
ovl = blo(1) <= amr_slots(amr_cur)%region%hi(1) .and. bhi(1) >= amr_slots(amr_cur)%region%lo(1)
2014+
if (n_glb > 0) ovl = ovl .and. blo(2) <= amr_slots(amr_cur)%region%hi(2) .and. bhi(2) >= amr_slots(amr_cur)%region%lo(2)
2015+
if (p_glb > 0) ovl = ovl .and. blo(3) <= amr_slots(amr_cur)%region%hi(3) .and. bhi(3) >= amr_slots(amr_cur)%region%lo(3)
2016+
if (ovl) then
2017+
call s_mpi_abort('amr with Lagrangian bubbles: the bubble cloud (positions + smearing support) ' &
2018+
& // 'overlaps an active fine block, where two-way coupling would be lost. Keep the initial ' &
2019+
& // 'block clear of the cloud; under dynamic regrid, reduce amr_regrid_int or increase ' &
2020+
& // 'amr_buf so the exclusion margin covers the cloud drift between regrids')
2021+
end if
2022+
2023+
end subroutine s_amr_check_lag_clear
2024+
19062025
!> Expand a candidate regrid box (global indices) to fully contain every immersed body it overlaps, with a buff_size margin (the
19072026
!! IB image-point stencils need resolved surroundings). Expansion is re-clamped to the domain interior by the caller's own
19082027
!! guards; a body too large for the per-rank block cap aborts with a named message. The bbox reads the live centroid, so a
@@ -2118,6 +2237,10 @@ contains
21182237
$:GPU_UPDATE(host='[q_cons_base(i)%sf]')
21192238
end do
21202239
2240+
! Lagrangian-cloud exclusion bbox for this regrid (collective): smearing (mapCells) +
2241+
! stencil headroom (2) + drift margin until the next regrid (amr_buf)
2242+
if (bubbles_lagrange) call s_amr_compute_lag_supp(mapCells + 2 + amr_buf)
2243+
21212244
! 1) per-cell tag field (density-gradient criterion, unchanged), skipping the two global boundary cells per active dim
21222245
sidx = 0
21232246
sidx(1) = start_idx(1)
@@ -2145,6 +2268,11 @@ contains
21452268
if (acoustic_source .and. tag_grid(ci, cj, ck)) then
21462269
if (f_in_acoustic_support(ci + sidx(1), cj + sidx(2), ck + sidx(3))) tag_grid(ci, cj, ck) = .false.
21472270
end if
2271+
! the Lagrangian bubble cloud stays coarse (two-way coupling lives on the
2272+
! coarse grid): suppress tags over its padded bbox
2273+
if (bubbles_lagrange .and. tag_grid(ci, cj, ck)) then
2274+
if (f_in_lag_support(ci + sidx(1), cj + sidx(2), ck + sidx(3))) tag_grid(ci, cj, ck) = .false.
2275+
end if
21482276
end do
21492277
end do
21502278
end do
@@ -2176,6 +2304,7 @@ contains
21762304
! keep candidate boxes clear of every acoustic source support (the source acts on the
21772305
! coarse grid only); clipping only shrinks, so boxes stay disjoint - empties drop below
21782306
if (acoustic_source) call s_amr_clip_box_from_sources(lo, hi)
2307+
if (bubbles_lagrange .and. lag_supp_on) call s_amr_clip_box_from_supp(lo, hi, lag_supp_lo, lag_supp_hi)
21792308
! a fine block that PARTIALLY covers an immersed body is an untested regime (ghost
21802309
! prolongation through body-interior cells, refluxing across the body): any box that
21812310
! overlaps a body's bounding box is expanded to contain the whole body plus margin
@@ -2220,10 +2349,11 @@ contains
22202349
do k = 1, nboxes
22212350
lo = boxes(k)%lo; hi = boxes(k)%hi
22222351
call s_amr_clip_box_from_sources(lo, hi)
2352+
if (bubbles_lagrange .and. lag_supp_on) call s_amr_clip_box_from_supp(lo, hi, lag_supp_lo, lag_supp_hi)
22232353
if (any(lo /= boxes(k)%lo) .or. any(hi /= boxes(k)%hi)) then
22242354
call s_mpi_abort('amr regrid: a block must contain an immersed body AND stay ' &
2225-
& // 'clear of an acoustic source support - the constraints conflict; ' &
2226-
& // 'move the body or the source apart')
2355+
& // 'clear of an acoustic source support / Lagrangian bubble cloud - the ' &
2356+
& // 'constraints conflict; move the body, source, or cloud apart')
22272357
end if
22282358
end do
22292359
end if

src/simulation/m_bubbles_EL.fpp

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1514,6 +1514,28 @@ contains
15141514

15151515
end subroutine s_write_restart_lag_bubbles
15161516

1517+
!> Physical-space bounding box of this rank's Lagrangian bubbles (committed step positions, register 1). Rank-local: the caller
1518+
!! allreduces if it needs the global cloud. Returns pmin > pmax when the rank holds no bubbles.
1519+
impure subroutine s_lag_cloud_bbox_local(pmin, pmax)
1520+
1521+
real(wp), dimension(3), intent(out) :: pmin, pmax
1522+
real(wp) :: x1n, x2n, x3n, x1x, x2x, x3x
1523+
integer :: k
1524+
1525+
x1n = huge(1._wp); x2n = huge(1._wp); x3n = huge(1._wp)
1526+
x1x = -huge(1._wp); x2x = -huge(1._wp); x3x = -huge(1._wp)
1527+
$:GPU_PARALLEL_LOOP(private='[k]', reduction='[[x1n, x2n, x3n], [x1x, x2x, x3x]]', reductionOp='[MIN, MAX]', copy='[x1n, &
1528+
& x2n, x3n, x1x, x2x, x3x]')
1529+
do k = 1, nBubs
1530+
x1n = min(x1n, mtn_pos(k, 1, 1)); x1x = max(x1x, mtn_pos(k, 1, 1))
1531+
x2n = min(x2n, mtn_pos(k, 2, 1)); x2x = max(x2x, mtn_pos(k, 2, 1))
1532+
x3n = min(x3n, mtn_pos(k, 3, 1)); x3x = max(x3x, mtn_pos(k, 3, 1))
1533+
end do
1534+
$:END_GPU_PARALLEL_LOOP()
1535+
pmin = [x1n, x2n, x3n]; pmax = [x1x, x2x, x3x]
1536+
1537+
end subroutine s_lag_cloud_bbox_local
1538+
15171539
!> Compute the maximum and minimum radius of each bubble
15181540
subroutine s_calculate_lag_bubble_stats()
15191541

0 commit comments

Comments
 (0)