MPI p4est parabolic 2D nonconforming AMR with loadbalancing

[TJP-Karpowski]

MPI for 2D parabolic system on nonconforming P4est Mesh (AMR with loadbalancing tested)

The PR adds MPI support for the parabolic rhs for 2D P4est Meshes with AMR and load balancing.
The PR needs #2886 and #2881 to be merged first.

The elixir_navierstokes_lid_driven_cavity_amr_mortarTestcase.jl should test the MPI mortar treatment and include further refinement, unrefinement, and load balancing due to the rapid change in the refinement region.
If you would like to see further testcases let me know. Otherwise, I welcome your feedback and will continue with the 3D cases.

Disclaimer

LLMs have been used to aid in the PR.

Funding Statement

This work has been funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – Project Number 237267381 – TRR 150.

[github-actions]

Review checklist

This checklist is meant to assist creators of PRs (to let them know what reviewers will typically look for) and reviewers (to guide them in a structured review process). Items do not need to be checked explicitly for a PR to be eligible for merging.

Purpose and scope

• The PR has a single goal that is clear from the PR title and/or description.
• All code changes represent a single set of modifications that logically belong together.
• No more than 500 lines of code are changed or there is no obvious way to split the PR into multiple PRs.

Code quality

• The code can be understood easily.
• Newly introduced names for variables etc. are self-descriptive and consistent with existing naming conventions.
• There are no redundancies that can be removed by simple modularization/refactoring.
• There are no leftover debug statements or commented code sections.
• The code adheres to our conventions and style guide, and to the Julia guidelines.

Documentation

• New functions and types are documented with a docstring or top-level comment.
• Relevant publications are referenced in docstrings (see example for formatting).
• Inline comments are used to document longer or unusual code sections.
• Comments describe intent ("why?") and not just functionality ("what?").
• If the PR introduces a significant change or new feature, it is documented in NEWS.md with its PR number.

Testing

• The PR passes all tests.
• New or modified lines of code are covered by tests.
• New or modified tests run in less then 10 seconds.

Performance

• There are no type instabilities or memory allocations in performance-critical parts.
• If the PR intent is to improve performance, before/after time measurements are posted in the PR.

Verification

• The correctness of the code was verified using appropriate tests.
• If new equations/methods are added, a convergence test has been run and the results
  are posted in the PR.

Created with ❤️ by the Trixi.jl community.

[codecov]

Codecov Report

❌ Patch coverage is 97.74436% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.13%. Comparing base (c5f0b51) to head (2c04cf8).

Files with missing lines	Patch %	Lines
...rc/solvers/dgsem_p4est/dg_2d_parabolic_parallel.jl	97.54%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2888      +/-   ##
==========================================
+ Coverage   97.12%   97.13%   +0.01%     
==========================================
  Files         622      623       +1     
  Lines       48253    48385     +132     
==========================================
+ Hits        46861    46996     +135     
+ Misses       1392     1389       -3     

Flag	Coverage Δ
unittests	97.13% <97.74%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sloede]

I of course haven't looked at the parallel solver implementation and the tests yet, waiting for #2886 to be merged to main first.

[sloede]

Except for the missing test coverage (which I think actually would be good to fix, since it covers an important and ugly-to-debug case), this looks ready to merge 👍

[TJP-Karpowski]

I added the shown testcase for the mesh_wobbly_channel.inp . It is a gaussian peak advected through the channel which is subject to diffusion. In the gif the colors indicate MPI ranks, the displacment is the scalar value. As seen in the GIF the MPI boundaries feature some mortars (unless I limited the testcase time to much (1/2 of the shown gif) .

[sloede]

Very nice - thanks! Let's hope this fixes the problem.

Can you please check how long the two additional tests run on your machine (after warmup)? We have a general "10 second rule" and while this can be stretched with good reason, it shouldn't be too long since it will be much longer even when run on the GitHub Action runners...

[TJP-Karpowski]

Thanks for the reminder. The new elixier runs in under 10s. The test only has half the simtime so should stay well within the limits. The other testcase I don't remember exactly, but I did check that it was below 10s at some point. But maybe I could also remove the lid-driven-cavity test again, since the new test should cover all aspects too.

[sloede]

Does anyone understand what's happening with the MPI test timeouts? When I checked last, there seemed to be also an actual test failing (the new one), but GitHub is not working reliably for me at the moment and I can't verify if this is still the case.

[TJP-Karpowski]

I reduced the tolerance for the new testcase, which might be too tight for MPI-AMR cases. So hopefully the testcase should no longer fail. I don't think, however, that this failure caused the timeout issue.

[sloede]

I reduced the tolerance for the new testcase, which might be too tight for MPI-AMR cases. So hopefully the testcase should no longer fail. I don't think, however, that this failure caused the timeout issue.

If AMR is finicky, we could also consider using a static, non-conforming setup. However, based in your previous tests - where you got identical results for serial and parallel simulations - the parallelization should have the same results, shouldn't it?

[TJP-Karpowski]

For the static grids, yes. But also in the other AMR testcase, I got the same value up to, let's say, tol=1e-7 or 1e-8, not 1e-13. On my machine, the test also passed before the tolerances were adjusted. Given the potentially computer-specific decomposition and thus order of summation in the MPI reduction step, I would accept 1e-8 as sufficient. The test has AMR, so we also test parabolic cache resizing at the same time, thus testing all changes with only 1 new testcase.

[sloede]

LGTM! The downstream failure is an HTTP 502 ("bad gateway") error and unrelated to the changes in this PR.

Thanks again for the nice work@