GPU porting

[benegee]

Rough guideline on making parts of Trixi.jl GPU-compatible:
https://github.com/trixi-framework/Trixi.jl/blob/main/docs/src/heterogeneous.md#writing-gpu-kernels

Discussion

• It would be much nicer to hide away the implementation and launch of kernels, see Implement common parallel functionality for trixi-framework TrixiBase.jl#66
• Pass typeof(mesh) to kernels, instead of just mesh ✅
• Pass typeof(dg) to kernels, instead of just dg?
• How to deal with cache?

Step 1: Minimal feature set

• Basic solution for the "no cache on GPU" problem
• Basic P4est functionality (2D & 3D): Initial GPU support for P4estMesh 2D and 3D using simple KernelAbstractions kernels #2590

Step 2a: "Full" implementation without host-device memory transfers

• Tutorial on how to use GPU backend (with different precisions)
• Output information on GPU usage in SummaryCallback Add trixi_backend_info! to show which GPU backend is being used #2906
• StepsizeCallback, in particular reduction of speeds GPU-compatible reduction of speeds in step size computation #2823
• AnalysisCallback Use AcceleratedKernels.mapreduce in max_scaled_speed and integrate_via_indices #2882
• SaveSolutionCallback
• SaveRestartCallback

Step 2b: Initial performance analysis & optimization

• More fine-grained parallelization for flux differencing kernel (Initial GPU support for P4estMesh 2D and 3D using simple KernelAbstractions kernels #2590 (comment))
• 2D calc_volume_integral! optimized kernel on quadrature nodes for conservative systems
• 3D calc_volume_integral! optimized kernel on quadrature nodes for conservative systems(Optimized 3D flux differencing kernel for GPU #3015)
• 2D calc_volume_integral! optimized kernel on quadrature nodes for nonconservative systems
• 3D calc_volume_integral! optimized kernel on quadrature nodes for nonconservative systems
• apply_jacobian! 2D kernel on quadrature nodes (Optimized 2D GPU Kernel for apply_jacobian! #3013)
• apply_jacobian! 3D kernel on quadrature nodes (Optimized apply_jacobian! 3D GPU kernel #3017)
• calc_source_terms! 2D and 3D kernels on quadrature nodes (Add GPU kernel for calc_sources!  #3012)
• 2D kernels for prolong2boundaries! and calc_boundary_flux! (started in WIP: Add 2D GPU kernels for boundary conditions  #3007, the 2D prolong2boundaries_per_boundary! in dg_2d_gpu.jl needs to be moved in the CPU file dgsem_p4est/dg_2d.jl, see comment regarding 2D kernel)
• 3D kernels for prolong2boundaries! and calc_boundary_flux! (Add 3D GPU calc_boundary_flux! kernels #3034)
• Analysis and eventual optimization of calc_surface_integral! and prolong2interfaces
• Fuse looped may be advantageous in some cases (for example calc_source_terms! and apply_jacobian! can be fused)
• 2D and 3D optimized kernels for prolong2boundaries and calc_boundary_flux! (see Avoid launching a kernel for each boundary
• 2D and 3D optimization for calc_surface_integrals! (Better parallelism for calc_surface_integral! GPU kernels #3058)

Step 3: Advanced features

• Proper solution for the "no cache on GPU" problem
• MPI / multi-GPU
• Get rid of VecOfArrays Switch VecOfArrays for RecursiveArrayTools's VectorOfArray #2952
• AMR
• Fix "undefined external symbol 'log'" (see comment)
• Figure out whether backend should be first argument to functions, or last, or ...

Later

• Unify/simplify mesh vs. MeshT
• T8code
• Figure out how to show backend in splash screen Add trixi_backend_info! to show which GPU backend is being used #2906

Not planned

• TreeMesh
• StructuredMesh
• UnstructuredMesh

[vchuravy]

It would be much nicer to hide away the implementation and launch of kernels, as done in PointNeighbors.jl:

An early proposal for that is in trixi-framework/TrixiBase.jl#66

[vchuravy]

We should also add a way to display information about the GPU environment instead of just

┌──────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Environment information                                                                          │
│ ═══════════════════════                                                                          │
│ #threads: …………………………………………………………………… 1                                                           │
│ threading backend: …………………………………………… polyester                                                   │
└──────────────────────────────────────────────────────────────────────────────────────────────────┘

[sloede]

We should also add a way to display information about the GPU environment

Agreed. I added it to the OP above

[MarcoArtiano]

With the Step 2b we will likely increase the amount of kernel GPU functions, as some of the kernel will probably have a different kernel from the CPU, in order to optimize the GPU version. Would it make sense to put all the functions in dg_2d_gpu.jl for example?

[benegee]

With the Step 2b we will likely increase the amount of kernel GPU functions, as some of the kernel will probably have a different kernel from the CPU, in order to optimize the GPU version. Would it make sense to put all the functions in dg_2d_gpu.jl for example?

Yes, the implementations are expected to diverge at this point. And yes, it would be good to put them in dedicated _gpu files.