Add optional PETSc Kokkos support

[boyceg]

Summary

• add a new installer flag --enable-petsc-kokkos in autoibamr.sh
• thread the new toggle into PETSc configure options so PETSc is built with --download-kokkos=1 --download-kokkos-kernels=1 when enabled
• document the new option in README.md

Why

This enables opting into PETSc's Kokkos backend through autoibamr without changing defaults.

Validation

• bash -n autoibamr.sh
• bash -n IBAMR-toolchain/packages/petsc.package
• ./autoibamr.sh --help | rg -n "petsc-kokkos"

[drwells]

I think this is worth doing but we will need to verify that deal.II uses the same Kokkos installation as PETSc in this case. That should be easy (just provide -DKOKKOS_DIR=\"${PETSC_DIR}\"). Similarly, since we have to install Kokkos one way or another, we should do this any time we install deal.II.

Do you have any measurements for how long it takes PETSc to compile Trilinos Kokkos? The version bundled with deal.II is mostly templates so it is nearly free to compile. If it is fast (perhaps one minute at most) then I'd be in favor of just turning it on by default to make configuration simpler.

Edit: I fixed a typo. One of my goals with autoibamr is to shrink the total number of possible configurations so that maintenance is easier: to that end I prefer turning things on my default.

[boyceg]

I think this is worth doing but we will need to verify that deal.II uses the same Kokkos installation as PETSc in this case. That should be easy (just provide -DKOKKOS_DIR=\"${PETSC_DIR}\" in this case). Similarly, since we have to install Kokkos one way or another, we should do this any time we install deal.II.

Yes, I agree. It should be easy enough to add.

Do you have any measurements for how long it takes PETSc to compile Trilinos Kokkos? The version bundled with deal.II is mostly templates so it is nearly free to compile. If it is fast (perhaps one minute at most) then I'd be in favor of just turning it on by default to make configuration simpler.

I am not sure. I sort of think we should not turn it on by default yet, because I don't know how Kokkos and RAJA are going to interact, and whenever we make it to SAMRAI 4, we are going to be needing to use RAJA.

Edit: I fixed a typo. One of my goals with autoibamr is to shrink the total number of possible configurations so that maintenance is easier: to that end I prefer turning things on my default.

I agree, but in this case, I think it makes sense not to turn it on until we have a better handle on the implications of turning it on.

What do you think about adding a "developer" or "experimental" flag that turns things like Kokkos "on" by default?

[drwells]

That could be a problem because deal.II uses Kokkos (so Cardinal uses Kokkos). Ultimately they are just two shared-object libraries: unless we try to use both at once I don't think we can run into problems. Raja supports some kokkos plugins so they aren't incompatible.

[drwells]

Hence I'd rather just turn it on now and not have some extra developer flag.

[boyceg]

Works for me.