Large scale simulations can quickly grow in memory requirements, so it could be a nice feature to have native support for low storage style time steps. The general idea is that less memory is required if certain assumptions about the type of updates that are possible with a certain amount of memory can be fulfilled. For example, take the forward Euler update that is common in the miniapps:
These kind of updates are always possible as T1 and T2 are separate memory locations. We can half the memory requirement however if the 1S style update can be achieved with only 1 memory location:
The issue with this is the potential of overwrite, as the right hand side f(T1) depends on locations adjacent to it, so if this is done naively parts of T1 get overwritten before they are needed elsewhere. This can be done with proper implementation however, if local blocks of T1 are loaded as cache with a halo of dependence taken into account. Then within a halo distance of the cache block, overwrites are safe as all required memory is cached.
That is not to say this is easy, but it reminds me of how ImplictGlobalGrid.jl works, just at thread level instead of device level. Information about how much cache is available to a device would need to inform the size of the local block and safe overwrite locations would need to be tracked, etc.
The payoff is substantial for forward Euler, changing it to a 1S scheme and halving the memory requirement can easily be the difference between fitting on the memory footprint of one GPU/CPU or not. There are also methods of 2S up to 4th order accuracy and 3S up to 5th order accuracy in the literature that I am aware of that would become possible in this architecture.
I completely understand if this is too hard to support, but I think if this was implemented, especially if it can reuse a lot of the current infrastructure in ImplictGlobalGrid.jl, it could be quite compelling.
Large scale simulations can quickly grow in memory requirements, so it could be a nice feature to have native support for low storage style time steps. The general idea is that less memory is required if certain assumptions about the type of updates that are possible with a certain amount of memory can be fulfilled. For example, take the forward Euler update that is common in the miniapps:
These kind of updates are always possible as
T1andT2are separate memory locations. We can half the memory requirement however if the 1S style update can be achieved with only 1 memory location:The issue with this is the potential of overwrite, as the right hand side
f(T1)depends on locations adjacent to it, so if this is done naively parts ofT1get overwritten before they are needed elsewhere. This can be done with proper implementation however, if local blocks ofT1are loaded as cache with a halo of dependence taken into account. Then within a halo distance of the cache block, overwrites are safe as all required memory is cached.That is not to say this is easy, but it reminds me of how
ImplictGlobalGrid.jlworks, just at thread level instead of device level. Information about how much cache is available to a device would need to inform the size of the local block and safe overwrite locations would need to be tracked, etc.The payoff is substantial for forward Euler, changing it to a 1S scheme and halving the memory requirement can easily be the difference between fitting on the memory footprint of one GPU/CPU or not. There are also methods of 2S up to 4th order accuracy and 3S up to 5th order accuracy in the literature that I am aware of that would become possible in this architecture.
I completely understand if this is too hard to support, but I think if this was implemented, especially if it can reuse a lot of the current infrastructure in
ImplictGlobalGrid.jl, it could be quite compelling.