Combine interact! kernels to reduce GPU latency

[efaulhaber]

Here is the pipe flow 2D example as a benchmark.

2k particles

CPU (72 threads)

For small problems, we can even see a 1.85x speedup on the CPU with combined threaded loops. No timers is slightly faster, but only because of the timer overhead. The only difference is that synchronization is skipped on GPUs.

With IndividualTimers():

  system interaction                    2.50k    237ms    7.5%  94.7μs   48.3MiB    3.9%  19.8KiB
    fluid1-fluid1                       2.50k   75.1ms    2.4%  30.0μs   6.26MiB    0.5%  2.56KiB
    ~system interaction~                2.50k   41.8ms    1.3%  16.7μs   5.36MiB    0.4%  2.20KiB
    fluid1-boundary3                    2.50k   30.9ms    1.0%  12.3μs   6.26MiB    0.5%  2.56KiB
    fluid1-open_boundary2               2.50k   24.8ms    0.8%  9.90μs   7.33MiB    0.6%  3.00KiB
    open_boundary2-boundary3            2.50k   21.3ms    0.7%  8.54μs   7.33MiB    0.6%  3.00KiB
    open_boundary2-open_boundary2       2.50k   21.3ms    0.7%  8.54μs   8.40MiB    0.7%  3.44KiB
    open_boundary2-fluid1               2.50k   20.9ms    0.7%  8.35μs   7.33MiB    0.6%  3.00KiB
    boundary3-open_boundary2            2.50k    277μs    0.0%   111ns     0.00B    0.0%    0.00B
    boundary3-fluid1                    2.50k    265μs    0.0%   106ns     0.00B    0.0%    0.00B
    boundary3-boundary3                 2.50k    185μs    0.0%  73.8ns     0.00B    0.0%    0.00B

With CombinedTimers():

  system interaction                    2.50k    128ms    5.1%  51.2μs   19.8MiB    1.7%  8.11KiB
    fluid1-*                            2.50k   87.1ms    3.5%  34.8μs   8.70MiB    0.7%  3.56KiB
    open_boundary2-*                    2.50k   23.8ms    1.0%  9.53μs   9.77MiB    0.8%  4.00KiB
    ~system interaction~                2.50k   15.4ms    0.6%  6.18μs   1.34MiB    0.1%     561B
    boundary3-*                         2.50k   1.72ms    0.1%   688ns     0.00B    0.0%    0.00B

With NoTimers():

  system interaction                    2.50k    113ms    4.6%  45.1μs   18.5MiB    1.6%  7.56KiB

GPU (H100)

Here, we get a 2x speedup from combining the kernels and another 1.5x by removing synchronization, giving a total speedup of 3x.

With IndividualTimers():

  system interaction                    2.50k    3.23s   20.5%  1.29ms   2.51GiB   39.1%  1.03MiB
    fluid1-fluid1                       2.50k    684ms    4.3%   274μs    357MiB    5.4%   146KiB
    fluid1-open_boundary2               2.50k    552ms    3.5%   221μs    439MiB    6.7%   180KiB
    open_boundary2-open_boundary2       2.50k    514ms    3.3%   205μs    519MiB    7.9%   212KiB
    open_boundary2-fluid1               2.50k    468ms    3.0%   187μs    438MiB    6.7%   179KiB
    fluid1-boundary3                    2.50k    458ms    2.9%   183μs    361MiB    5.5%   148KiB
    open_boundary2-boundary3            2.50k    431ms    2.7%   172μs    443MiB    6.7%   181KiB
    ~system interaction~                2.50k   96.8ms    0.6%  38.7μs   5.36MiB    0.1%  2.20KiB
    boundary3-fluid1                    2.50k   10.5ms    0.1%  4.22μs   1.91MiB    0.0%     800B
    boundary3-open_boundary2            2.50k   9.33ms    0.1%  3.73μs   1.91MiB    0.0%     800B
    boundary3-boundary3                 2.50k   9.32ms    0.1%  3.73μs   1.91MiB    0.0%     800B

With CombinedTimers():

  system interaction                    2.50k    1.63s   14.7%   651μs   1.22GiB   24.3%   511KiB
    fluid1-*                            2.50k    978ms    8.8%   391μs    582MiB   11.3%   238KiB
    open_boundary2-*                    2.50k    597ms    5.4%   239μs    662MiB   12.9%   271KiB
    ~system interaction~                2.50k   36.3ms    0.3%  14.5μs   1.34MiB    0.0%     561B
    boundary3-*                         2.50k   15.6ms    0.1%  6.22μs   3.43MiB    0.1%  1.41KiB

With NoTimers():

  system interaction                    2.50k    1.07s   13.5%   429μs   1.22GiB   25.0%   510KiB

1.6M particles

CPU (72 cores)

This problem is now so large that the extra overhead from having multiple multithreaded loops is negligible. The performance is identical in all three versions.

With IndividualTimers():

  system interaction                    2.85k    98.2s   80.2%  34.5ms   54.9MiB    0.4%  19.8KiB
    fluid1-fluid1                       2.85k    89.4s   73.0%  31.4ms   7.12MiB    0.0%  2.56KiB
    fluid1-boundary3                    2.85k    6.48s    5.3%  2.28ms   7.12MiB    0.0%  2.56KiB
    fluid1-open_boundary2               2.85k    2.10s    1.7%   738μs   8.34MiB    0.1%  3.00KiB
    ~system interaction~                2.85k    111ms    0.1%  38.9μs   6.10MiB    0.0%  2.20KiB
    open_boundary2-open_boundary2       2.85k   39.9ms    0.0%  14.0μs   9.55MiB    0.1%  3.44KiB
    open_boundary2-fluid1               2.85k   36.8ms    0.0%  12.9μs   8.34MiB    0.1%  3.00KiB
    open_boundary2-boundary3            2.85k   31.2ms    0.0%  11.0μs   8.34MiB    0.1%  3.00KiB
    boundary3-open_boundary2            2.85k   2.24ms    0.0%   786ns     0.00B    0.0%    0.00B
    boundary3-fluid1                    2.85k   1.51ms    0.0%   529ns     0.00B    0.0%    0.00B
    boundary3-boundary3                 2.85k   1.06ms    0.0%   372ns     0.00B    0.0%    0.00B

With CombinedTimers():

  system interaction                    2.85k     100s   80.4%  35.2ms   75.5MiB    0.5%  27.2KiB
    fluid1-*                            2.85k    99.1s   79.5%  34.8ms   9.94MiB    0.1%  3.58KiB
    open_boundary2-*                    2.85k    1.03s    0.8%   362μs   64.0MiB    0.4%  23.0KiB
    ~system interaction~                2.85k   76.3ms    0.1%  26.8μs   1.52MiB    0.0%     560B
    boundary3-*                         2.85k   7.85ms    0.0%  2.76μs     0.00B    0.0%    0.00B

With NoTimers():

  system interaction                    2.85k     100s   80.3%  35.0ms   21.0MiB    0.1%  7.56KiB

GPU (H100)

On the GPU, we can gain some performance here, with a 7% speedup from combining the kernels and another 7% from removing synchronization.

With IndividualTimers():

  system interaction                    2.85k    11.4s   56.7%  3.99ms   2.85GiB   54.0%  1.03MiB
    fluid1-fluid1                       2.85k    7.60s   38.0%  2.67ms    407MiB    7.5%   146KiB
    fluid1-open_boundary2               2.85k    1.46s    7.3%   512μs    499MiB    9.2%   180KiB
    fluid1-boundary3                    2.85k    721ms    3.6%   253μs    411MiB    7.6%   148KiB
    open_boundary2-open_boundary2       2.85k    499ms    2.5%   175μs    591MiB   10.9%   212KiB
    open_boundary2-fluid1               2.85k    489ms    2.4%   172μs    499MiB    9.2%   180KiB
    open_boundary2-boundary3            2.85k    441ms    2.2%   155μs    504MiB    9.3%   181KiB
    ~system interaction~                2.85k    113ms    0.6%  39.6μs   6.10MiB    0.1%  2.20KiB
    boundary3-fluid1                    2.85k   11.8ms    0.1%  4.14μs   2.17MiB    0.0%     800B
    boundary3-boundary3                 2.85k   10.6ms    0.1%  3.73μs   2.17MiB    0.0%     800B
    boundary3-open_boundary2            2.85k   10.6ms    0.1%  3.71μs   2.17MiB    0.0%     800B

With CombinedTimers():

  system interaction                    2.85k    10.6s   55.0%  3.72ms   1.39GiB   36.3%   511KiB
    fluid1-*                            2.85k    9.86s   51.2%  3.46ms    662MiB   16.9%   238KiB
    open_boundary2-*                    2.85k    673ms    3.5%   237μs    753MiB   19.3%   271KiB
    ~system interaction~                2.85k   44.4ms    0.2%  15.6μs   1.52MiB    0.0%     560B
    boundary3-*                         2.85k   17.6ms    0.1%  6.17μs   3.91MiB    0.1%  1.41KiB

With NoTimers():

  system interaction                    2.85k    9.94s   53.4%  3.49ms   1.39GiB   36.3%   511KiB

10M particles

GPU (H100)

This is now large enough that even the small kernels are large enough for the GPU, so there is no speedup from this PR. I don't know why this is getting slightly slower.

With IndividualTimers():

  system interaction                    1.78k    32.0s   71.9%  18.0ms   1.79GiB   54.1%  1.03MiB
    fluid1-fluid1                       1.78k    27.9s   62.7%  15.7ms    255MiB    7.5%   146KiB
    fluid1-open_boundary2               1.78k    2.23s    5.0%  1.25ms    312MiB    9.2%   180KiB
    fluid1-boundary3                    1.78k    975ms    2.2%   548μs    257MiB    7.6%   148KiB
    open_boundary2-open_boundary2       1.78k    296ms    0.7%   166μs    370MiB   10.9%   212KiB
    open_boundary2-fluid1               1.78k    272ms    0.6%   153μs    312MiB    9.2%   180KiB
    open_boundary2-boundary3            1.78k    260ms    0.6%   146μs    315MiB    9.3%   181KiB
    ~system interaction~                1.78k   70.2ms    0.2%  39.4μs   3.82MiB    0.1%  2.20KiB
    boundary3-fluid1                    1.78k   7.38ms    0.0%  4.14μs   1.36MiB    0.0%     800B
    boundary3-boundary3                 1.78k   6.64ms    0.0%  3.73μs   1.36MiB    0.0%     800B
    boundary3-open_boundary2            1.78k   6.58ms    0.0%  3.69μs   1.36MiB    0.0%     800B

With CombinedTimers():

  system interaction                    1.78k    35.4s   74.2%  19.9ms    889MiB   36.4%   511KiB
    fluid1-*                            1.78k    34.8s   72.8%  19.5ms    414MiB   16.9%   238KiB
    open_boundary2-*                    1.78k    641ms    1.3%   360μs    471MiB   19.3%   271KiB
    ~system interaction~                1.78k   27.6ms    0.1%  15.5μs    975KiB    0.0%     561B
    boundary3-*                         1.78k   11.0ms    0.0%  6.18μs   2.45MiB    0.1%  1.41KiB

With NoTimers():

  system interaction                    1.78k    34.8s   73.5%  19.5ms    888MiB   36.4%   511KiB