Chunk distribution strategies (#824)

* Add chunk distribution algorithms

* Python bindings: Chunk distribution algorithms

* Use chunk distribtion algorithms in openpmd-pipe

* Testing

* Add DiscardingStrategy

* Make Strategy class extensible from Python

@todo Why do we need to increase the refcount twice??

* Make mergeChunks function public

* Add mergeChunksFromSameSourceID

* Add RoundRobinOfSourceRanks strategy

* Add Blocks distribution strategy

* Add BlocksOfSourceRanks strategy

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add rank info to assign() params

* Fix the Python trampoline logic

* Use discard strategy as second run, only consider my own hostname

* IncreaseGranularity strategy

supports different granularities at write and read sides

* Add blocksofsourcerank to pipe script

* CI fixes

* Move BlockSlicer somewhere else

* More thorough documentation

* Cleanup, documentation, proper use of rank specification

* Actual testing

* Test and fix ByHostname strategy

* CI fixes

* Automate CoreTest

* Guard against unprintableString issue on Windows

* Use generic flag instead of -n 2

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Author: franzpoeschel

CMakeLists.txt

@@ -422,7 +422,7 @@ set(CORE_SOURCE
         src/backend/PatchRecord.cpp
         src/backend/PatchRecordComponent.cpp
         src/backend/Writable.cpp
-        src/benchmark/mpi/OneDimensionalBlockSlicer.cpp
+        src/auxiliary/OneDimensionalBlockSlicer.cpp
         src/helper/list_series.cpp
         src/snapshots/ContainerImpls.cpp
         src/snapshots/ContainerTraits.cpp
@@ -1346,30 +1346,30 @@ if(openPMD_BUILD_TESTING)
                 )
                 add_test(NAME CLI.pipe.py
                     COMMAND sh -c
-                        "${MPI_TEST_EXE} ${Python_EXECUTABLE}                      \
-                            ${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe       \
-                            --infile ../samples/git-sample/data%T.h5               \
-                            --outfile ../samples/git-sample/data%T.bp &&           \
-                                                                                   \
-                        ${MPI_TEST_EXE} ${Python_EXECUTABLE}                       \
-                            ${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe       \
-                            --infile ../samples/git-sample/data00000100.h5         \
-                            --outfile                                              \
-                                ../samples/git-sample/single_iteration_%T.bp &&    \
-                                                                                   \
-                        ${MPI_TEST_EXE} ${Python_EXECUTABLE}                       \
-                            ${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe       \
-                            --infile ../samples/git-sample/thetaMode/data%T.h5     \
-                            --outfile                                              \
-                                ../samples/git-sample/thetaMode/data_%T.bp &&      \
-                                                                                   \
-                        ${MPI_TEST_EXE} ${Python_EXECUTABLE}                       \
-                            ${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe       \
-                            --infile ../samples/git-sample/thetaMode/data_%T.bp    \
-                            --outfile ../samples/git-sample/thetaMode/data%T.json  \
-                            --outconfig '                                          \
-                                json.attribute.mode = \"short\"                  \n\
-                                json.dataset.mode = \"template_no_warn\"'          \
+                        "${MPI_TEST_EXE} ${MPIEXEC_NUMPROC_FLAG} 2 ${Python_EXECUTABLE} \
+                            ${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe            \
+                            --infile ../samples/git-sample/data%T.h5                    \
+                            --outfile ../samples/git-sample/data%T.bp &&                \
+                                                                                        \
+                        ${MPI_TEST_EXE} ${Python_EXECUTABLE}                            \
+                            ${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe            \
+                            --infile ../samples/git-sample/data00000100.h5              \
+                            --outfile                                                   \
+                                ../samples/git-sample/single_iteration_%T.bp &&         \
+                                                                                        \
+                        ${MPI_TEST_EXE} ${MPIEXEC_NUMPROC_FLAG} 2 ${Python_EXECUTABLE}  \
+                            ${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe            \
+                            --infile ../samples/git-sample/thetaMode/data%T.h5          \
+                            --outfile                                                   \
+                                ../samples/git-sample/thetaMode/data_%T.bp &&           \
+                                                                                        \
+                        ${MPI_TEST_EXE} ${Python_EXECUTABLE}                            \
+                            ${openPMD_RUNTIME_OUTPUT_DIRECTORY}/openpmd-pipe            \
+                            --infile ../samples/git-sample/thetaMode/data_%T.bp         \
+                            --outfile ../samples/git-sample/thetaMode/data%T.json       \
+                            --outconfig '                                               \
+                                json.attribute.mode = \"short\"                       \n\
+                                json.dataset.mode = \"template_no_warn\"'               \
                         "
                     WORKING_DIRECTORY ${openPMD_RUNTIME_OUTPUT_DIRECTORY}
                 )