|
| 1 | +# Coupling with other Libraries |
| 2 | +A lot of scientific Python packages already know how to distribute work over many processes - they just need to be |
| 3 | +handed an object that behaves like an executor or a worker pool. Because executorlib implements the |
| 4 | +[Executor interface](https://docs.python.org/3/library/concurrent.futures.html#executor-objects) of the Python standard |
| 5 | +library, it can be passed to these packages in place of the |
| 6 | +[ProcessPoolExecutor](https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor) or the |
| 7 | +[multiprocessing.Pool](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool). The practical |
| 8 | +benefit for you as a scientist is that the workflow stays exactly the same when you move from your laptop to a high |
| 9 | +performance computer (HPC): you develop and test with the [Single Node Executor](https://executorlib.readthedocs.io/en/latest/1-single-node.html) |
| 10 | +and then switch to the [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html) or the |
| 11 | +[HPC Job Executor](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html) by changing a single class name. |
| 12 | + |
| 13 | +```{note} |
| 14 | +The examples below require the respective third-party package to be installed in addition to executorlib (for example |
| 15 | +`pip install emcee`). They are shown as reference code rather than executed examples, because these optional packages are |
| 16 | +not part of the executorlib test environment. In every example the `SingleNodeExecutor` can be replaced by a |
| 17 | +`FluxJobExecutor`, `FluxClusterExecutor`, `SlurmJobExecutor` or `SlurmClusterExecutor` to scale the same workflow to an |
| 18 | +HPC cluster. |
| 19 | +``` |
| 20 | + |
| 21 | +## emcee (Markov Chain Monte Carlo) |
| 22 | +[emcee](https://emcee.readthedocs.io) is a widely used Python package for Markov Chain Monte Carlo (MCMC) sampling, for |
| 23 | +example to estimate the posterior distribution of model parameters from experimental data. The likelihood function has |
| 24 | +to be evaluated many times per sampling step, and these evaluations are independent of each other, so they can be |
| 25 | +executed in parallel. The `EnsembleSampler` of emcee accepts any worker pool which provides a `map()` function via the |
| 26 | +`pool` parameter. As executorlib provides this interface, an executorlib `Executor` can be used directly as a drop-in |
| 27 | +replacement for the [multiprocessing.Pool](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool) |
| 28 | +which is recommended in the [emcee parallelization tutorial](https://emcee.readthedocs.io/en/stable/tutorials/parallel/): |
| 29 | +```python |
| 30 | +import numpy as np |
| 31 | +import emcee |
| 32 | +from executorlib import SingleNodeExecutor |
| 33 | + |
| 34 | + |
| 35 | +def log_prob(theta): |
| 36 | + return -0.5 * np.sum(theta**2) |
| 37 | + |
| 38 | + |
| 39 | +initial = np.random.randn(32, 5) |
| 40 | +nwalkers, ndim = initial.shape |
| 41 | + |
| 42 | +with SingleNodeExecutor(block_allocation=True) as exe: |
| 43 | + sampler = emcee.EnsembleSampler(nwalkers, ndim, log_prob, pool=exe) |
| 44 | + sampler.run_mcmc(initial, 100, progress=True) |
| 45 | +``` |
| 46 | +Here the `block_allocation=True` parameter is set to reuse the same Python processes for the repeated evaluation of the |
| 47 | +`log_prob()` function, which reduces the overhead for these many short function calls - |
| 48 | +[block allocation](https://executorlib.readthedocs.io/en/latest/1-single-node.html#block-allocation). For computationally |
| 49 | +more expensive likelihood functions the parallel evaluation provides a substantial speed-up, and by replacing the |
| 50 | +`SingleNodeExecutor` with a `FluxJobExecutor` the very same sampling can be distributed over multiple compute nodes of an |
| 51 | +HPC cluster. |
| 52 | + |
| 53 | +## pipefunc (Function Pipelines) |
| 54 | +[pipefunc](https://pipefunc.readthedocs.io) is a library to build function pipelines, where the output of one function |
| 55 | +is used as the input for the next function, including map-reduce patterns over many parameters. pipefunc takes care of |
| 56 | +the book keeping of the pipeline, while the actual execution of the individual functions is delegated to an executor. |
| 57 | +The `map()` function of a pipefunc `Pipeline` accepts an `executor` parameter, which can either be a single executorlib |
| 58 | +`Executor` or a dictionary which assigns a dedicated executor to each output: |
| 59 | +```python |
| 60 | +import numpy as np |
| 61 | +from pipefunc import Pipeline, pipefunc |
| 62 | +from executorlib import SingleNodeExecutor |
| 63 | + |
| 64 | + |
| 65 | +@pipefunc(output_name="y", mapspec="x[i] -> y[i]") |
| 66 | +def f(x): |
| 67 | + return x**2 |
| 68 | + |
| 69 | + |
| 70 | +@pipefunc(output_name="z", mapspec="y[i] -> z[i]") |
| 71 | +def g(y): |
| 72 | + return y + 1 |
| 73 | + |
| 74 | + |
| 75 | +pipeline = Pipeline([f, g]) |
| 76 | +inputs = {"x": [1, 2, 3]} |
| 77 | + |
| 78 | +with SingleNodeExecutor() as exe: |
| 79 | + results = pipeline.map(inputs, executor=exe) |
| 80 | + print(results["z"].output.tolist()) |
| 81 | +``` |
| 82 | +Assigning a different executor to each output enables fine-grained control over the computing resources. For example a |
| 83 | +serial preprocessing step can be executed on a single core while a computationally expensive simulation step is |
| 84 | +distributed over multiple compute nodes: |
| 85 | +```python |
| 86 | +executor = { |
| 87 | + "y": SingleNodeExecutor(max_workers=2), |
| 88 | + "z": SingleNodeExecutor(max_workers=4), |
| 89 | +} |
| 90 | +results = pipeline.map(inputs, executor=executor) |
| 91 | +``` |
| 92 | +The combination of pipefunc and executorlib is explained in more detail in the |
| 93 | +[pipefunc documentation on execution and parallelism](https://pipefunc.readthedocs.io/en/latest/concepts/execution-and-parallelism/). |
| 94 | + |
| 95 | +## omp4py (OpenMP for Python) |
| 96 | +The [thread based parallelism](https://executorlib.readthedocs.io/en/latest/1-single-node.html#thread-parallel-functions) |
| 97 | +of executorlib is most commonly used to control the number of threads in linked libraries like NumPy. With |
| 98 | +[omp4py](https://omp4py.readthedocs.io) - a Python implementation of [OpenMP](https://www.openmp.org) - it is also |
| 99 | +possible to write thread parallel Python code directly. The number of threads assigned to the Python function is set via |
| 100 | +the `threads_per_core` parameter in the `resource_dict`. The following example approximates the value of pi using a |
| 101 | +parallel for loop with an OpenMP reduction: |
| 102 | +```python |
| 103 | +import random |
| 104 | +from omp4py import omp |
| 105 | +from executorlib import SingleNodeExecutor |
| 106 | + |
| 107 | + |
| 108 | +@omp |
| 109 | +def calc_pi(num_points): |
| 110 | + count = 0 |
| 111 | + with omp("parallel for reduction(+:count)"): |
| 112 | + for i in range(num_points): |
| 113 | + x = random.random() |
| 114 | + y = random.random() |
| 115 | + if x * x + y * y <= 1.0: |
| 116 | + count += 1 |
| 117 | + return 4 * (count / num_points) |
| 118 | + |
| 119 | + |
| 120 | +with SingleNodeExecutor() as exe: |
| 121 | + future = exe.submit(calc_pi, 10000000, resource_dict={"threads_per_core": 4}) |
| 122 | + print(future.result()) |
| 123 | +``` |
| 124 | +The `threads_per_core` parameter sets the environment variables which control the number of threads, so the requested |
| 125 | +number of cores is reserved for the threads created by omp4py inside the `calc_pi()` function. |
| 126 | + |
| 127 | +## pylammpsmpi (MPI-parallel LAMMPS) |
| 128 | +[pylammpsmpi](https://pylammpsmpi.readthedocs.io) provides a Python interface to the molecular dynamics code |
| 129 | +[LAMMPS](https://www.lammps.org) which distributes the simulation over multiple MPI ranks while the Python process |
| 130 | +itself remains serial. Internally pylammpsmpi uses an executor to start the MPI-parallel LAMMPS processes, so an |
| 131 | +executorlib `Executor` can be provided via the `executor` parameter. In combination with |
| 132 | +[atomistics](https://atomistics.readthedocs.io) this can be used to run an MPI-parallel molecular dynamics simulation: |
| 133 | +```python |
| 134 | +from ase.build import bulk |
| 135 | +from atomistics.calculators import ( |
| 136 | + calc_molecular_dynamics_nvt_with_lammpslib, |
| 137 | + get_potential_by_name, |
| 138 | +) |
| 139 | +from pylammpsmpi import LammpsASELibrary |
| 140 | +from executorlib import SingleNodeExecutor |
| 141 | + |
| 142 | +structure = bulk("Ti") |
| 143 | +potential = get_potential_by_name(potential_name="2016--Mendelev-M-I--Ti-3--LAMMPS--ipr1") |
| 144 | + |
| 145 | +with SingleNodeExecutor(resource_dict={"cores": 2}) as exe: |
| 146 | + lmp = LammpsASELibrary(executor=exe) |
| 147 | + result_dict = calc_molecular_dynamics_nvt_with_lammpslib( |
| 148 | + structure=structure, |
| 149 | + potential_dataframe=potential, |
| 150 | + lmp=lmp, |
| 151 | + ) |
| 152 | + lmp.close() |
| 153 | +``` |
| 154 | +The `resource_dict={"cores": 2}` assigns two MPI ranks to the LAMMPS simulation. As for the other examples, replacing |
| 155 | +the `SingleNodeExecutor` with one of the HPC executors distributes the LAMMPS simulation over the compute nodes of an |
| 156 | +HPC cluster without any further changes to the simulation code. |
| 157 | + |
| 158 | +## General Pattern |
| 159 | +The four examples above follow the same pattern: a library which already supports parallel execution accepts an |
| 160 | +executorlib `Executor` (or worker pool), so executorlib takes over the distribution of the work. Whenever a Python |
| 161 | +package accepts a [concurrent.futures.Executor](https://docs.python.org/3/library/concurrent.futures.html#executor-objects) |
| 162 | +or a [multiprocessing.Pool](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool) - typically |
| 163 | +exposed via a parameter named `executor` or `pool` - it can be combined with executorlib. The recommended approach |
| 164 | +remains the same in all cases: |
| 165 | + |
| 166 | +* Develop and test the workflow with the [Single Node Executor](https://executorlib.readthedocs.io/en/latest/1-single-node.html) |
| 167 | + on a laptop or workstation. |
| 168 | +* Switch to the [HPC Job Executor](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html) or the |
| 169 | + [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html) to scale to an HPC cluster by |
| 170 | + changing only the executor class name. |
0 commit comments