|
| 1 | +# Comparison |
| 2 | + |
| 3 | +executorlib is the lightest path to take *existing* Python functions and scale them across high performance computing |
| 4 | +(HPC) nodes — with per-function-call resource control and native [SLURM](https://slurm.schedmd.com) and |
| 5 | +[flux](http://flux-framework.org) integration — without rewriting your code into a new paradigm. It extends the standard |
| 6 | +library [Executor interface](https://docs.python.org/3/library/concurrent.futures.html#executor-objects) you already |
| 7 | +know, rather than asking you to adopt a new data, actor, or workflow model. |
| 8 | + |
| 9 | +This page compares executorlib with the tools scientists most often weigh it against, and is honest about when each |
| 10 | +alternative is the better choice. |
| 11 | + |
| 12 | +## At a glance |
| 13 | + |
| 14 | +| | executorlib | `concurrent.futures` | Dask | Parsl | Ray | Snakemake | |
| 15 | +|---|---|---|---|---|---|---| |
| 16 | +| Drop-in `Executor` API | ✅ | ✅ | ⚠️ | ❌ | ❌ | ❌ | |
| 17 | +| Per-call resource assignment | ✅ | ❌ | ⚠️ | ✅ | ✅ | ✅ | |
| 18 | +| Native HPC scheduler (SLURM/flux) | ✅ | ❌ | ⚠️ | ✅ | ⚠️ | ✅ | |
| 19 | +| MPI-parallel functions | ✅ | ❌ | ⚠️ | ✅ | ⚠️ | ⚠️ | |
| 20 | +| Caching of results | ✅ | ❌ | ⚠️ | ✅ | ❌ | ✅ | |
| 21 | +| Setup / learning overhead | Low | Very low | Medium | Medium | Medium | High | |
| 22 | + |
| 23 | +✅ first-class · ⚠️ possible via an add-on or extra configuration · ❌ not supported. |
| 24 | + |
| 25 | +## [Concurrent futures](https://docs.python.org/3/library/concurrent.futures.html) |
| 26 | + |
| 27 | +The [`concurrent.futures`](https://docs.python.org/3/library/concurrent.futures.html) module is where most parallel |
| 28 | +Python starts: `ProcessPoolExecutor` and `ThreadPoolExecutor` run functions in parallel on a single machine. executorlib |
| 29 | +deliberately mirrors this `Executor` interface so the step up to HPC is minimal. |
| 30 | + |
| 31 | +**Use `concurrent.futures` instead when** your work fits comfortably on one machine and you do not need HPC schedulers, |
| 32 | +per-call resource control, MPI, or caching. |
| 33 | + |
| 34 | +## [Dask](https://www.dask.org) |
| 35 | + |
| 36 | +Dask scales NumPy/pandas-style workloads with parallel arrays, dataframes, and a `delayed`/futures API, and reaches HPC |
| 37 | +via [dask-jobqueue](https://jobqueue.dask.org). It is excellent for large out-of-core data structures, but its futures |
| 38 | +API is its own, and per-task resources and MPI rely on add-ons. |
| 39 | + |
| 40 | +**Use Dask instead when** your problem is fundamentally about large arrays/dataframes or out-of-core data, rather than |
| 41 | +scheduling independent Python functions across an HPC allocation. |
| 42 | + |
| 43 | +## [Parsl](https://parsl-project.org) |
| 44 | + |
| 45 | +Parsl is the closest conceptual neighbor: a parallel scripting library with strong HPC support, MPI apps, and app-level |
| 46 | +caching. It uses its own decorator/app model (`@python_app`) and an executor-configuration layer rather than the standard |
| 47 | +library `Executor` interface. |
| 48 | + |
| 49 | +**Use Parsl instead when** you are authoring a larger dataflow of apps and want its app/configuration model, or you need |
| 50 | +a provider it supports that executorlib does not. |
| 51 | + |
| 52 | +## [Ray](https://www.ray.io) |
| 53 | + |
| 54 | +Ray is a distributed framework built around remote tasks and stateful actors, widely used for AI/ML and reinforcement |
| 55 | +learning. It assigns CPUs/GPUs per task, but adopting Ray means adopting its `@ray.remote` programming model, and HPC |
| 56 | +scheduler integration is via cluster launchers rather than native SLURM/flux. |
| 57 | + |
| 58 | +**Use Ray instead when** you need long-lived stateful actors, an AI/ML ecosystem, or a distributed-object model — and you |
| 59 | +are willing to write code in Ray's paradigm. |
| 60 | + |
| 61 | +## [Snakemake](https://snakemake.github.io) |
| 62 | + |
| 63 | +Snakemake is a file-oriented workflow manager: you declare rules with inputs/outputs and it builds a dependency graph, |
| 64 | +with strong HPC support and file-based caching. It is a workflow DSL, not a drop-in way to parallelize Python functions. |
| 65 | + |
| 66 | +**Use Snakemake instead when** your pipeline is naturally expressed as files transformed by rules, and you want |
| 67 | +reproducible, file-driven workflow management rather than in-process Python futures. |
| 68 | + |
| 69 | +## Choose executorlib when |
| 70 | + |
| 71 | +- You already have Python functions and want to scale them across HPC nodes with minimal rewriting. |
| 72 | +- You want to assign cores, threads, or GPUs **per function call**. |
| 73 | +- You want native [SLURM](https://slurm.schedmd.com) / [flux](http://flux-framework.org) integration and optional MPI |
| 74 | + parallelism inside your functions. |
| 75 | +- You want optional caching of intermediate results for rapid, iterative prototyping in notebooks. |
0 commit comments