repro: parallel execution of stages

[sjawhar]

• ❗ I have followed the Contributing to DVC checklist.

• 📖 If this PR requires documentation updates, I have created a separate PR (or issue, at least) in dvc.org and linked it here.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

TODO

• Tests
• Add --jobs argument to CLI

OTHER THOUGHTS

Obviously, when adding a feature like parallel execution, one of the first objections will be something like "but what if two stages can't run in parallel?". The quick answer is: then don't run in parallel!

But I think it would also be pretty easy to accommodate this in a fairly generic way. For example:

stages:
  foo:
    cmd: python blah.py
    mutex: [foobar]
    # other stage properties

  bar:
    cmd: python other_thing.py
    mutex: [foobar, also_another_one]

Probably self-explanatory, but the idea behind this is:

• No two stages that belong to the same mutex group should be run concurrently
• A stage can belong to arbitrarily many mutex groups
• You can call the mutex groups whatever you want, so that they make sense to you

I'd be happy to contribute a follow-up PR with the mutex feature, if you like the idea

[codecov]

Codecov Report

❌ Patch coverage is 97.90210% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.99%. Comparing base (2431ec6) to head (3a821ca).
⚠️ Report is 192 commits behind head on main.

Files with missing lines	Patch %	Lines
dvc/repo/reproduce.py	95.89%	1 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #10922      +/-   ##
==========================================
+ Coverage   90.68%   90.99%   +0.31%     
==========================================
  Files         504      505       +1     
  Lines       39795    41112    +1317     
  Branches     3141     3255     +114     
==========================================
+ Hits        36087    37410    +1323     
- Misses       3042     3064      +22     
+ Partials      666      638      -28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sjawhar]

I think there are some flaky tests :(

[startibartfast]

Looking at the code I noticed that you're using the ThreadPoolExecutor which implements async-based parallelism. This is certainly good for I/O bound tasks but, according to our experience, DVC stages are mainly dealing with CPU bound tasks.
We have built a framework to support our MLOps processes which is designed around the standard DVC package. This framework, among other features, implements parallelism based on multiprocessing to overcome the limitations introduced by Python's GIL (see here for some more infos about our approach https://www.linkedin.com/pulse/rateracer-smarter-pipelines-faster-pricing-carlo-ieva-phd-cfcje/?trackingId=3%2F5ij4ihQemsx3O1EDJ%2B5w%3D%3D).