Skip to content

Do not implicitly shut down the shared reusable executor#498

Merged
basnijholt merged 1 commit into
mainfrom
fix-reusable-executor-shutdown
Jun 10, 2026
Merged

Do not implicitly shut down the shared reusable executor#498
basnijholt merged 1 commit into
mainfrom
fix-reusable-executor-shutdown

Conversation

@basnijholt

Copy link
Copy Markdown
Member

What

Follow-up to #497. Runners no longer shut down the internally created executor when they finish; shutdown_executor now defaults to plain False semantics (explicit opt-in), and the docstrings are updated accordingly.

Why

Since #497, every runner with executor=None shares loky's reusable executor singleton. BaseRunner._cleanup used to call executor.shutdown(wait=True) whenever the runner had created the executor itself (shutdown_executor or (executor is None)), which now kills the shared executor out from under any other runner still using it:

loky.process_executor.ShutdownExecutorError: cannot schedule new futures after shutdown

This broke the Read the Docs build of #497 while executing tutorial.advanced-topics.md (a cancelled goal-less runner's cleanup raced with the next runner's await runner.task). The race predates #497 on macOS/Windows, where loky was already the default, but docs only execute on Linux so it was never hit.

Idle loky workers shut down on their own after their timeout, and loky cleans up at interpreter exit, so nothing leaks.

Testing

  • Minimal repro (cancel a goal-less runner, then run a second runner) raised ShutdownExecutorError before, passes now.
  • Full suite passes on Python 3.14.5 (except the pre-existing test_to_dataframe failures that also occur on main).
  • pre-commit passes.

All runners with executor=None now share loky's reusable executor
singleton, so a finishing (or cancelled) runner must not shut it down:
another runner may still be submitting work to it, which raised
loky.process_executor.ShutdownExecutorError (seen in the Read the Docs
build of tutorial.advanced-topics.md). The executor is only shut down
when the user explicitly passes shutdown_executor=True; idle loky
workers time out on their own.
@basnijholt basnijholt enabled auto-merge (squash) June 10, 2026 22:27
@basnijholt basnijholt disabled auto-merge June 10, 2026 22:29
@basnijholt basnijholt merged commit 15a81c7 into main Jun 10, 2026
17 of 18 checks passed
@basnijholt basnijholt deleted the fix-reusable-executor-shutdown branch June 10, 2026 22:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant