Skip to content

Fix shutdown hang on session processor thread lock#108

Merged
lstein merged 2 commits intolstein/feature/nicer-shutdownfrom
copilot/fix-shutdown-thread-lock
Mar 2, 2026
Merged

Fix shutdown hang on session processor thread lock#108
lstein merged 2 commits intolstein/feature/nicer-shutdownfrom
copilot/fix-shutdown-thread-lock

Conversation

Copy link
Copy Markdown

Copilot AI commented Mar 2, 2026

Summary

After a generation completes, the server hangs on ^C because Python's threading._shutdown() blocks waiting for the non-daemon session_processor thread to exit — which is sleeping in poll_now_event.wait(polling_interval) and won't notice _stop_event until the 1-second timeout elapses. If the processor was paused via the API, resume_event.wait() (no timeout) would block indefinitely.

Changes to session_processor_default.py:

  • daemon=True on the session processor thread — Python's interpreter shutdown will no longer block on this thread even if it's stuck (e.g., mid-generation)
  • stop() now sets _poll_now_event and _resume_event — immediately wakes the thread from any sleeping/blocked state so it checks stop_event and exits without waiting for the polling interval to elapse
# Before
def stop(self, *args, **kwargs) -> None:
    self._stop_event.set()

# After
def stop(self, *args, **kwargs) -> None:
    self._stop_event.set()
    # Wake the thread if it is sleeping in poll_now_event.wait() or blocked in resume_event.wait() (paused).
    self._poll_now_event.set()
    self._resume_event.set()

Related Issues / Discussions

QA Instructions

  1. Generate an image to completion
  2. Press ^C to shut down the server
  3. Verify the process exits cleanly immediately after "InvokeAI shutting down..." without requiring a second ^C

Also verify with a paused processor: pause via the API, then shut down — process should still exit immediately.

Merge Plan

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Tests added / updated (if applicable)
  • ❗Changes to a redux slice have a corresponding migration
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)
Original prompt

This section details on the original issue you should resolve

<issue_title>[bug]: Shutdown is hanging on a thread lock</issue_title>
<issue_description>### Is there an existing issue for this problem?

  • I have searched the existing issues

Install method

Invoke's Launcher

Operating system

Linux

GPU vendor

Nvidia (CUDA)

GPU model

No response

GPU VRAM

No response

Version number

lstein/feature/nicer-shutdown

Browser

No response

System Information

No response

What happened

After generating an image I tried to shut down the invokeai server by pressing ^C. It printed "InvokeAI shutting down..." and then hung indefinitely. I hit ^C again and it exited after printing a warning message about lock.acquire().

Here is the log:

^C
[2026-03-01 21:52:51,384]::[ModelInstallService]::INFO --> Installer thread 138019081299648 exiting
[2026-03-01 21:52:52,388]::[InvokeAI]::INFO --> InvokeAI shutting down...

^C
Exception ignored in: <module 'threading' from '/home/lstein/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/threading.py'>
Traceback (most recent call last):
  File "/home/lstein/.local/share/uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12/threading.py", line 1624, in _shutdown
    lock.acquire()
KeyboardInterrupt: 

What you expected to happen

I expect InvokeAI to shut down on its own soon after the "shutting down..." message. I suspect that there is a thread somewhere that is holding onto a lock.

How to reproduce the problem

No response

Additional context

No response

Discord username

No response</issue_description>

Comments on the Issue (you are @copilot in this section)


✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

…aemon

Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix shutdown hanging issue on thread lock Fix shutdown hang on session processor thread lock Mar 2, 2026
@lstein lstein marked this pull request as ready for review March 2, 2026 03:26
@lstein lstein self-requested a review as a code owner March 2, 2026 03:26
Copy link
Copy Markdown
Owner

@lstein lstein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works as advertised.

@lstein lstein merged commit 1654dd0 into lstein/feature/nicer-shutdown Mar 2, 2026
14 of 19 checks passed
@lstein lstein deleted the copilot/fix-shutdown-thread-lock branch March 2, 2026 03:29
lstein added a commit that referenced this pull request Mar 2, 2026
* Initial plan

* Fix shutdown hang: wake session processor thread on stop() and mark daemon

Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
lstein added a commit that referenced this pull request Mar 2, 2026
* Initial plan

* Handle KeyboardInterrupt in run_app to allow single Ctrl+C shutdown

Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

* Force os._exit(0) on KeyboardInterrupt to avoid hanging on background threads

Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

Fix graceful shutdown to wait for download/install worker threads (#102)

* Initial plan

* Replace os._exit(0) with ApiDependencies.shutdown() on KeyboardInterrupt

Instead of immediately force-exiting the process on CTRL+C, call
ApiDependencies.shutdown() to gracefully stop the download and install
manager services, allowing active work to complete or cancel cleanly
before the process exits.

Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

* Make stop() idempotent in download and model install services

When CTRL+C is pressed, uvicorn's graceful shutdown triggers the FastAPI
lifespan which calls ApiDependencies.shutdown(), then a KeyboardInterrupt
propagates from run_until_complete() hitting the except block which tries
to call ApiDependencies.shutdown() a second time.

Change both stop() methods to return silently (instead of raising) when
the service is not running. This handles:
- Double-shutdown: lifespan already stopped the services
- Early interrupt: services were never fully started

Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

Fix shutdown hang on session processor thread lock (#108)

* Initial plan

* Fix shutdown hang: wake session processor thread on stop() and mark daemon

Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
lstein added a commit that referenced this pull request Mar 6, 2026
…i#8936)

* Fix: Kill the server with one keyboard interrupt (#94)

* Initial plan

* Handle KeyboardInterrupt in run_app to allow single Ctrl+C shutdown

Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

* Force os._exit(0) on KeyboardInterrupt to avoid hanging on background threads

Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

Fix graceful shutdown to wait for download/install worker threads (#102)

* Initial plan

* Replace os._exit(0) with ApiDependencies.shutdown() on KeyboardInterrupt

Instead of immediately force-exiting the process on CTRL+C, call
ApiDependencies.shutdown() to gracefully stop the download and install
manager services, allowing active work to complete or cancel cleanly
before the process exits.

Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

* Make stop() idempotent in download and model install services

When CTRL+C is pressed, uvicorn's graceful shutdown triggers the FastAPI
lifespan which calls ApiDependencies.shutdown(), then a KeyboardInterrupt
propagates from run_until_complete() hitting the except block which tries
to call ApiDependencies.shutdown() a second time.

Change both stop() methods to return silently (instead of raising) when
the service is not running. This handles:
- Double-shutdown: lifespan already stopped the services
- Early interrupt: services were never fully started

Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

Fix shutdown hang on session processor thread lock (#108)

* Initial plan

* Fix shutdown hang: wake session processor thread on stop() and mark daemon

Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

* Fix: shut down asyncio executor on KeyboardInterrupt to prevent post-generation hang (#112)

Fix: cancel pending asyncio tasks before loop.close() to suppress destroyed-task warnings
Fix: suppress stack trace when dispatching events after event loop is closed on shutdown
Fix: cancel in-progress generation on stop() to prevent core dump during mid-flight Ctrl+C

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>

---------

Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants