bpo-39190: Fix deadlock when callback raises

[sindrig]

• Make sure we always clear up from result._cache
• Make sure we catch exceptions when setting values in cache

https://bugs.python.org/issue39190

[the-knights-who-say-ni]

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

CLA Missing

Our records indicate the following people have not signed the CLA:

@sindrig

For legal reasons we need all the people listed to sign the CLA before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

[sindrig]

I've signed the CLA, should I do something more?

[sindrig]

Any chance of getting a review?

[MaxwellDupre]

All the multiprocessing tests passed but the given example shows:
!  ~/D/G/cpython   bpo-39190-fix-pool-callback-error $…  ./python ~/tmp/test_pool_error.py
value 1
Traceback (most recent call last):
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 592, in _handle_results
cache[job]._set(i, obj)
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 783, in _set
self._callback(self._value)
File "/home/me/PycharmProjects/TestingPy/test_pool_error.py", line 20, in call
raise ValueError('callback raises')
ValueError: callback raises
value 1
Traceback (most recent call last):
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 592, in _handle_results
cache[job]._set(i, obj)
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 783, in _set
self._callback(self._value)
File "/home/me/PycharmProjects/TestingPy/test_pool_error.py", line 20, in call
raise ValueError('callback raises')
ValueError: callback raises
value 1
Traceback (most recent call last):
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 592, in _handle_results
cache[job]._set(i, obj)
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 783, in _set
self._callback(self._value)
File "/home/me/PycharmProjects/TestingPy/test_pool_error.py", line 20, in call
raise ValueError('callback raises')
ValueError: callback raises
value 1
Traceback (most recent call last):
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 592, in _handle_results
cache[job]._set(i, obj)
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 783, in _set
self._callback(self._value)
File "/home/me/PycharmProjects/TestingPy/test_pool_error.py", line 20, in call
raise ValueError('callback raises')
ValueError: callback raises
value 1
Traceback (most recent call last):
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 592, in _handle_results
cache[job]._set(i, obj)
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 783, in _set
self._callback(self._value)
File "/home/me/PycharmProjects/TestingPy/test_pool_error.py", line 20, in call
raise ValueError('callback raises')
ValueError: callback raises
Pool <class 'multiprocessing.pool.ThreadPool'> joined
value 1
Traceback (most recent call last):
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 592, in _handle_results
cache[job]._set(i, obj)
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 783, in _set
self._callback(self._value)
File "/home/me/PycharmProjects/TestingPy/test_pool_error.py", line 20, in call
raise ValueError('callback raises')
ValueError: callback raises
value 1
Traceback (most recent call last):
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 592, in _handle_results
cache[job]._set(i, obj)
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 783, in _set
self._callback(self._value)
File "/home/me/PycharmProjects/TestingPy/test_pool_error.py", line 20, in call
raise ValueError('callback raises')
ValueError: callback raises
value 1
Traceback (most recent call last):
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 592, in _handle_results
cache[job]._set(i, obj)
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 783, in _set
self._callback(self._value)
File "/home/me/PycharmProjects/TestingPy/test_pool_error.py", line 20, in call
raise ValueError('callback raises')
ValueError: callback raises
value 1
Traceback (most recent call last):
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 592, in _handle_results
cache[job]._set(i, obj)
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 783, in _set
self._callback(self._value)
File "/home/me/PycharmProjects/TestingPy/test_pool_error.py", line 20, in call
raise ValueError('callback raises')
ValueError: callback raises
value 1
Traceback (most recent call last):
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 592, in _handle_results
cache[job]._set(i, obj)
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 783, in _set
self._callback(self._value)
File "/home/me/PycharmProjects/TestingPy/test_pool_error.py", line 20, in call
raise ValueError('callback raises')
ValueError: callback raises
Pool <class 'multiprocessing.pool.Pool'> joined
value [1, 1, 1, 1, 1]
Traceback (most recent call last):
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 592, in _handle_results
cache[job]._set(i, obj)
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 822, in _set
self._callback(self._value)
File "/home/me/PycharmProjects/TestingPy/test_pool_error.py", line 20, in call
raise ValueError('callback raises')
ValueError: callback raises
Pool <class 'multiprocessing.pool.ThreadPool'> joined
value [1, 1, 1, 1, 1]
Traceback (most recent call last):
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 592, in _handle_results
cache[job]._set(i, obj)
File "/home/me/Documents/GitHub/cpython/Lib/multiprocessing/pool.py", line 822, in _set
self._callback(self._value)
File "/home/me/PycharmProjects/TestingPy/test_pool_error.py", line 20, in call
raise ValueError('callback raises')
ValueError: callback raises

Doesn't look so different..

[sindrig]

Have you tried running the example on python:main?

[github-actions]

This PR is stale because it has been open for 30 days with no activity.