Race in `inner_timout_checker`: iterating over `m.start_time_by_pid.items()` can crash with "dictionary changed size during iteration" and leave `mutmut run` hanging

[pomponchik]

Command

python3.10 -m mutmut run

Observed behavior

During mutation testing, mutmut prints this exception from the timeout-checker background thread:

Exception in thread Thread-1061 (inner_timout_checker):
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/pomponchik/Desktop/Projects/suby/.venv/lib/python3.8/site-packages/mutmut/__main__.py", line 1201, in inner_timout_checker
    for pid, start_time in m.start_time_by_pid.items():
RuntimeError: dictionary changed size during iteration

After that, mutmut run appears to hang indefinitely.

I checked the process tree, and the parent python -m mutmut run process plus multiple worker python -m mutmut run children were still alive, but all of them were sleeping at 0.0% CPU. So it looks like once inner_timout_checker crashes, the main orchestration can get stuck forever instead of failing and cleaning up workers.

Expected behavior

inner_timout_checker should not iterate over a dict that can be concurrently mutated without synchronization/copying. And if a background orchestration thread crashes anyway, mutmut run should fail loudly and terminate workers instead of hanging.

Why this looks like a race

The failing line iterates directly over:

for pid, start_time in m.start_time_by_pid.items():

The exception strongly suggests that some other thread mutates m.start_time_by_pid at the same time.

So a minimal fix might be to iterate over a snapshot, e.g. list(m.start_time_by_pid.items()), or guard access with a lock, depending on the intended synchronization model.

Extra note

I first tried running mutmut from a Python 3.8 venv and hit a separate issue:

AttributeError: module 'os' has no attribute 'waitstatus_to_exitcode'

So the hang above was reproduced by running mutmut itself with Python 3.10.8.

Environment

• mutmut: 3.2.3
• OS: macOS 15.3.1 (24D70)
• Python used to run mutmut: 3.10.8
• Test runner: pytest
• Project size: ~245 mutants

[Otto-AA]

Hi, thanks for reporting the issue. However, I think this is already fixed in a newer version (see #410). Can you try it with a more recent version?

I first tried running mutmut from a Python 3.8 venv and hit a separate issue:
AttributeError: module 'os' has no attribute 'waitstatus_to_exitcode'

This method was introduced in Python 3.9, so this makes sense. We only run CI tests with current python versions (3.10 -> 3.14 currently; https://devguide.python.org/versions/).

And if a background orchestration thread crashes anyway, mutmut run should fail loudly and terminate workers instead of hanging.

Would be nice to have, but not something I can take a look at currently.