Limit --maxprocesses for scripts/ci/run-tests when in GitHub runners#10148
Limit --maxprocesses for scripts/ci/run-tests when in GitHub runners#10148
Conversation
There was a problem hiding this comment.
This is a good simple fix for the problem!
Not against this change, but since we have pytest-xdist available, we could also use the methods suggested by that package to set the definition of -n auto, like through the environment variable PYTEST_XDIST_AUTO_NUM_WORKERS:
https://pytest-xdist.readthedocs.io/en/stable/distribution.html#running-tests-across-multiple-cpus
I suggest this because when running tests locally I sometimes do the opposite and increase max processes manually. This could affect folks running the tests locally if we dropped the default --maxprocesses to allow the env variable to work, though.
In f7f227e, we now check if |
kdaily
left a comment
There was a problem hiding this comment.
Left a comment, but since the proposed change has the same effect and our test suite is failing extremely often now, I'm fine to move forward.
| if os.getenv('PYTEST_XDIST_AUTO_NUM_WORKERS'): | ||
| # If the user set PYTEST_XDIST_AUTO_NUM_WORKERS, | ||
| # use that to limit max processes | ||
| max_processes = int(os.environ['PYTEST_XDIST_AUTO_NUM_WORKERS']) | ||
| elif os.getenv('GITHUB_ACTIONS', 'false') == 'true': | ||
| # Limiting parallelism in GitHub, to avoid the | ||
| # 'Error: The operation was canceled.' issues in 2026 | ||
| max_processes=3 | ||
| else: | ||
| max_processes=4 | ||
| test_args += f'--numprocesses=auto --dist=loadfile --maxprocesses={max_processes} ' |
There was a problem hiding this comment.
I was thinking that --max-processes would no longer be needed if PYTEST_XDIST_AUTO_NUM_WORKERS was set and the fallback is needed for backwards compatibility if its not set:
| if os.getenv('PYTEST_XDIST_AUTO_NUM_WORKERS'): | |
| # If the user set PYTEST_XDIST_AUTO_NUM_WORKERS, | |
| # use that to limit max processes | |
| max_processes = int(os.environ['PYTEST_XDIST_AUTO_NUM_WORKERS']) | |
| elif os.getenv('GITHUB_ACTIONS', 'false') == 'true': | |
| # Limiting parallelism in GitHub, to avoid the | |
| # 'Error: The operation was canceled.' issues in 2026 | |
| max_processes=3 | |
| else: | |
| max_processes=4 | |
| test_args += f'--numprocesses=auto --dist=loadfile --maxprocesses={max_processes} ' | |
| test_args += f'--numprocesses=auto --dist=loadfile ' | |
| # If the user set PYTEST_XDIST_AUTO_NUM_WORKERS, | |
| # let pytest use that to limit max processes | |
| # Otherwise, fall back to 4 since that's what was | |
| # previously used in this script. | |
| if not os.getenv('PYTEST_XDIST_AUTO_NUM_WORKERS'): | |
| test_args += f'--maxprocesses=4 ' |
Then it would be specifically set to 3 in the GitHub Actions yaml configurations that run the test suites.
There was a problem hiding this comment.
From discussion, planning to merge this for now since tests are really bad yesterday, we can keep tweaking if there are users setting PYTEST_XDIST_AUTO_NUM_WORKERS.
Issue #, if available: N/A
Description of changes:
Error: The operation was canceled.we've been seeing recently on the GitHub runners. The operation was canceled. actions/runner#2468 and other threads about it suggest it may be due to excessive resource consumption.By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.