Skip to content

Limit --maxprocesses for scripts/ci/run-tests when in GitHub runners#10148

Merged
ashovlin merged 4 commits intoaws:v2from
ashovlin:limit-test-parallelism
Mar 26, 2026
Merged

Limit --maxprocesses for scripts/ci/run-tests when in GitHub runners#10148
ashovlin merged 4 commits intoaws:v2from
ashovlin:limit-test-parallelism

Conversation

@ashovlin
Copy link
Copy Markdown
Member

@ashovlin ashovlin commented Mar 23, 2026

Issue #, if available: N/A

Description of changes:

  • Reduce parallelization for unit/functional tests in GItHub to avoid the Error: The operation was canceled. we've been seeing recently on the GitHub runners. The operation was canceled. actions/runner#2468 and other threads about it suggest it may be due to excessive resource consumption.
  • Updated some actions pins while I was here to avoid Node deprecation warnings.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@ashovlin ashovlin marked this pull request as ready for review March 23, 2026 19:04
Copy link
Copy Markdown
Member

@kdaily kdaily left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good simple fix for the problem!

Not against this change, but since we have pytest-xdist available, we could also use the methods suggested by that package to set the definition of -n auto, like through the environment variable PYTEST_XDIST_AUTO_NUM_WORKERS:

https://pytest-xdist.readthedocs.io/en/stable/distribution.html#running-tests-across-multiple-cpus

I suggest this because when running tests locally I sometimes do the opposite and increase max processes manually. This could affect folks running the tests locally if we dropped the default --maxprocesses to allow the env variable to work, though.

@ashovlin
Copy link
Copy Markdown
Member Author

ashovlin commented Mar 25, 2026

PYTEST_XDIST_AUTO_NUM_WORKERS

In f7f227e, we now check if PYTEST_XDIST_AUTO_NUM_WORKERS is set first. If it is, we use that to set --maxprocesses to match the user's desired number. This will let users override our defaults without modifying the script.

@ashovlin ashovlin requested a review from kdaily March 25, 2026 16:36
Copy link
Copy Markdown
Member

@kdaily kdaily left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a comment, but since the proposed change has the same effect and our test suite is failing extremely often now, I'm fine to move forward.

Comment thread scripts/ci/run-tests
Comment on lines +55 to +65
if os.getenv('PYTEST_XDIST_AUTO_NUM_WORKERS'):
# If the user set PYTEST_XDIST_AUTO_NUM_WORKERS,
# use that to limit max processes
max_processes = int(os.environ['PYTEST_XDIST_AUTO_NUM_WORKERS'])
elif os.getenv('GITHUB_ACTIONS', 'false') == 'true':
# Limiting parallelism in GitHub, to avoid the
# 'Error: The operation was canceled.' issues in 2026
max_processes=3
else:
max_processes=4
test_args += f'--numprocesses=auto --dist=loadfile --maxprocesses={max_processes} '
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking that --max-processes would no longer be needed if PYTEST_XDIST_AUTO_NUM_WORKERS was set and the fallback is needed for backwards compatibility if its not set:

Suggested change
if os.getenv('PYTEST_XDIST_AUTO_NUM_WORKERS'):
# If the user set PYTEST_XDIST_AUTO_NUM_WORKERS,
# use that to limit max processes
max_processes = int(os.environ['PYTEST_XDIST_AUTO_NUM_WORKERS'])
elif os.getenv('GITHUB_ACTIONS', 'false') == 'true':
# Limiting parallelism in GitHub, to avoid the
# 'Error: The operation was canceled.' issues in 2026
max_processes=3
else:
max_processes=4
test_args += f'--numprocesses=auto --dist=loadfile --maxprocesses={max_processes} '
test_args += f'--numprocesses=auto --dist=loadfile '
# If the user set PYTEST_XDIST_AUTO_NUM_WORKERS,
# let pytest use that to limit max processes
# Otherwise, fall back to 4 since that's what was
# previously used in this script.
if not os.getenv('PYTEST_XDIST_AUTO_NUM_WORKERS'):
test_args += f'--maxprocesses=4 '

Then it would be specifically set to 3 in the GitHub Actions yaml configurations that run the test suites.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From discussion, planning to merge this for now since tests are really bad yesterday, we can keep tweaking if there are users setting PYTEST_XDIST_AUTO_NUM_WORKERS.

@ashovlin ashovlin merged commit c02348d into aws:v2 Mar 26, 2026
72 of 73 checks passed
@ashovlin ashovlin deleted the limit-test-parallelism branch March 26, 2026 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants