Skip to content

Send serialized ASTs to parallel workers#20991

Merged
ilevkivskyi merged 4 commits intopython:masterfrom
ilevkivskyi:parallel-no-parse
Mar 13, 2026
Merged

Send serialized ASTs to parallel workers#20991
ilevkivskyi merged 4 commits intopython:masterfrom
ilevkivskyi:parallel-no-parse

Conversation

@ilevkivskyi
Copy link
Copy Markdown
Member

@ilevkivskyi ilevkivskyi commented Mar 8, 2026

This way we can properly benefit from native parser in parallel type checking. Self-check is now ~2.2x faster with 5 workers compared to in-process checking (also with native parser). Also it uses less memory, but still with 5 workers, self-check takes ~twice more memory compared to in-process.

Implementation is mostly straightforward. The GC freeze hack needed some tuning, as there is no single hot-spot in terms of allocations anymore.

Note: do not use maturin develop for any performance measurements, as this creates some very slow wheel.

@github-actions

This comment has been minimized.

@ilevkivskyi
Copy link
Copy Markdown
Member Author

Ping on this one. I want to continue with other improvements to parallel checking that depends on this PR.

@JukkaL
Copy link
Copy Markdown
Collaborator

JukkaL commented Mar 12, 2026

Sorry, was busy today, but I should be able to test this with our big internal codebase tomorrow. Overall looks good, I just want to try this out a little bit.

@github-actions
Copy link
Copy Markdown
Contributor

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

Copy link
Copy Markdown
Collaborator

@JukkaL JukkaL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't have time to try this out with the internal repo, but I can do it after this has been merged. Approving to unblock follow-up work. Anyway, great results so far -- I think this already makes parallel type checking quite useful, even if we can continue to improve scalability further!

@ilevkivskyi ilevkivskyi merged commit 69a0925 into python:master Mar 13, 2026
40 of 41 checks passed
@ilevkivskyi ilevkivskyi deleted the parallel-no-parse branch March 13, 2026 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants