Skip to content

Cache pkgutil namespace probes during import resolution#4017

Open
alexander-beedie wants to merge 1 commit into
facebook:mainfrom
alexander-beedie:perf-cache-pkgutil-namespace
Open

Cache pkgutil namespace probes during import resolution#4017
alexander-beedie wants to merge 1 commit into
facebook:mainfrom
alexander-beedie:perf-cache-pkgutil-namespace

Conversation

@alexander-beedie

@alexander-beedie alexander-beedie commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Resolving a module and its submodules walks each path component, and every intermediate package's __init__.py is probed with is_pkgutil_namespace.

Crucially the same __init__ files are re-opened/re-read once per resolved submodule, leading to a lot of redundant calls (each of which is a File::open + 4KiB read + a regex to check for pkgutil.extend_path).

Solution

Memoise the result in a new pkgutil_cache on DirEntryCache, keyed by __init__ path, sharing the same "per-loader, never-invalidated" lifetime as the existing entry cache. The existing cache name was updated to entry_cache to help differentiate/clarify now there are two caches.

Impact

Reduces ~14,900 reads to ~1,150 when testing against the core package.
(One per unique __init__ reached, for a ~92% reduction).

Didn't see a meaningful performance impact on macOS (cached reads are cheap?). I'd expect a more noticeable positive change on Windows though.

This is additive to #3994, with respect to #3993.

Test Plan

No type-inference logic changes; tests return identical before/after test results.

@meta-cla meta-cla Bot added the cla signed label Jul 2, 2026
@github-actions github-actions Bot added the size/s label Jul 2, 2026
@github-actions

This comment has been minimized.

@alexander-beedie alexander-beedie force-pushed the perf-cache-pkgutil-namespace branch from 4b4b0fc to 56651d1 Compare July 2, 2026 10:27
@github-actions github-actions Bot added size/s and removed size/s labels Jul 2, 2026
@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

@yangdanny97

Copy link
Copy Markdown
Contributor

I can try to find someone with a windows machine to do some benchmarking

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants