Skip to content

fix(perf): avoid repeated scan of entire venv via packages_distributions() at import time#16579

Merged
daniel-sanche merged 1 commit intogoogleapis:mainfrom
bonauer-pf:fix/avoid-packages-distributions-scan
Apr 8, 2026
Merged

fix(perf): avoid repeated scan of entire venv via packages_distributions() at import time#16579
daniel-sanche merged 1 commit intogoogleapis:mainfrom
bonauer-pf:fix/avoid-packages-distributions-scan

Conversation

@bonauer-pf
Copy link
Copy Markdown
Contributor

@bonauer-pf bonauer-pf commented Apr 8, 2026 •
edited
Loading

packages_distributions() scans every installed package in the environment to build a complete module-to-distribution mapping. In large venvs (500+ packages, common with many google-cloud-* libs), this causes multi-second import delays for google.api_core and every library that depends on it.

This PR contains 2 changes:

  • Wrap packages_distributions() with functools.cache so the expensive O(n) scan happens at most once per process.
  • Defer the package label resolution in check_python_version() so it only runs when a warning is actually emitted, not on the common happy path of a supported Python version.

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

  • Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
  • Ensure the tests and linter pass
  • Code coverage does not decrease (if any source code was changed)
  • Appropriate docs were updated (if necessary)

Fixes #15015 and #16552.

@google-cla
Copy link
Copy Markdown

google-cla bot commented Apr 8, 2026

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

gemini-code-assist[bot]
gemini-code-assist bot reviewed Apr 8, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request optimizes the _get_pypi_package_name function by replacing the slow packages_distributions() scan with targeted metadata.distribution() lookups. It also introduces lazy resolution for package labels to improve performance on supported Python versions. However, the top-level import of importlib.metadata breaks compatibility with Python 3.7. Feedback was provided to handle this as an optional import and to ensure _get_pypi_package_name handles cases where the metadata module is unavailable.

@bonauer-pf
perf: cache packages_distributions() and defer call to unhappy path
6425506 
  packages_distributions() scans every installed package in the environment to build a complete module-to-distribution mapping. In large venvs (500+ packages, common with many google-cloud-* libs), this causes multi-second import delays for google.api_core and every library that depends on it.

  Two changes:
  - Wrap packages_distributions() with functools.cache so the expensive O(n) scan happens at most once per process.
  - Defer the package label resolution in check_python_version() so it only runs when a warning is actually emitted, not on the common happy path of a supported Python version.
@bonauer-pf bonauer-pf force-pushed the fix/avoid-packages-distributions-scan branch from e63555e to 6425506 Compare April 8, 2026 10:49
@bonauer-pf bonauer-pf changed the title perf: avoid full venv scan via packages_distributions() at import time perf: avoid repeated scan of entire venv via packages_distributions() at import time Apr 8, 2026
@bonauer-pf bonauer-pf marked this pull request as ready for review April 8, 2026 14:03
@bonauer-pf bonauer-pf requested review from a team as code owners April 8, 2026 14:03
@bonauer-pf bonauer-pf mentioned this pull request Apr 8, 2026
Closed
@vchudnov-g
Copy link
Copy Markdown
Contributor

vchudnov-g commented Apr 8, 2026

Thanks for your PR! It looks very sensible. We'll approve once it passes the presubmits.

@vchudnov-g vchudnov-g mentioned this pull request Apr 8, 2026
Closed
1 task
@vchudnov-g vchudnov-g added kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests. labels Apr 8, 2026
@daniel-sanche daniel-sanche changed the title perf: avoid repeated scan of entire venv via packages_distributions() at import time fix(perf): avoid repeated scan of entire venv via packages_distributions() at import time Apr 8, 2026
daniel-sanche
daniel-sanche approved these changes Apr 8, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@daniel-sanche daniel-sanche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for putting this together!

@daniel-sanche daniel-sanche merged commit c5728b2 into googleapis:main Apr 8, 2026
31 of 32 checks passed
This was referenced Apr 9, 2026
Closed
Closed
Merged
daniel-sanche pushed a commit that referenced this pull request Apr 9, 2026
@cloud-sdk-librarian-robot
chore: librarian release pull request: 20260409T210905Z (#16604)
d3d6840 
PR created by the Librarian CLI to initialize a release. Merging this PR
will auto trigger a release.

Librarian Version: v0.0.0-20260407211006-7cd74ea13ba9
Language Image:
us-central1-docker.pkg.dev/cloud-sdk-librarian-prod/images-prod/python-librarian-generator@sha256:234b9d1f2ddb057ed7ac6a38db0bf8163d839c65c6cf88ade52530cddebce59e
<details><summary>bigquery-magics: v0.13.0</summary>

##
[v0.13.0](bigquery-magics-v0.12.2...bigquery-magics-v0.13.0)
(2026-04-09)

### Features

* drop support for Python 3.9 (#16587)
([961eacd](961eacd0))

* deprecate `--engine=bigframes`, run `%load_ext bigframes` and use
`%%bqsql` magics instead (#16573)
([dbaf8f5](dbaf8f54))

</details>


<details><summary>google-api-core: v2.30.3</summary>

##
[v2.30.3](google-api-core-v2.30.2...google-api-core-v2.30.3)
(2026-04-09)

### Bug Fixes

* avoid repeated scan of entire venv via packages_distributions() at
import time (#16579)
([c5728b2](c5728b24))

</details>


<details><summary>google-apps-chat: v0.8.0</summary>

##
[v0.8.0](google-apps-chat-v0.7.0...google-apps-chat-v0.8.0)
(2026-04-09)

### Features

* Addition of Section and SectionItem APIs (PiperOrigin-RevId:
893488928)
([56ccbd8](56ccbd86))

* Support app authentication with admin-consent scopes for Chat API
ListMessages, GetMessage, GetSpaceEvent and ListSpaceEvents
(PiperOrigin-RevId: 893619899)
([56ccbd8](56ccbd86))

### Documentation

* Update reference documentation for Chat API ListMessages, GetMessage,
GetSpaceEvent and ListSpaceEvents. (PiperOrigin-RevId: 893619899)
([56ccbd8](56ccbd86))

</details>


<details><summary>google-auth: v2.49.2</summary>

##
[v2.49.2](google-auth-v2.49.1...google-auth-v2.49.2)
(2026-04-09)

### Bug Fixes

* use requests transport for GCE MDS (#16480)
([614a3d0](614a3d04))

</details>


<details><summary>google-cloud-appoptimize: v0.1.0</summary>

##
[v0.1.0](google-cloud-appoptimize-v0.0.0...google-cloud-appoptimize-v0.1.0)
(2026-04-09)

### Features

* onboard a new library (PiperOrigin-RevId: 895558469)
([8e92bae](8e92bae8))

</details>


<details><summary>google-cloud-dlp: v3.36.0</summary>

##
[v3.36.0](google-cloud-dlp-v3.35.0...google-cloud-dlp-v3.36.0)
(2026-04-09)

### Features

* added support for detecting key-value pairs in client provided
metadata (PiperOrigin-RevId: 895098649)
([56ccbd8](56ccbd86))

</details>


<details><summary>google-cloud-iamconnectorcredentials: v0.1.0</summary>

##
[v0.1.0](google-cloud-iamconnectorcredentials-v0.0.0...google-cloud-iamconnectorcredentials-v0.1.0)
(2026-04-09)

### Features

* onboard a new library (PiperOrigin-RevId: 889980347)
([2a3458c](2a3458c8))

</details>


<details><summary>google-cloud-pubsub: v2.37.0</summary>

##
[v2.37.0](google-cloud-pubsub-v2.36.0...google-cloud-pubsub-v2.37.0)
(2026-04-09)

### Documentation

* Fix documentation URL AIInference MessageTransform
service_account_email field (PiperOrigin-RevId: 895485801)
([56ccbd8](56ccbd86))

</details>


<details><summary>google-cloud-translate: v3.26.0</summary>

##
[v3.26.0](google-cloud-translate-v3.25.0...google-cloud-translate-v3.26.0)
(2026-04-09)

### Features

* A new field `mime_type` is added to message
`.google.cloud.translation.v3.AdaptiveMtTranslateRequest`
(PiperOrigin-RevId: 895422613)
([56ccbd8](56ccbd86))

</details>


<details><summary>pandas-gbq: v0.35.0</summary>

##
[v0.35.0](pandas-gbq-v0.34.1...pandas-gbq-v0.35.0)
(2026-04-09)

### Features

* drop support for Python 3.9 (#16476)
([f38cd44](f38cd44b))

</details>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@daniel-sanche daniel-sanche daniel-sanche approved these changes

@vchudnov-g vchudnov-g vchudnov-g approved these changes

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

kokoro:force-run Add this label to force Kokoro to re-run the tests. kokoro:run Add this label to force Kokoro to re-run the tests.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Import time is very slow (>5s)

3 participants

@bonauer-pf @vchudnov-g @daniel-sanche