Maintenance/code cleanup 2026-05 Phase VI (pure python resolver)#6669
Open
matteius wants to merge 64 commits into
Open
Conversation
…lding Spins up Phase 3 of Initiative G — the in-tree pure-Python ``resolvelib.Provider`` backend that replaces pip's ``PackageFinder`` + ``LinkEvaluator`` + ``InstallRequirement`` machinery with our own ``Requirement`` + ``MetadataFetcher`` + ``PurePythonProvider`` chain. Branched off ``maintenance/code-cleanup-phase5-perf-2026-06`` at commit ``3d16ca04`` (the post-Initiative-G-Phases-1-2 + ``resolve_constraints`` perf-cut tip). What lands here --------------- - ``docs/dev/initiative-g-phase3-design.md`` — focused design doc (260 lines). Picks up where the umbrella ``initiative-g-pure-python-design.md`` §5.4 / §7.4 left off. Open questions Q-A through Q-D explicitly catalogued for maintainer sign-off; default recommendations baked into the plan. - ``initiative-g-phase3-plan.md`` — 17-task swarm-ready plan modelled on the Phase 1+2 plan. Dependency graph; per-task ``depends_on`` / ``validation`` / ``status`` / ``log``. Parallel execution waves (12 waves, max concurrency 3). Risks table enumerates the lockfile-parity, sdist-fallback, ``get_preference`` mirror, and 30 % perf-gate concerns. - Three module scaffolds with rich docstrings naming the implementation task and the design-doc section that motivates them: * ``pipenv/resolver/pure_python_requirement.py`` (T1). * ``pipenv/resolver/pure_python_metadata.py`` (T2). * ``pipenv/resolver/backends/pure_python.py`` (T9 / T10). ``pure_python_provider.py`` is created lazily by T3 — keeping the module empty here would invite a half-finished class structure to be carried in from scratch. What's NOT here (per plan §"Out of Scope") ------------------------------------------ - sdist resolution (Q-A — fall back to pip backend). - Removing the pip backend (Phase 4). - HTTP/2 transport (separate effort). - Replacing ``resolvelib`` itself. - Keyring auth (Q-D — deferred). Phase-3 acceptance gates (design §8) ------------------------------------ - Zero ``pip._internal.*`` imports in the new code (enforced by Phase 1's pre-commit grep gate). - Lockfile byte-identity vs pip backend across the 100-pkg bench + 10 real-world projects (T_PARITY_REAL) + pip versions N-1 / N / N+1 (T_MATRIX). - CI ``lock-warm`` ≤ 14.5 s on the 100-pkg bench (≥ 30 % off the pre-phase-5 baseline of 21.3 s). - Parity matrix doc shipped with every divergence justified. - News fragment + ``docs/pipfile.md`` entry for the new selector. No code change in any module under ``pipenv/`` other than the three new scaffolds. Phase 1+2's modules are unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six previously-open design questions are now answered with maintainer sign-off 2026-05-12. The plan no longer treats them as "open"; T9 and T14 are rewritten to reflect the load-bearing decisions: - Q-A: fail loud on sdist-only candidates (no transparent fallback). - Q-B: pre-fetch PEP 658 metadata for top-level packages only. - Q-C: strict mirror of pip's get_preference with byte-identity gate. - Q-D: no keyring auth in Phase 3. - Q-E: T_PARITY_REAL exercises 10 wheel-heavy mainstream projects. - Q-F: backend-startup wheel-availability pre-check on top-level packages. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…for pure-python backend (Initiative G phase 3)
Implements ``pipenv/resolver/pure_python_metadata.py`` per design
§5.2 — the wheel-METADATA fetcher that T7's
``PurePythonProvider.get_dependencies`` will consume.
Components:
* ``CoreMetadata`` — frozen + slotted dataclass carrying ``name``,
``version``, ``requires_python``, ``requires_dist``,
``provides_extras``, ``summary``.
* ``MetadataCache`` — on-disk cache keyed by ``sha256(wheel_url)``;
same temp-file + ``os.replace`` atomic-write contract as
``ParsedManifestCache``. Cache entries are valid forever
(wheels are immutable).
* ``fetch_metadata(candidate, session, *, cache, metadata_url,
metadata_hash)`` — read-through cache + two-tier fetch:
1. PEP 658 fast path when ``metadata_url`` is advertised; the
body's ``sha256`` is verified against ``metadata_hash``.
Mismatch raises ``MetadataFetchError``.
2. Wheel-head fallback otherwise: ``HEAD`` for length (probing
``GET Range: bytes=0-1`` on 405/403), range-GET the last
64 kB for the zip central directory, locate the
``<dist-info>/METADATA`` entry, range-GET its bytes,
decompress, parse.
* ``_PartialFile`` — ``io.RawIOBase`` shim that lets
``zipfile.ZipFile`` see a seekable view over the wheel; the
shim transparently re-issues HTTP range GETs for bytes outside
the in-memory window, so neither the central-directory walk
nor the METADATA extraction has to buffer the full wheel.
* ``_parse_metadata_text`` — stdlib ``email.parser.HeaderParser``
with ``email.policy.compat32``; collects every ``Requires-Dist``
header in source order and ``Provides-Extra`` into a frozenset.
Tests at ``tests/unit/test_pure_python_metadata.py`` cover the four
plan-T2 acceptance gates:
* PEP 658 fast path returns parsed metadata.
* Wheel-head fallback (synthetic wheel via ``tmp_path`` +
``zipfile``) returns parsed metadata.
* Wheel-head with HEAD 405 falls back to probing GET.
* Cache round-trip: second fetch is served from disk (no network).
* PEP 658 hash mismatch raises ``MetadataFetchError``.
* ``MetadataCache.get`` returns ``None`` on miss; ``put`` → ``get``
round-trip.
* ``_parse_metadata_text`` minimum-fields + repeated-headers.
Constraint compliance (Initiative G acceptance criterion):
``grep -nE "^[[:space:]]*(from|import)[[:space:]]+pipenv\.patched\.pip\._internal"
pipenv/resolver/pure_python_metadata.py`` matches nothing. No new
third-party deps; only stdlib (``hashlib``, ``io``, ``json``,
``logging``, ``os``, ``tempfile``, ``zipfile``, ``email``,
``pathlib``) + ``pipenv.vendor.packaging``.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nitiative G phase 3) Frozen `@dataclass(frozen=True, slots=True)` per design §5.1, replacing pip's `InstallRequirement` as the resolution-graph constraint node for the in-tree `resolvelib.Provider` path. Fields: `name` (PEP 503 canonical), `specifier` (`SpecifierSet`), `extras` (`frozenset[str]`), `marker` (`Marker | None`), `source` (Literal pipfile/transitive/ constraint), `parent` (`str | None`). Hashability comes for free from the frozen dataclass — both `SpecifierSet` and `Marker` are hashable in `pipenv.vendor.packaging`, so `frozenset[Requirement]` works out of the box (verified by the T1 test suite). The `from_pipfile_entry` classmethod handles the canonical Pipfile shapes (bare string version, "*", dict with version/extras/markers, "version": "*"). Names canonicalised via `pipenv.vendor.packaging.utils.canonicalize_name` (PEP 503). Zero `pip._internal.*` imports (enforced by Phase 1's pre-commit gate). RED→GREEN: `tests/unit/test_pure_python_requirement.py` (15 tests). T11 will extend this file with the broader coverage matrix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ve G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…Initiative G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…iative G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…rror, Initiative G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…untered (Q-A fail-loud, Initiative G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…solver helper (Initiative G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nitiative G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…eck (Initiative G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…(Initiative G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…itiative G phase 3) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…from ResolverRequest (Initiative G phase 3)
Closes the gap T10 introduced when defaulting __init__ collaborators
to None: the backend now constructs cache/fetcher/session/metadata_cache
from the request envelope when not injected, so registry-path
`get_backend("pure-python").resolve(request)` works end-to-end.
Unit-test kwarg injection still wins.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nd dispatcher (Initiative G phase 3) Adds the user-facing dispatcher for Initiative G Phase 3: - `pipenv lock --backend [pip|pure-python]` selects backend per-invocation - `[pipenv] resolver_backend` Pipfile setting selects default - Resolver subprocess routes the typed ResolverRequest through `get_backend(name).resolve(request)`; default stays "pip" so the no-flag-no-setting path is byte-identical to today. Unblocks T15 + T_PARITY_REAL + T_BENCH to run against the production CLI surface rather than in-process. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…(Initiative G phase 3) PEP691Client._dispatch_response and pure_python_metadata's _http_get / _http_get_range / _discover_wheel_length all assumed urllib3-style responses (.status / .data). In production the bootstrap path threads a PipSession (a requests.Session subclass) whose responses expose .status_code / .content. The existing prefetch path in routines/lock.py swallowed the resulting AttributeError under try/except (best-effort behaviour), but the pure-python backend's hot path surfaced the bug as "populate returned FetchError(transient, 'Response' object has no attribute status')" — every find_matches call came back empty, so `pipenv lock --backend pure-python` always raised ResolutionImpossible. Helpers now pick attribute by presence (hasattr), not by value, so a urllib3-style mock with `data=None` doesn't get re-interpreted as a requests response. `pipenv lock --backend pure-python` on a single-package fixture (click=*) now produces a valid Pipfile.lock; pip and pure-python agree on _meta.hash and the resolved version, with three known remaining divergences (sdist hashes, index URL vs name, marker propagation) tracked for T_PARITY_MATRIX. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 3a (a5ac1ef..8b24ace) shipped a working pure-python backend but punted on sdists (Q-A fail-loud), marker propagation, and the small index-name + multi-distfile-hashes parity gaps surfaced by the smoke test against `click`. Phase 3b reframes the acceptance gate from "lockfile byte-identity on the 100-pkg bench" to "the existing pipenv test suite passes under PIPENV_RESOLVER_BACKEND=pure-python". A non-blocking CI matrix job surfaces divergences as they appear; sdist METADATA extraction lands via direct PEP 517 invocation through the already-vendored `pyproject_hooks`; markers propagate both from Requires-Python and from the Requires-Dist line that introduced each transitive. Maintainer-locked decisions (sign-off 2026-05-12): - Sdist build: direct PEP 517 via `pyproject_hooks` (no new vendoring). - Marker propagation: full (Requires-Python + Requires-Dist markers). - CI matrix: add now, `continue-on-error: true`, promote to required in a follow-up. 11 tasks across 3 tracks (markers, sdists, CI hookup); tracks 1+2 parallel-safe in waves 1-5. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…marker propagation (Initiative G phase 3b) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…r (Initiative G phase 3b) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… pure_python_sdist (Initiative G phase 3b) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…into Requirement.introducing_marker (Initiative G phase 3b) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…-Python + introducing_marker (Initiative G phase 3b) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…oduction path (Initiative G phase 3b) Sdists now extract METADATA transparently via T_S1's PEP 517 path routed in T_S2. The Phase 3a fail-loud signal is no longer raised from `get_dependencies` and the corresponding `except` branch in `PurePythonBackend.resolve` is gone. The class itself is preserved as a back-compat import. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
… field (Initiative G phase 3b) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ness check (Initiative G phase 3b) Phase 3a's "no wheel available for top-level" gate is obsolete now that T_S2 builds sdists transparently. Phase 3b's pre-check fires only when a top-level package has ZERO candidates across all configured indexes — typos, yanked-only releases, or network failures that left the cache empty. Sdist-only top-levels resolve normally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…stfile siblings (Initiative G phase 3b) Mirrors pip's lockfile convention: emit hashes for every distfile of the resolved version (wheel + sdist + any cross-platform variants) rather than only the chosen candidate's single hash. Walks the cache for sibling candidates at the same (name, version) and unions their `hashes` frozensets. Falls back to the candidate's own hashes if the cache is empty (preserves test-fixture-injected-only workflows). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…iative G phase 3b) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR expands Pipenv’s opt-in pure-Python resolver path and related install performance work. It adds resolver backend plumbing/tests, sdist metadata extraction via newly vendored PyPA build tooling, parallel wheel prefetching, and Pipfile package-name casing controls.
Changes:
- Adds/extends pure-Python resolver models, backend registration, sdist metadata build support, and backend selection plumbing.
- Adds parallel wheel prefetch support for locked installs and adjusts pip install flags.
- Vendors
build/pyproject_hooks, adds CI dogfood coverage, tests, docs, and news fragments.
Reviewed changes
Copilot reviewed 36 out of 60 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/ci.yaml |
Adds non-blocking pure-python backend test job. |
docs/dev/initiative-g-phase3-design.md |
Adds/updates design details for pure-Python resolver phase 3/3b. |
news/+install-drop-redundant-upgrade-flag.behavior.rst |
Documents dropping redundant --upgrade for sync installs. |
news/+install-parallel-wheel-prefetch.feature.rst |
Documents parallel wheel prefetch feature. |
news/+pipfile-recase-opt-in.behavior.rst |
Documents opt-in Pipfile recasing behavior. |
news/+pure-python-extras-roundtrip.bugfix.rst |
Documents pure-python extras parity fixes. |
news/+pure-python-prerelease-parity.bugfix.rst |
Documents prerelease filtering parity. |
news/+pure-python-sdist-build-isolation.feature.rst |
Documents isolated sdist metadata builds. |
pipenv/cli/options.py |
Adds --backend lock option and backend precedence handling. |
pipenv/resolver/backends/__init__.py |
Registers pure-python backend. |
pipenv/resolver/candidate.py |
Adds extras field to resolver candidates. |
pipenv/resolver/core.py |
Reads resolver_backend Pipfile setting in dispatcher fallback. |
pipenv/resolver/manifest_cache.py |
Adds Windows retry around atomic cache replacement. |
pipenv/resolver/pep691.py |
Allows urllib3- and requests-style response bodies/statuses. |
pipenv/resolver/pure_python_requirement.py |
Adds typed pure-python resolver requirement model. |
pipenv/resolver/pure_python_sdist.py |
Adds sdist download/extract/build metadata path. |
pipenv/routines/install.py |
Wires wheel prefetch into batch install. |
pipenv/routines/lock.py |
Resolves backend precedence before locking. |
pipenv/utils/pip.py |
Makes --upgrade conditional on dependency mode. |
pipenv/utils/pipfile.py |
Adds package-name casing mode normalization and canonical recasing. |
pipenv/utils/prefetch.py |
Adds parallel wheel prefetch implementation. |
pipenv/utils/resolver.py |
Adds backend-aware resolver cache key/request propagation. |
pipenv/utils/settings.py |
Adds Settings.resolver_backend. |
pipenv/vendor/build/LICENSE |
Vendors build license. |
pipenv/vendor/build/__init__.py |
Vendors build package entry point. |
pipenv/vendor/build/__main__.py |
Vendors build CLI module. |
pipenv/vendor/build/_builder.py |
Vendors build project builder. |
pipenv/vendor/build/_compat/__init__.py |
Adds build compat package marker. |
pipenv/vendor/build/_compat/importlib.py |
Vendors build importlib compat helper. |
pipenv/vendor/build/_compat/tarfile.py |
Vendors build tarfile compat helper. |
pipenv/vendor/build/_compat/tomllib.py |
Vendors build TOML compat helper. |
pipenv/vendor/build/_ctx.py |
Vendors build logging/subprocess context helpers. |
pipenv/vendor/build/_exceptions.py |
Vendors build exception classes. |
pipenv/vendor/build/_types.py |
Vendors build type aliases. |
pipenv/vendor/build/_util.py |
Vendors build utility helpers. |
pipenv/vendor/build/env.py |
Vendors isolated build environment support. |
pipenv/vendor/build/py.typed |
Marks vendored build as typed. |
pipenv/vendor/build/util.py |
Vendors build metadata utility API. |
pipenv/vendor/pyproject_hooks/LICENSE |
Vendors pyproject-hooks license. |
pipenv/vendor/pyproject_hooks/__init__.py |
Vendors pyproject-hooks public API. |
pipenv/vendor/pyproject_hooks/_impl.py |
Vendors pyproject-hooks implementation. |
pipenv/vendor/pyproject_hooks/_in_process/__init__.py |
Vendors in-process hook package helper. |
pipenv/vendor/pyproject_hooks/_in_process/_in_process.py |
Vendors hook subprocess runner. |
pipenv/vendor/pyproject_hooks/py.typed |
Marks vendored pyproject-hooks as typed. |
pipenv/vendor/vendor.txt |
Adds vendored build and pyproject-hooks versions. |
tests/unit/test_candidate.py |
Adds Requires-Python preservation tests. |
tests/unit/test_pipfile_subsystem.py |
Adds recase mode tests. |
tests/unit/test_prefetch.py |
Adds wheel prefetch unit tests. |
tests/unit/test_pure_python_provider_smoke.py |
Adds pure-python provider smoke tests. |
tests/unit/test_pure_python_requirement.py |
Adds requirement model tests. |
tests/unit/test_resolver_backends.py |
Adds backend CLI/Pipfile propagation tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…tenance/code-cleanup-phase6-pure-python-resolver-2026-07 # Conflicts: # pipenv/resolver/manifest_cache.py
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/fc71f81d-8096-46e3-8617-a1b2325103c5 Co-authored-by: matteius <479892+matteius@users.noreply.github.com>
…_RESOLVER in cache key - pure_python_sdist.py: reduce archive_name to basename via Path().name before joining with dest_dir, preventing path-traversal if a simple-API response returns a filename containing separators or an absolute path - prefetch.py _download_and_verify: use urllib3.Timeout(connect=..., read=...) instead of a (connect, read) tuple; urllib3.PoolManager does not accept the tuple shape, falling back to tuple only if the import fails - resolver.py _generate_resolution_cache_key: include PIPENV_RESOLVER env var in the effective backend so a cache hit built by the pip backend cannot satisfy a pure-python lookup (and vice versa) when the backend is selected via the environment variable Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/bcacb5b8-63f0-4fda-ba65-72d386c52faa Co-authored-by: matteius <479892+matteius@users.noreply.github.com>
Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/fc71f81d-8096-46e3-8617-a1b2325103c5 Co-authored-by: matteius <479892+matteius@users.noreply.github.com>
- pure_python_sdist.py: use archive_name.strip() to also reject whitespace-only filenames - prefetch.py: rename _Timeout import alias to Urllib3Timeout for clarity - resolver.py: simplify effective_backend chain-or expression Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/bcacb5b8-63f0-4fda-ba65-72d386c52faa Co-authored-by: matteius <479892+matteius@users.noreply.github.com>
…n mock Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/dc1b5386-f7bb-4a1d-ac2a-88a1ccb6d155 Co-authored-by: matteius <479892+matteius@users.noreply.github.com>
…tion, document source limitation Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/072ee90a-9755-49b3-9ecb-61fc16c1ac4f Co-authored-by: matteius <479892+matteius@users.noreply.github.com>
Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/dc1b5386-f7bb-4a1d-ac2a-88a1ccb6d155 Co-authored-by: matteius <479892+matteius@users.noreply.github.com>
Agent-Logs-Url: https://github.com/pypa/pipenv/sessions/072ee90a-9755-49b3-9ecb-61fc16c1ac4f Co-authored-by: matteius <479892+matteius@users.noreply.github.com>
…tenance/code-cleanup-phase6-pure-python-resolver-2026-07
…#6670) GHSA-8xgg-v3jj-95m2 moved credentials off pip's argv onto a merged netrc, but the new ``write_credentials_netrc`` writes our Pipfile-derived machine blocks BEFORE appending the user's existing netrc. Python's ``netrc.authenticators()`` returns the LAST matching entry for a host, so any stale system entry for the same host silently overrode the freshly-expanded creds — exactly the symptom of gh-6670 (env-var creds in ``[[source]]`` fail to authenticate after the 2026.6.1 upgrade). Reverse the order so the user's existing netrc lands first and our blocks win the tie-break. Defensively, the ``pylock.toml`` reader now runs ``expand_url_credentials`` over its sources too, mirroring ``Lockfile.load``; without this, users with ``[pipenv] use_pylock = true`` would see env-var auth silently fall through as literal ``${VAR}`` tokens. Add unit regression tests for both paths. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase VI of the 2026-05 maintenance cleanup series. Ships the
pure-Python resolver backend (Initiative G phase 3 + 3b, end-to-end),
hardens it on the bench-fixture and ten real-world wheel-heavy combos
of T_PARITY_REAL, and lands a parallel wheel pre-fetch that cuts
the cold-cache
pipenv sync/pipenv installwall time by ~30%.Builds on top of #6668 (Phase V — perf + pure-python prework); merge
phase V first, then this.
What's in scope
1. Pure-Python resolver backend (Initiative G phase 3 + 3b)
Drops the dependency on pip-internal resolve calls. The new backend
sits on the typed schema surface introduced in Phase IV, drives
resolvelibdirectly, and reuses Phase I's PEP 691 client +Phase II's
ParallelFetcher. Selectable via:pipenv lock --backend pure-python(CLI)PIPENV_RESOLVER=pure-python(env)[pipenv] resolver_backend = "pure-python"(Pipfile)with
--backend pip(the default) preserving byte-identicalpre-Phase-VI behaviour.
Major components:
pipenv/resolver/pure_python_requirement.pyRequirementdataclass — name, specifier, extras, marker, source, parent, introducing_markerpipenv/resolver/pure_python_metadata.pyMetadataFetcher(PEP 658 + wheel-head Range fallback) + sdist routingpipenv/resolver/pure_python_sdist.pybuildpipenv/resolver/pure_python_provider.pyresolvelib.AbstractProviderimpl (find_matches, get_dependencies, is_satisfied_by, get_preference, narrow_requirement_selection)pipenv/resolver/backends/pure_python.pyPurePythonBackendplumbing — drives the provider, translates the result mapping intoLockedRequirementpipenv/vendor/build/build— gives the sdist extractor a self-contained PEP 517 frontend2. Resolver correctness fixes shipped on the way to parity
find_matchesexcludesyanked candidates from automatic selection unless the user
explicitly pins to that exact version. Bench-fixture trigger:
sentry-relay==1.1.4(every artifact yanked) was being pickedas the highest match for
sentry-relay>=0.8.45and crashingthe sdist build path.
exist at the same release, the wheel wins. Previously the
per-version sdist could slip through and force an unnecessary
PEP 517 build (
python3-saml 1.16.0,redis-py-cluster 2.1.3).CandidateEvaluator:strict-no-prereleases first, PEP-440 fallback only when zero
stables match the merged specifier. Per-candidate
SpecifierSet.contains(..., prereleases=None)was applyingPEP 440's "no-final-release-matched" fallback on a single-element
iterable and admitting every prerelease.
narrow_requirement_selection— thepsycopg[binary]→psycopg-binaryshape now flows end-to-end:wire-shape parses extras,
find_matchespropagates them ontocloned candidates,
get_dependenciesstripsextra == Xclauses from runtime markers (kept on
introducing_markerforthe lockfile emitter) and emits a synthetic base-version
requirement to keep the bare and extras-flavoured identifiers
tied to the same version. Pip-style conflict-promote ordering
in
get_preference+ a port ofnarrow_requirement_selectionlet the resolver navigate the wider constraint graph (overlapping
upper bounds on
protobuf/grpcio/grpcio-statusfromthe
sentry-protos+ google-cloud-* fleet).longer crashes when a transitive's
build-backend(poetry-core,hatchling, flit-core, ...) isn't installed in pipenv's own
interpreter.
3. Parallel wheel pre-fetch (perf)
pipenv install/pipenv syncnow fan out wheel downloads acrossa 16-worker urllib3 pool BEFORE invoking
pip install, populating a--find-linksdirectory pip then reads from. Pip's install step issequential, so on a cold pip cache the network download phase
dominated wall time — the sentry-base bench's 151 wheels spent
~12 s of pure network in the pip subprocess before this.
The pre-fetch:
ParallelFetcher(same auth / netrc / cert handling already vetted under
GHSA-8xgg-v3jj-95m2),
hashes,
policy.skip_lockisFalse — without hashes there's no authoritative match key),
wheel, hash mismatch, network hiccup) falls through silently and
pip downloads via the index as usual.
--upgradewas also dropped frompipenv sync's pip-installinvocation — redundant under
--no-deps+ pinned versions + theupstream
is_satisfiedfilter, and it forced a per-packagemetadata check.
pipenv install(Pipfile-driven, may need todowngrade) still passes
--upgrade.Numbers
T_PARITY_REAL — ten wheel-heavy combos vs pip backend
10/10 byte-identical hash parity with the pip backend.
Sentry-base bench fixture (151 packages, Python 3.11, cold cache)
pipenv lock(pure-python backend)pipenv sync --backend pipcold installpipenv sync --backend pipwarm installThe lock-time number assumes the pure-python backend; the install-time
numbers benefit every backend because the pre-fetch sits at the
install plumbing layer, not the resolver.
Tests
--backend pure-pythonis exercised in the CImatrix alongside the pip backend (T_CI1).
Notable news fragments
+install-parallel-wheel-prefetch.feature.rst— the perf win+install-drop-redundant-upgrade-flag.behavior.rst+pure-python-extras-roundtrip.bugfix.rst+pure-python-prerelease-parity.bugfix.rst+pure-python-sdist-build-isolation.feature.rstT_F.5.feature.rst/T_F.6.behavior.rst—--backendplumbingdocs/dev/Test plan
tests/unit/test_pure_python_*,tests/unit/test_resolver_*)above
tests/unit/test_prefetch.py— happy +hash-mismatch + non-200 + target-tag failure + no-match paths)
benchmark all green at last push
🤖 Generated with Claude Code