Skip to content

Commit 45fd388

Browse files
tirantiran
committed
[proposal] New resolver configuration
Add design proposal for new resolver configuration. See: #937 Signed-off-by: Christian Heimes <cheimes@redhat.com>
1 parent d86f938 commit 45fd388

3 files changed

Lines changed: 222 additions & 0 deletions

File tree

‎docs/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@ Contributing to fromager.
7676
:maxdepth: 2
7777

7878
develop.md
79+
proposals/index.rst
7980

8081
What's with the name?
8182
---------------------

‎docs/proposals/index.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
Fromager Enhancement Proposals
2+
==============================
3+
4+
.. toctree::
5+
:maxdepth: 1
6+
7+
new-resolver-config

‎docs/proposals/new-resolver-config.md‎

Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
# New resolver and download configuration
2+
3+
- Author: Christian Heimes
4+
- Created: 2026-02-24
5+
- Status: Open
6+
7+
## What
8+
9+
This enhancement document proposes a new approach to configure the package
10+
resolver and source / sdist downloader. The new settings are covering a
11+
wider range of use cases. Common patterns like building a package from a
12+
git checkout will no longer need custom Python plugins.
13+
14+
## Why
15+
16+
In downstream, we are encountering an increasing amount of packages that do
17+
not build from sdists on PyPI. Either package maintainers are not uploading
18+
source distributions to PyPI or sdists have issues. In some cases, packages
19+
use a midstream fork that is not on PyPI. The sources need to be build from
20+
git.
21+
22+
Because Fromager \<= 0.76 does not have declarative settings for GitHub/GitLab
23+
resolver or cloning git repositories, we have to write custom Python plugins.
24+
The plugins are a maintenance burden.
25+
26+
## Goals
27+
28+
- support common use cases with package settings instead of custom plugin code
29+
- cover most common resolver scenarios:
30+
- resolve package on PyPI (sdist, wheel, or both)
31+
- resolve package on GitHub or GitLab with custom tag matcher
32+
- cover common sdist download and build scenarios:
33+
- sdist from PyPI
34+
- prebuilt wheel from PyPI
35+
- download tarball from URL
36+
- clone git repository
37+
- download an artifact from GitHub / GitLab release or tag
38+
- build sdist with PEP 517 hook or plain tarball
39+
- support per-variant setting, e.g. one variant uses prebuilt wheel while the
40+
rest uses sdist.
41+
- gradual migration path from old system to new configuration
42+
43+
## Non-goals
44+
45+
- The new system will not cover all use cases. Some specific use cases will
46+
still require custom code.
47+
- Retrieval of additional sources is out of scope, e.g. a package `egg` that
48+
needs `libegg-{version}.tar.gz`.
49+
- Provide SSH transport for git. The feature can be added at a later point
50+
when it's needed.
51+
- Extra options for authentication. The `requests` library and `git` CLI can
52+
use `$HOME/.netrc` for authentication.
53+
> **NOTE:** `requests` also supports `NETRC` environment variable,
54+
> `libcurl` and therefore `git` did not support `NETRC` before
55+
> libcurl [8.16.0](https://curl.se/ch/8.16.0.html) (2025-09-10). Before
56+
> `git` _only_ supports `$HOME/.netrc`.
57+
58+
## How
59+
60+
The new system will use a new top-level configuration key `source`. The old
61+
`download_source` and `resolver_dist` settings will stay supported for a
62+
while. Eventually the old options will be deprecated and removed.
63+
64+
The resolver and source downloader can be configuration for all variants of
65+
a package as well as re-defined for each variant. A package can be configured
66+
as prebuilt for all variants or a variant can have a different resolver and
67+
sources than other.
68+
69+
Each use case is handled a provider profile. The profile name acts as a tag
70+
([discriminated union](https://docs.pydantic.dev/latest/concepts/unions/#discriminated-unions)).
71+
Each use case has a well-defined set of mandatory and optional arguments.
72+
73+
**Example:**
74+
75+
```yaml
76+
source:
77+
# `pypi-sdist` is the default provider
78+
provider: pypi-sdist
79+
variants:
80+
egg:
81+
source:
82+
# resolve and download prebuilt wheel
83+
provider: pypi-prebuilt
84+
index_url: https://custom-index.example/simple
85+
spam:
86+
source:
87+
# resolve tag on GitLab, clone tag over https, build an sdist with PEP 517 hook
88+
provider: gitlab-tag-git
89+
project_url: https://gitlab.example/spam/spam
90+
matcher_factory: package_plugins.matchers:midstream_matcher_factory
91+
build_sdist: pep517
92+
viking:
93+
source:
94+
# resolve on PyPI, git clone, and build as tarball
95+
provider: pypi-git
96+
clone_url: https://git.example/viking/viking.git
97+
tag: 'v{version}'
98+
build_sdist: tarball
99+
caerbannog:
100+
# resolve with a mapping of version number to git refs, git clone, and build with PEP 517 hook
101+
source:
102+
provider: versionmap-git
103+
clone_url: https://git.example/viking/viking.git
104+
build_sdist: pep517
105+
versionmap:
106+
'1.0': abad1dea
107+
'1.1': refs/tags/1.1
108+
camelot:
109+
source:
110+
# On second thought, let's not go to Camelot. It is a silly place.
111+
provider: not-available
112+
```
113+
114+
### Profiles
115+
116+
- The `pypi-sdist` profile resolve versions on PyPI or PyPI-compatible index.
117+
It only takes sdists into account and downloads the sdist from the index.
118+
The profile is equivalent to the current default settings with
119+
`include_sdists: true` and `include_wheels: false`.
120+
121+
- The `pypi-prebuilt` profile resolve versions of platform-specific wheels
122+
on PyPI and downloads the pre-built wheel. The profile is equivalent to
123+
`include_sdists: false`, `include_wheels: true`, and variant setting
124+
`pre_build: true`.
125+
126+
- The `pypi-download` resolve versions of any package on PyPI and downloads
127+
a tarball from an external URL (with `{version}` variable in download URL).
128+
It takes any sdist and any wheel into account. The profile is equivalent
129+
with `include_sdists: true`, `include_wheels: true`, `ignore_platform: true`,
130+
and a `download_source.url`.
131+
132+
- The `pypi-git` is similar to the `pypi-download` profile. Instead of
133+
downloading a tarball, it clones a git repository at a specific tag.
134+
135+
- The `versionmap-git` profiles maps known version numbers to known git
136+
commits. It clones a git repo at the configured tag.
137+
138+
- The `gitlab-tag-git` and `github-tag-git` profiles use the
139+
`GitLabTagProvider` or `GitHubTagProvider` to resolve versions. The
140+
profiles git clone a project over `https` or `ssh` protocol.
141+
142+
- The `gitlab-tag-download` and `github-tag-download` are similar to
143+
`gitlab-tag-git` and `github-tag-git` profiles. Instead of cloning a git
144+
repository, they download a git tarball or an release artifact.
145+
146+
- The `not-available` profile raises an error. It can be used to block a
147+
package and only enable it for a single variant.
148+
149+
Like pip's VCS feature, all git clone operations automatically retrieve all
150+
submodules recursively. The final sdist does not include a `.git` directory.
151+
Instead Fromager generates a `.git_archival.txt` file for setuptools-scm's
152+
[builtin mechanism for obtaining version numbers](https://setuptools-scm.readthedocs.io/en/latest/usage/#builtin-mechanisms-for-obtaining-version-numbers).
153+
154+
### Interaction with existing hooks
155+
156+
**TBD**
157+
158+
How should the new settings interact with the `resolver_provider` and
159+
`download_source` hooks?
160+
161+
### URL schema for git
162+
163+
The resolver and `Candidate` class do not support VCS URLs, yet. Fromager can
164+
adopt pip's [VCS support](https://pip.pypa.io/en/stable/topics/vcs-support/)
165+
syntax. The URL `git+https://git.example/viking/viking.git@v1.1.0` clones the
166+
git repository over HTTPS and checks out the tag `v1.1.0`.
167+
168+
### Matcher factory
169+
170+
The matcher factory argument is an import string. The string must resolve to
171+
a callable that accepts a `ctx` argument and returns a `re.Pattern`
172+
(recommended) or `MatchFunction`. If the return value is a pattern object,
173+
then it must have exactly one match group. The pattern is matched with
174+
`re.match`.
175+
176+
The default matcher factory parsed the tag with `packaging.version.Version`
177+
and ignores any error. Fromager will provide additional matcher factories for
178+
common tag patterns like `v1.2`, `1.2`, and `v1.2-stable`.
179+
180+
```python
181+
import re
182+
183+
from fromager import context, resolver
184+
from packaging.version import Version
185+
186+
187+
def matcher_factory_pat(ctx: context.WorkContext) -> re.Pattern | resolver.MatchFunction:
188+
# tag must match 'v1.2+midstream.1.cpu' and results in Version("1.2+midstream.1")
189+
variant = re.escape(ctx.variant)
190+
pat = rf"^v(.*\+midstream\.\d+)\.{variant}$"
191+
return re.compile(pat)
192+
193+
194+
def matcher_factory_func(ctx: context.WorkContext) -> re.Pattern | resolver.MatchFunction:
195+
def pep440_matcher(identifier: str, item: str) -> Version | None:
196+
try:
197+
return Version(item)
198+
except ValueError:
199+
return None
200+
return pep440_matcher
201+
```
202+
203+
### Deprecations
204+
205+
- `download_source.url` is handled by `pypi-download` profile or
206+
`release_artifact` parameter of `github` or `gitlab` provider
207+
- `download_source.destination_filename` is not needed. All sdists use
208+
standard `{dist_name}-{version}.tar.gz` file name
209+
- `resolver_dist.sdist_server_url` is replaced by `index_url` parameter.
210+
All `pypi-*` profile support a custom index.
211+
- `git_options.submodules` is not needed. Like pip, Fromager will always
212+
clone all submodules.
213+
- variant settings `wheel_server_url` and `pre_build` are replaced by
214+
`pypi-prebuilt` profile

0 commit comments

Comments
 (0)