|
| 1 | +# New resolver and download configuration |
| 2 | + |
| 3 | +- Author: Christian Heimes |
| 4 | + |
| 5 | +## What |
| 6 | + |
| 7 | +This enhancement document proposal a new approach to configure the package |
| 8 | +resolver and source / sdist downloader. The new settings are covering a |
| 9 | +wider range of use cases. Common patterns like building a package from a |
| 10 | +git checkout will no longer need custom Python plugins. |
| 11 | + |
| 12 | +## Why |
| 13 | + |
| 14 | +In downstream, we are encountering an increasing amount of packages that do |
| 15 | +not build from sdists on PyPI. Either package maintainers are not uploading |
| 16 | +source distributions to PyPI or sdists have issues. In some cases, packages |
| 17 | +use a midstream fork that is not on PyPI. The sources need to be build from |
| 18 | +git. |
| 19 | + |
| 20 | +Because Fromager <= 0.76 does not have declarative settings for Github/Gitlab |
| 21 | +resolver or cloning git repos, we have to write custom Python plugins. The |
| 22 | +plugins are a maintenance burden. |
| 23 | + |
| 24 | +## Goals |
| 25 | + |
| 26 | +- support common use cases with package settings instead of custom plugin code |
| 27 | +- cover most common resolver scenarios: |
| 28 | + - resolve package on PyPI (sdist, wheel, or both) |
| 29 | + - resolve package on Github or Gitlab with custom tag matcher |
| 30 | +- cover common sdist download and build scenarios: |
| 31 | + - sdist from PyPI |
| 32 | + - prebuilt wheel from PyPI |
| 33 | + - download tarball from URL |
| 34 | + - clone git repository |
| 35 | + - build sdist with PEP 517 hook or plain tarball |
| 36 | +- support per-variant setting, e.g. one variant uses prebuilt wheel while the |
| 37 | + rest uses sdist. |
| 38 | +- gradual migration path from old system to new configuration |
| 39 | + |
| 40 | +## Non-goals |
| 41 | + |
| 42 | +- The new system will not cover all use cases. Some specific use cases will |
| 43 | + still require custom code. |
| 44 | +- Retrival of additional sources is out of scope, e.g. a package `egg` that |
| 45 | + needs `libegg-{version}.tar.gz`. |
| 46 | + |
| 47 | +## How |
| 48 | + |
| 49 | +The new system will use a new top-level configuration key `source`. The old |
| 50 | +`download_source` and `resolver_dist` settings will stay supported for a |
| 51 | +while. Eventually the old options will be deprecated and removed. |
| 52 | + |
| 53 | +The resolver and source downloader can be configuration for all variants of |
| 54 | +a package as well as re-defined for each variant. Each use case is handled |
| 55 | +a provider profile. The profile name acts as a discriminator field. |
| 56 | + |
| 57 | +Example: |
| 58 | + |
| 59 | +```yaml |
| 60 | +source: |
| 61 | + # `pypi-sdist` is the default provider |
| 62 | + provider: pypi-sdist |
| 63 | +variants: |
| 64 | + egg: |
| 65 | + source: |
| 66 | + # resolve and download prebuilt wheel |
| 67 | + provider: pypi-prebuilt |
| 68 | + index_url: https://custom-index.test/simple |
| 69 | + spam: |
| 70 | + source: |
| 71 | + # resolve on Gitlab, clone, build an sdist with PEP 517 hook |
| 72 | + provider: gitlab |
| 73 | + url: https://gitlab.test/spam/spam |
| 74 | + matcher_factory: package_plugins.matchers:midstream_matcher_factory |
| 75 | + retrieve_method: git+https |
| 76 | + build_sdist: pep517 |
| 77 | + viking: |
| 78 | + source: |
| 79 | + # resolve on PyPI, git clone, and build as tarball |
| 80 | + provider: pypi-git |
| 81 | + clone_url: https://github.test/viking/viking.git |
| 82 | + tag: 'v{version}' |
| 83 | + build_sdist: tarball |
| 84 | +``` |
| 85 | +
|
| 86 | +
|
| 87 | +### Profiles |
| 88 | +
|
| 89 | +- The `pypi-sdist` profile resolve versions on PyPI or PyPI-compatible index. |
| 90 | + It only takes sdists into account and downloads the sdist from the index. |
| 91 | + The profile is equivalent to the current default settings with |
| 92 | + `include_sdists: true` and `include_wheels: false`. |
| 93 | + |
| 94 | +- The `pypi-prebuilt` profile resolve versions of platform-specific wheels |
| 95 | + on PyPI and downloads the pre-built wheel. The profile is equivalent to |
| 96 | + `include_sdists: false`, `include_wheels: true`, and variant setting |
| 97 | + `pre_build: true`. |
| 98 | + |
| 99 | +- The `pypi-download` resolve versions of any package on PyPI and downloads |
| 100 | + a tarball from an external URL (with `{version}` variable in download URL). |
| 101 | + It takes any sdist and any wheel into account. The profile is equivalent |
| 102 | + with `include_sdists: true`, `include_wheels: true`, `ignore_platform: true`, |
| 103 | + and a `download_source.url`. |
| 104 | + |
| 105 | +- The `pypi-git` is similar to the `pypi-download` profile. Instead of |
| 106 | + downloading a tarball, it clones a git repository at a specific tag. |
| 107 | + |
| 108 | +- The `gitlab` and `github` profiles use the `GitlabTagProvider` or |
| 109 | + `GitHubTagProvider` to resolve versions. The profiles can either download |
| 110 | + a git tag tarball or clone the repo at a specific tag. |
| 111 | + |
| 112 | +Like pip's VCS feature, all git clone operations automatically retrieve all |
| 113 | +submodules recursively. |
| 114 | + |
| 115 | + |
| 116 | +### Deprecations |
| 117 | + |
| 118 | +- `download_source.url` is handled by `pypi-download` profile or |
| 119 | + `release_artifact` parameter of `github` or `gitlab` provider |
| 120 | +- `download_source.destination_filename` is not needed. All sdists use |
| 121 | + standard `{dist_name}-{version}.tar.gz` file name |
| 122 | +- `resolver_dist.sdist_server_url` is replaced by `index_url` parameter. |
| 123 | + All `pypi-*` profile support a custom index. |
| 124 | +- `git_options.submodules` is not needed. Like pip, Fromager will always |
| 125 | + clone all submodules. |
| 126 | +- variant settings `wheel_server_url` and `pre_build` are replaced by |
| 127 | + `pypi-prebuilt` profile |
0 commit comments