feat!(pypi): enable bazel downloader by default (#3691)

Summary:
- change: Set `https://pypi.org/simple` as the default index.
- refactor: Leave the code for the legacy behaviour intact in case we
  need to do more work on the configuration to exclude the cases where
  the downloader should be used.
- add: add `index_url` configuration option for setting the defaults.
- add: add a small utility for parsing the arguments.
- fix: downloader will not be used if there is no url associated with
  the source irrespective of what the index_url setting is.
- fix: ensure all of the URLs are normalized when used.

Fixes #260
Fixes #1357
Fixes #2241
Fixes #2951

Author: aignas

.bazelrc.deleted_packages

@@ -27,8 +27,8 @@ common --deleted_packages=gazelle/manifest/hasher
 common --deleted_packages=gazelle/manifest/test
 common --deleted_packages=gazelle/modules_mapping
 common --deleted_packages=gazelle/python
-common --deleted_packages=gazelle/python/private
 common --deleted_packages=gazelle/pythonconfig
+common --deleted_packages=gazelle/python/private
 common --deleted_packages=tests/integration/compile_pip_requirements
 common --deleted_packages=tests/integration/compile_pip_requirements_test_from_external_repo
 common --deleted_packages=tests/integration/custom_commands

CHANGELOG.md

@@ -67,6 +67,12 @@ END_UNRELEASED_TEMPLATE
   on supported platforms (Linux/Mac with Bazel 8+, or Windows).
 * `--build_python_zip` on Windows is ignored. Use {obj}`py_zipapp_binary` to create
   zips of Python programs.
+* (pypi) Previously `experimental_index_url` users would not need to specify
+  target platforms if cross-building is required. From now we will only pull
+  wheels for the host OS to better align with how the rules work with the legacy
+  `pip` implementation. Use {obj}`pip.parse.target_platforms` to customize the
+  behavior.
+  Related to [#260](https://github.com/bazel-contrib/rules_python/issues/260).
 
 Other changes:
 * (pypi) Update dependencies used for `compile_pip_requirements`, building
@@ -75,12 +81,10 @@ Other changes:
   we will from now on fetch the lists of available packages on each index. The
   used package mappings will be written as facts to the `MODULE.bazel.lock` file
   on supported bazel versions and it should be done at most once. As a result,
-  per-package {obj}`experimental_index_url_overrides` is no longer needed if the
-  index URLs are passed to the `pip.parse` via `experimental_index_url` and
-  `experimental_extra_index_urls`.  What is more, we start implementing the flags
-  for `--index_url` and `--extra_index_urls` more in line to how it is used in
-  `uv` and `pip`, i.e. we default to `--index_url` if the package is not found in
-  `--extra_index_urls`.  Fixes
+  per-package {obj}`experimental_index_url_overrides` is no longer needed . What
+  is more, the flags for `--index_url` and `--extra-index-url` now behave in the
+  same way as in `uv` or `pip`, i.e. we default to `--index-url` if the package
+  is not found in `--extra-index-url`.  Fixes
   ([#3260](https://github.com/bazel-contrib/rules_python/issues/3260) and
   [#2632](https://github.com/bazel-contrib/rules_python/issues/2632)).
 * (uv) We will now use the download URL specified in the `uv`'s
@@ -130,6 +134,17 @@ Other changes:
   initializations and should no longer require the network access if the cache is
   hydrated.  Implements
   [#2731](https://github.com/bazel-contrib/rules_python/issues/2731).
+* (pypi) The `--index-url` and `--extra-index-url` is now parsed from the lock
+  file and the {obj}`pip.parse.experimental_index_url` and
+  {obj}`pip.parse.experimental_extra_index_urls` is
+  no longer mandatory to leverage the bazel downloader.
+  Implements
+  [#1357](https://github.com/bazel-contrib/rules_python/issues/1357),
+  [#2951](https://github.com/bazel-contrib/rules_python/issues/2951).
+* (pypi) If cross-compilation is needed, use the {obj}`pip.parse.target_platforms`
+  to specify exactly which platforms should be supported.
+  Implements
+  [#260](https://github.com/bazel-contrib/rules_python/issues/260).
 * (wheel) Specifying a path ending in `/` as a destination in `data_files`
   will now install file(s) to a folder, preserving their basename.
 * Various attributes and fields added to support venvs on Windows:
@@ -2315,4 +2330,4 @@ Breaking changes:
 * (pip) Create all_data_requirements alias
 * Expose Python C headers through the toolchain.
 
-[0.24.0]: https://github.com/bazel-contrib/rules_python/releases/tag/0.24.0
+[0.24.0]: https://github.com/bazel-contrib/rules_python/releases/tag/0.24.0

MODULE.bazel

@@ -65,9 +65,8 @@ register_toolchains("@pythons_hub//:all")
 
 pip = use_extension("//python/extensions:pip.bzl", "pip")
 
-# NOTE @aignas 2025-07-06: we define these platforms to keep backwards compatibility with the
-# current `experimental_index_url` implementation. Whilst we stabilize the API this list may be
-# updated with a mention in the CHANGELOG.
+# NOTE @aignas 2025-07-06: we define these platforms to keep backwards compatibility. Whilst we
+# stabilize the API this list may be updated with a mention in the CHANGELOG.
 [
     pip.default(
         arch_name = cpu,
@@ -306,7 +305,6 @@ dev_pip = use_extension(
 [
     dev_pip.parse(
         download_only = True,
-        experimental_index_url = "https://pypi.org/simple",
         hub_name = "dev_pip",
         parallel_download = False,
         python_version = python_version,
@@ -329,7 +327,6 @@ dev_pip = use_extension(
 
 dev_pip.parse(
     download_only = True,
-    experimental_index_url = "https://pypi.org/simple",
     hub_name = "pypiserver",
     python_version = "3.11",
     requirements_lock = "//examples/wheel:requirements_server.txt",

docs/pypi/download.md

@@ -104,9 +104,8 @@ the years, people started needing support for building containers, and usually,
 fetching dependencies for a particular target platform that may be different from the host
 platform.
 
-Multi-platform support for cross-building the wheels can be done in two ways:
-1. using {attr}`experimental_index_url` for the {bzl:obj}`pip.parse` bzlmod tag class
-2. using the {attr}`pip.parse.download_only` setting.
+Multi-platform support for cross-building the wheels can be done by
+using {attr}`target_platforms` for the {bzl:obj}`pip.parse` bzlmod tag class
 
 :::{warning}
 This will not work for sdists with C extensions, but pure Python sdists may still work using the first
@@ -207,16 +206,6 @@ additional keys, which become available during dependency evaluation.
 (bazel-downloader)=
 ### Bazel downloader and multi-platform wheel hub repository.
 
-:::{warning}
-This is currently still experimental, and whilst it has been proven to work in quite a few
-environments, the APIs are still being finalized, and there may be changes to the APIs for this
-feature without much notice.
-
-The issues that you can subscribe to for updates are:
-* {gh-issue}`260`
-* {gh-issue}`1357`
-:::
-
 The {obj}`pip` extension supports pulling information from `PyPI` (or a compatible mirror), and it
 will ensure that the [bazel downloader][bazel_downloader] is used for downloading the wheels.
 
@@ -228,14 +217,10 @@ This provides the following benefits:
 * Allow using transitions and targeting free-threaded and musl platforms more easily.
 * Avoids `pip` for wheel fetching and results in much faster dependency fetching.
 
-To enable the feature specify {attr}`pip.parse.experimental_index_url` as shown in
-the {gh-path}`examples/bzlmod/MODULE.bazel` example.
-
-Similar to [uv](https://docs.astral.sh/uv/configuration/indexes/), one can override the
-index that is used for a single package. By default, we first search in the index specified by
-{attr}`pip.parse.experimental_index_url`, then we iterate through the
-{attr}`pip.parse.experimental_extra_index_urls` unless there are overrides specified via
-{attr}`pip.parse.experimental_index_url_overrides`.
+Similar to [uv](https://docs.astral.sh/uv/configuration/indexes/), one can override the index that
+is used for a single package. By default, we first search in the indexes specified by
+`--extra-index-url`, then we fall back to the `--index-url` setting unless there are overrides
+specified via {attr}`pip.parse.experimental_index_url_overrides`.
 
 When using this feature during the `pip` extension evaluation you will see the accessed indexes similar to below:
 ```console

examples/bzlmod/MODULE.bazel

@@ -183,18 +183,6 @@ pip.default(
 pip.parse(
     # We can use `envsubst in the above
     envsubst = ["PIP_INDEX_URL"],
-    # Use the bazel downloader to query the simple API for downloading the sources
-    # Note, that we can use envsubst for this value.
-    experimental_index_url = "${PIP_INDEX_URL:-https://pypi.org/simple}",
-    # One can also select a particular index for a particular package.
-    # This ensures that the setup is resistant against confusion attacks.
-    # experimental_index_url_overrides = {
-    #    "my_package": "https://different-index-url.com",
-    # },
-    # Or you can specify extra indexes like with `pip`:
-    # experimental_extra_index_urls = [
-    #    "https://different-index-url.com",
-    # ],
     experimental_requirement_cycles = {
         "sphinx": [
             "sphinx",
@@ -208,6 +196,16 @@ pip.parse(
     extra_hub_aliases = {
         "wheel": ["generated_file"],
     },
+    extra_pip_args = [
+        # Use the bazel downloader to query the simple API for downloading the sources
+        # Note, that we can use envsubst for this value.
+        # One can also select a particular index for a particular package.
+        # This ensures that the setup is resistant against confusion attacks.
+        # experimental_index_url_overrides = {
+        #    "my_package": "https://different-index-url.com",
+        # },
+        "--index-url=${PIP_INDEX_URL:-https://pypi.org/simple}",
+    ],
     hub_name = "pip",
     python_version = "3.9",
     requirements_lock = "requirements_lock_3_9.txt",

python/private/pypi/BUILD.bazel

@@ -59,6 +59,11 @@ filegroup(
 
 # Keep sorted by library name and keep the files named by the main symbol they export
 
+bzl_library(
+    name = "argparse_bzl",
+    srcs = ["argparse.bzl"],
+)
+
 bzl_library(
     name = "attrs_bzl",
     srcs = ["attrs.bzl"],
@@ -224,6 +229,7 @@ bzl_library(
     name = "parse_requirements_bzl",
     srcs = ["parse_requirements.bzl"],
     deps = [
+        ":argparse_bzl",
         ":index_sources_bzl",
         ":parse_requirements_txt_bzl",
         ":pypi_repo_utils_bzl",
@@ -403,6 +409,7 @@ bzl_library(
     name = "requirements_files_by_platform_bzl",
     srcs = ["requirements_files_by_platform.bzl"],
     deps = [
+        ":argparse_bzl",
         ":whl_target_platforms_bzl",
     ],
 )

python/private/pypi/argparse.bzl

@@ -0,0 +1,40 @@
+"""A small set of utilities for parsing pip args."""
+
+def _get_pip_args(args, *arg_names, value = None, repeated = False):
+    set_next = False
+    if repeated:
+        value = [] + (value or [])
+
+    for arg in (args or []):
+        if arg in arg_names:
+            set_next = True
+            continue
+
+        val = None
+        if set_next:
+            set_next = False
+            val = arg
+        else:
+            for arg_name in arg_names:
+                start = "{}=".format(arg_name)
+
+                if arg.startswith(start):
+                    val = arg[len(start):]
+                    break
+
+            if val == None:
+                continue
+
+        if repeated:
+            if val not in value:
+                value.append(val)
+        else:
+            value = val
+
+    return value
+
+argparse = struct(
+    index_url = lambda args, default: _get_pip_args(args, "-i", "--index-url", value = default),
+    extra_index_url = lambda args, default: _get_pip_args(args, "--extra-index-url", value = default, repeated = True),
+    platform = lambda args, default: _get_pip_args(args, "--platform", value = default, repeated = True),
+)

python/private/pypi/extension.bzl

@@ -115,6 +115,7 @@ def build_config(
             _configure(
                 defaults,
                 override = mod.is_root,
+                index_url = tag.index_url,
                 # extra values that we just add
                 auth_patterns = tag.auth_patterns,
                 netrc = tag.netrc,
@@ -125,6 +126,7 @@ def build_config(
 
     return struct(
         auth_patterns = defaults.get("auth_patterns", {}),
+        index_url = defaults.get("index_url", "https://pypi.org/simple").rstrip("/"),
         netrc = defaults.get("netrc", None),
         platforms = {
             name: _plat(**values)
@@ -449,6 +451,30 @@ Supported keys:
 ::::{note}
 This is only used if the {envvar}`RULES_PYTHON_ENABLE_PIPSTAR` is enabled.
 ::::
+""",
+    ),
+    "index_url": attr.string(
+        doc = """\
+The index URL to use as a default when downloading packages from PyPI. This is used if nothing is
+specified via `--index-url` or `--extra-index-url` parameters in the `requirements.txt` file or via
+the {attr}`pip.parse.extra_pip_args`.
+
+This value is going to be subject to `envsubst` substitutions if necessary, look at the
+{attr}`pip.parse.envsubst` documentation for more information..
+
+The indexes must support Simple API as described here:
+https://packaging.python.org/en/latest/specifications/simple-repository-api/
+
+Index metadata will be used to get `sha256` values for packages even if the
+`sha256` values are not present in the requirements.txt lock file.
+
+Defaults to `https://pypi.org/simple`.
+
+:::{versionadded} 2.0.0
+This has been added as a replacement for 
+{obj}`pip.parse.experimental_index_url` and 
+{obj}`pip.parse.experimental_extra_index_urls`.
+:::
 """,
     ),
     "marker": attr.string(
@@ -566,43 +592,23 @@ def _pip_parse_ext_attrs(**kwargs):
     attrs = dict({
         "experimental_extra_index_urls": attr.string_list(
             doc = """\
-The extra index URLs to use for downloading wheels using bazel downloader.
-Each value is going to be subject to `envsubst` substitutions if necessary.
+May be removed in future releases.
 
-The indexes must support Simple API as described here:
-https://packaging.python.org/en/latest/specifications/simple-repository-api/
-
-This is equivalent to `--extra-index-urls` `pip` option.
-
-:::{versionchanged} 1.1.0
-Starting with this version we will iterate over each index specified until
-we find metadata for all references distributions.
+:::{versionchanged} 2.0.0
+This is deprecated, please use {obj}`pip.default.index_url` or pass the `--index-url` parameter via the
+lock-file or {obj}`pip.parse.extra_pip_args`.
 :::
 """,
             default = [],
         ),
         "experimental_index_url": attr.string(
             default = kwargs.get("experimental_index_url", ""),
             doc = """\
-The index URL to use for downloading wheels using bazel downloader. This value is going
-to be subject to `envsubst` substitutions if necessary.
-
-The indexes must support Simple API as described here:
-https://packaging.python.org/en/latest/specifications/simple-repository-api/
-
-In the future this could be defaulted to `https://pypi.org` when this feature becomes
-stable.
-
-This is equivalent to `--index-url` `pip` option.
+May be removed in future releases.
 
-:::{versionchanged} 0.37.0
-If {attr}`download_only` is set, then `sdist` archives will be discarded and `pip.parse` will
-operate in wheel-only mode.
-:::
-
-:::{versionchanged} 1.4.0
-Index metadata will be used to deduct `sha256` values for packages even if the
-`sha256` values are not present in the requirements.txt lock file.
+:::{versionchanged} 2.0.0
+This is deprecated, please use {obj}`pip.default.index_url` or pass the `--index-url` parameter via the
+lock-file or {obj}`pip.parse.extra_pip_args`.
 :::
 """,
         ),
@@ -835,9 +841,6 @@ the BUILD files for wheels.
 This tag class allows for more customization of how the configuration for the hub repositories is built.
 
 
-:::{include} /_includes/experimental_api.md
-:::
-
 :::{seealso}
 The [environment markers][environment_markers] specification for the explanation of the
 terms used in this extension.