crates.io 403 on python-requests UA is breaking nixpkgs rust builds

[andresilva]

nixpkgs rustPlatform.fetchCargoVendor uses python's requests library to vendor crates from crates.io, and it never sets an explicit user-agent, so every request goes out aspython-requests/<version>. Recently these requests started getting rejected with 403.

This affects a very large chunk of the Rust ecosystem on NixOS right now, essentially any Rust package that isn't already cached.

It looks like https://crates.io/api/v1/crates/<name>/<version>/download returns 403 Forbidden for certain python-requests/<version> user-agents. Current observed behavior:

python-requests/2.32.5 -> 403
python-requests/2.33.1 -> 403
python-requests/2.32.3 -> 302
(empty UA) -> 302
foo/1.0 -> 302

I put up fix at NixOS/nixpkgs#512735 that sets an identifying UA (nixpkgs fetchCargoVendor (https://github.com/NixOS/nixpkgs)), which aligns with crates.io's documented UA policy and bypasses the blocklist. But landing it in a usable form takes time:

• Review + merge into staging
• Propagation through staging → staging-next → master (since rustPlatform.fetchCargoVendor is a mass-rebuild trigger)
• Binary cache rebuild
• Channel bump to nixos-unstable / release branches

Realistically we're looking at weeks before affected users have a working fetchCargoVendor via channels, and in the meantime every user on an affected nixpkgs revision sees broken Rust builds.

Would it be possible to temporarily unblock the specific python-requests/<version> UAs that nixpkgs is currently sending in the wild (2.32.5 and 2.33.1 at least, not sure if more are needed)? Just long enough to cover the window until our PR lands and reaches users.

[emilazy]

Apologies on the Nixpkgs side for us not setting a UA sooner. I believe we can get this fixed on our end in a matter of days rather than weeks per NixOS/nixpkgs#512735 (comment), but if crates.io is amenable to adding a temporary exception to smooth things over for the period then that would certainly be very much appreciated.

[Turbo87]

I've temporarily disabled the block for the python-requests UAs again.

I would also recommend that you download the files directly from the CDN instead of going through our API servers: e.g. https://static.crates.io/crates/libc/0.2.185/download or https://static.crates.io/crates/libc/libc-0.2.185.crate. That addresses the rate limit concern mentioned in NixOS/nixpkgs#512735 (comment) since the 1 req/sec limit only applies to our API servers.

for completeness, the technically correct path for downloading crates is: load https://index.crates.io/config.json, look at the dl key in the JSON file, build the download URL via {dl}/{name}/{version}/download.

[emilazy]

Thank you very much! I suppose we can keep this open for now to give updates while we work on addressing this on our end?

I believe using the config.json paths should not require too much adjustment (though I know we have a couple fetcher implementations and I'm not an expert in their code).

FWIW, not ensuring all our fetchers have a UA set and being out of compliance with the policy is definitely our mistake and I certainly don't want to come off entitled to what I expect is a small team running a public service, but if it would have been feasible to do a stochastic brown-out of non-compliant requests earlier on then this would most likely have come to our attention sooner and we'd have had the workaround of manual retries while we got it fixed.

Edit: I suppose automatic retries would mask a brown-out, so it would possibly require state for a per-IP cool-off to avoid needing a too-high rejection probability to be smooth, which I can understand potentially making it less practical/worthwhile.

[emilazy]

We’ve merged a fix into both of our active branches; it’ll take a few days for them to be included in builds of the package set and reach our release channels, tracked at https://nixpk.gs/pr-tracker.html?pr=512735 and https://nixpk.gs/pr-tracker.html?pr=513551. At that point it should be fine for us to reinstate the block.

We shouldn’t be hitting the API server for crate downloads for the vast majority of packages after that, and we will have a distinct UA for the CDN requests.

[systec-csc]

There seems to be a similar problem with bitbake during fetching - specifically using the wget fetcher.
Bitbake calls wget -t 2 -T 100 -O askama-0.12.1.crate.tmp '<https://crates.io/api/v1/crates/askama/0.12.1/download>' --progress=dot -v and exits with the error Bitbake Fetcher Error: FetchError('Unable to fetch URL from any source.', 'crate://crates.io/askama/0.12.1')

Calling this command on my build system (Docker container, Ubuntu 22.04.) natively with wget, gives an similar error:

build@f1db6945cb58:~/shared/yocto/build$ wget -t 2 -T 100 -O askama-0.12.1.crate.tmp 'https://crates.io/api/v1/crates/askama/0.12.1/download' --progress=dot -v
--2026-04-27 08:09:40--  https://crates.io/api/v1/crates/askama/0.12.1/download
Resolving crates.io (crates.io)... 151.101.130.137, 151.101.194.137, 151.101.66.137, ...
Connecting to crates.io (crates.io)|151.101.130.137|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2026-04-27 08:09:40 ERROR 403: Forbidden.

I tested this with different Ubuntu and wget versions and come to the following conclusion:
Version 1.21.2 (Ubuntu 22.04.) and 1.21.4 (Ubuntu 24.04) of wget do not work.
Version 1.20.3 (Ubuntu 20.04.) and 1.25.0 (Ubuntu 26.04 and on my host system) of wget work.

The Yocto project is not (yet?) supported for Ubuntu 26.04. So there currently is no supported Ubuntu LTS version, we can possibly build rust recipes with.

This behavior seems relatively recent, a few weeks/months back, the build process worked without any issue. What changed?

[anta5010]

@systec-csc
wget 1.19.5 (old Centos) works fine as well.

Yocto worked for me last Thursday and stopped working lat Friday.

[Turbo87]

@systec-csc @anta5010 please report this to the yocto folks. they yocto fetcher is the system that does not seem to comply with the crates.io data access policy.

[anta5010]

@Turbo87 Yes, reporting now.
Yocto crate fetcher uses the API servers: https://git.openembedded.org/bitbake/tree/lib/bb/fetch2/crate.py#n69

It doesn't explain why the approach had worked until last Friday

[Turbo87]

It doesn't explain why the approach had worked until last Friday

see https://crates.io/data-access. probably because yocto is not sending a "user-agent header that identifies your application" and was blocked by our system due to that.

[anta5010]

@Turbo87 no, it's not the case. Yocto uses wget. Wget sends user agent as 'Wget/version'. See comments from me and @systec-csc it works for some wget versions, but not for other

[Turbo87]

Wget is not an identifying user agent. it happens to be the lib that yocto apparently uses, but I would expect a user agent that says "yocto" plus ideally contact information, as mentioned in the policy linked above...