Update models-downloading with downloading behind a proxy or firewall…#2628
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
rajatarya
left a comment
There was a problem hiding this comment.
Thanks @michelle — this is a genuinely useful addition. The exact set of hosts that downloads redirect to (Xet, CDN, legacy LFS) has never been documented, and firewalled/proxied users have been reverse-engineering it from forum threads, so closing that gap is great. A few things I especially liked: the Purpose column on the table, the ReadTimeoutError-as-a-symptom tip (that's exactly the signal people hit when a storage/CDN host is blocked mid-download), the explicit callout that legacy cdn-lfs*.huggingface.co isn't under *.hf.co, and the WARNING that the concrete list can drift. Style-wise it's consistent with the rest of docs/hub too — ## heading level matches the sibling sections, and > [!TIP]/> [!WARNING] are the prevailing callout syntax in this repo. No new links are introduced, so nothing to break there.
A few things to consider before merging:
- Wildcard coverage of the CDN edges is a bit optimistic. The reason
*.xethub.hf.cois listed separately is that*.hf.coonly matches a single label — but by that same logic*.hf.coalso won't cover the CDN edge hosts (us.aws.cdn.hf.co,us.gcp.cdn.hf.co) since those sit undercdn.hf.co. Worth either clarifying that coverage depends on whether the proxy does single-label vs. suffix matching, or recommending*.cdn.hf.coalongside*.xethub.hf.co. (inline) - The CDN edge rows look US-only while the Xet and LFS rows have explicit EU entries — worth confirming whether there are EU CDN edges to list. (inline)
- Routing through a proxy vs. allowlisting in a firewall. This section covers allowlisting hosts; consider a one-liner pointing users who route outbound traffic through a proxy at the standard
HTTP_PROXY/HTTPS_PROXYenvironment variables (honored by the underlying HTTP clients), since that's the adjacent question this audience tends to have. - Discoverability for datasets/Spaces. The intro mentions "models and datasets," and the same hosts apply, but this lives only in
models-downloading.md. Consider surfacing it indatasets-downloading.mdas well (or a shared page linked from both) so dataset users can find it. (inline) - nit: small comma splice in the wildcard tip. (inline)
Nothing blocking — these are all clarity/completeness polish on a solid, needed doc.
rajatarya
left a comment
There was a problem hiding this comment.
My comments are minor. Thanks for adding this.
Addresses comments from @rajatarya 💪
Co-authored-by: Hugo Larcher <hugo.larcher@huggingface.co>
julien-c
left a comment
There was a problem hiding this comment.
let's either add to this one as well (they have some content) https://huggingface.co/docs/hub/main/en/datasets-downloading#faster-downloads
OR split this into a dedicated page
|
Also i'm not sure i would document the legacy ones (they're not used anymore no?) |
|
Tadah! #2647 |
Adds a "Downloading behind a proxy or firewall" section to the model downloading guide.
Downloads redirect from huggingface.co to separate storage, Xet, and CDN hosts, so allowlisting huggingface.co alone isn't enough; from_pretrained and hf download fail (typically a ReadTimeoutError mid-download) when those hosts are blocked. This list wasn't documented anywhere, so users in firewalled/proxied environments were reverse-engineering it from forum threads and third-party posts.
The new section lists the required hostnames (Hub API, Xet US/EU, CDN edges, and legacy LFS) with their purpose, plus wildcard guidance (*.hf.co / *.xethub.hf.co / *.huggingface.co) and a note on the legacy huggingface.co LFS hosts that *.hf.co doesn't cover.
Note
Low Risk
Documentation-only change to the Hub model downloading guide; no application code, auth, or infrastructure behavior is modified.
Overview
Adds a Downloading behind a proxy or firewall section to
models-downloading.mdafter the hf-mount section.It explains that downloads redirect from
huggingface.coto separate Xet, storage, and CDN hosts, so allowlisting only the Hub domain is insufficient andfrom_pretrained/hf downloadcan fail (often withReadTimeoutErrormid-download). The section documents an explicit HTTPS allowlist table (Hub API, Xet US/EU, CDN edges, legacy LFS hosts) and tips on suffix vs single-label wildcard rules, plus a warning that endpoint hostnames may change over time.Reviewed by Cursor Bugbot for commit 6a9e062. Bugbot is set up for automated code reviews on this repo. Configure here.