Skip to content

Commit c832e8b

Browse files
authored
docs: add fast.hf.co and hf-speedtest to download pages (#2466)
Add a "Test your download speed" section to models-downloading and a "Faster downloads" section to datasets-downloading, linking to fast.hf.co and the hf-speedtest CLI extension.
1 parent 34402ce commit c832e8b

2 files changed

Lines changed: 24 additions & 0 deletions

File tree

docs/hub/datasets-downloading.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,17 @@ If you have write-access to the particular dataset repo, you'll also have the ab
5151

5252
Add your SSH public key to [your user settings](https://huggingface.co/settings/keys) to push changes and/or access private repos.
5353

54+
## Faster downloads
55+
56+
You can test your download speed from Hugging Face's CDN at [fast.hf.co](https://fast.hf.co). This runs a quick bandwidth test against the nearest HF edge server, helping you understand your baseline throughput. You can also measure download speed from the terminal using the [`hf-speedtest`](https://github.com/julien-c/hf-speedtest) CLI extension:
57+
58+
```bash
59+
hf extensions install julien-c/hf-speedtest
60+
hf speedtest
61+
```
62+
63+
For faster transfers, the Hub uses the [Xet storage backend](https://huggingface.co/docs/hub/en/xet/index) with adaptive concurrency that automatically tunes parallel streams based on network conditions. See [Faster downloads for models](./models-downloading#faster-downloads) for details on tuning options like `HF_XET_HIGH_PERFORMANCE=1`.
64+
5465
## Using hf-mount
5566

5667
For large datasets, you can mount a repo as a local filesystem with [hf-mount](https://github.com/huggingface/hf-mount) instead of downloading the full repo. Files are fetched lazily — only the bytes your code reads hit the network. Useful when your workflow expects local file paths (e.g. `tarfile`, `zipfile`, `imagefolder`) rather than Python iterators.

docs/hub/models-downloading.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,19 @@ Add your SSH public key to [your user settings](https://huggingface.co/settings/
5151

5252
## Faster downloads
5353

54+
### Test your download speed
55+
56+
You can test your download speed from Hugging Face's CDN at [fast.hf.co](https://fast.hf.co). This runs a quick bandwidth test against the nearest HF edge server, helping you understand your baseline throughput before tuning any settings.
57+
58+
You can also measure download speed directly from the terminal using the [`hf-speedtest`](https://github.com/julien-c/hf-speedtest) CLI extension:
59+
60+
```bash
61+
hf extensions install julien-c/hf-speedtest
62+
hf speedtest
63+
```
64+
65+
### Adaptive concurrency with hf_xet
66+
5467
`hf_xet` is a Rust-based package leveraging the [Xet storage backend](https://huggingface.co/docs/hub/en/xet/index) to optimize file transfers with chunk-based deduplication. By default, `hf_xet` uses **adaptive concurrency** — it automatically tunes the number of parallel transfer streams based on real-time network conditions, starting conservatively (1 stream) and scaling up to 64 concurrent streams as bandwidth permits.
5568

5669
For most machines — including data center environments — the default settings will already saturate the available network bandwidth. For advanced users on machines with high bandwidth **and at least 64 GB of RAM**, `HF_XET_HIGH_PERFORMANCE=1` raises concurrency bounds and significantly increases memory buffer sizes, which can help when downloading many large files in parallel.

0 commit comments

Comments
 (0)