diff --git a/docs/hub/datasets-downloading.md b/docs/hub/datasets-downloading.md index f653323c9c..63eb8bc778 100644 --- a/docs/hub/datasets-downloading.md +++ b/docs/hub/datasets-downloading.md @@ -51,6 +51,17 @@ If you have write-access to the particular dataset repo, you'll also have the ab Add your SSH public key to [your user settings](https://huggingface.co/settings/keys) to push changes and/or access private repos. +## Faster downloads + +You can test your download speed from Hugging Face's CDN at [fast.hf.co](https://fast.hf.co). This runs a quick bandwidth test against the nearest HF edge server, helping you understand your baseline throughput. You can also measure download speed from the terminal using the [`hf-speedtest`](https://github.com/julien-c/hf-speedtest) CLI extension: + +```bash +hf extensions install julien-c/hf-speedtest +hf speedtest +``` + +For faster transfers, the Hub uses the [Xet storage backend](https://huggingface.co/docs/hub/en/xet/index) with adaptive concurrency that automatically tunes parallel streams based on network conditions. See [Faster downloads for models](./models-downloading#faster-downloads) for details on tuning options like `HF_XET_HIGH_PERFORMANCE=1`. + ## Using hf-mount For large datasets, you can mount a repo as a local filesystem with [hf-mount](https://github.com/huggingface/hf-mount) instead of downloading the full repo. Files are fetched lazily — only the bytes your code reads hit the network. Useful when your workflow expects local file paths (e.g. `tarfile`, `zipfile`, `imagefolder`) rather than Python iterators. diff --git a/docs/hub/models-downloading.md b/docs/hub/models-downloading.md index 2dcdc84bc7..88413edc1a 100644 --- a/docs/hub/models-downloading.md +++ b/docs/hub/models-downloading.md @@ -51,6 +51,19 @@ Add your SSH public key to [your user settings](https://huggingface.co/settings/ ## Faster downloads +### Test your download speed + +You can test your download speed from Hugging Face's CDN at [fast.hf.co](https://fast.hf.co). This runs a quick bandwidth test against the nearest HF edge server, helping you understand your baseline throughput before tuning any settings. + +You can also measure download speed directly from the terminal using the [`hf-speedtest`](https://github.com/julien-c/hf-speedtest) CLI extension: + +```bash +hf extensions install julien-c/hf-speedtest +hf speedtest +``` + +### Adaptive concurrency with hf_xet + `hf_xet` is a Rust-based package leveraging the [Xet storage backend](https://huggingface.co/docs/hub/en/xet/index) to optimize file transfers with chunk-based deduplication. By default, `hf_xet` uses **adaptive concurrency** — it automatically tunes the number of parallel transfer streams based on real-time network conditions, starting conservatively (1 stream) and scaling up to 64 concurrent streams as bandwidth permits. For most machines — including data center environments — the default settings will already saturate the available network bandwidth. For advanced users on machines with high bandwidth **and at least 64 GB of RAM**, `HF_XET_HIGH_PERFORMANCE=1` raises concurrency bounds and significantly increases memory buffer sizes, which can help when downloading many large files in parallel.