Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions docs/hub/datasets-downloading.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,17 @@ If you have write-access to the particular dataset repo, you'll also have the ab

Add your SSH public key to [your user settings](https://huggingface.co/settings/keys) to push changes and/or access private repos.

## Faster downloads

You can test your download speed from Hugging Face's CDN at [fast.hf.co](https://fast.hf.co). This runs a quick bandwidth test against the nearest HF edge server, helping you understand your baseline throughput. You can also measure download speed from the terminal using the [`hf-speedtest`](https://github.com/julien-c/hf-speedtest) CLI extension:

```bash
hf extensions install julien-c/hf-speedtest
hf speedtest
```

For faster transfers, the Hub uses the [Xet storage backend](https://huggingface.co/docs/hub/en/xet/index) with adaptive concurrency that automatically tunes parallel streams based on network conditions. See [Faster downloads for models](./models-downloading#faster-downloads) for details on tuning options like `HF_XET_HIGH_PERFORMANCE=1`.

## Using hf-mount

For large datasets, you can mount a repo as a local filesystem with [hf-mount](https://github.com/huggingface/hf-mount) instead of downloading the full repo. Files are fetched lazily — only the bytes your code reads hit the network. Useful when your workflow expects local file paths (e.g. `tarfile`, `zipfile`, `imagefolder`) rather than Python iterators.
Expand Down
13 changes: 13 additions & 0 deletions docs/hub/models-downloading.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,19 @@ Add your SSH public key to [your user settings](https://huggingface.co/settings/

## Faster downloads

### Test your download speed

You can test your download speed from Hugging Face's CDN at [fast.hf.co](https://fast.hf.co). This runs a quick bandwidth test against the nearest HF edge server, helping you understand your baseline throughput before tuning any settings.

You can also measure download speed directly from the terminal using the [`hf-speedtest`](https://github.com/julien-c/hf-speedtest) CLI extension:

```bash
hf extensions install julien-c/hf-speedtest
hf speedtest
```

### Adaptive concurrency with hf_xet

`hf_xet` is a Rust-based package leveraging the [Xet storage backend](https://huggingface.co/docs/hub/en/xet/index) to optimize file transfers with chunk-based deduplication. By default, `hf_xet` uses **adaptive concurrency** — it automatically tunes the number of parallel transfer streams based on real-time network conditions, starting conservatively (1 stream) and scaling up to 64 concurrent streams as bandwidth permits.

For most machines — including data center environments — the default settings will already saturate the available network bandwidth. For advanced users on machines with high bandwidth **and at least 64 GB of RAM**, `HF_XET_HIGH_PERFORMANCE=1` raises concurrency bounds and significantly increases memory buffer sizes, which can help when downloading many large files in parallel.
Expand Down
Loading