Align dataset upload guide + storage limits with upload_folder/hf upload multi-commit (huggingface_hub 1.20) (#2580)

* Docs: reflect upload_folder/hf upload multi-commit (huggingface_hub 1.20)

upload_folder/hf upload now auto-split large folders into multiple
resumable commits, so the old manual guidance has drifted:

- datasets-upload-guide-llm: drop the removed `multi_commits=True`
  recommendation; add a note on automatic multi-commit, resumable/
  idempotent re-runs, the create_pr+revision restriction, and
  agent/non-TTY progress behavior
- storage-limits: manual "<100 files per commit" splitting is now
  handled automatically by upload_folder/hf upload

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Update docs/hub/datasets-upload-guide-llm.md

Co-authored-by: Lucain <lucainp@gmail.com>

* Update docs/hub/datasets-upload-guide-llm.md

Co-authored-by: Lucain <lucainp@gmail.com>

* Update docs/hub/datasets-upload-guide-llm.md

Co-authored-by: Lucain <lucainp@gmail.com>

* Update docs/hub/datasets-upload-guide-llm.md

Co-authored-by: Lucain <lucainp@gmail.com>

* Update docs/hub/datasets-upload-guide-llm.md

Co-authored-by: Lucain <lucainp@gmail.com>

* Update docs/hub/datasets-upload-guide-llm.md

Co-authored-by: Lucain <lucainp@gmail.com>

* Update docs/hub/datasets-upload-guide-llm.md

Co-authored-by: Lucain <lucainp@gmail.com>

* Docs: address review — reword agent TIP, drop upload_large_folder snippet + redundant 1.20 note

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Lucain <lucainp@gmail.com>

Author: 3 people

docs/hub/datasets-upload-guide-llm.md

@@ -6,6 +6,9 @@
 
 > Decision guide for uploading datasets to Hugging Face Hub. Optimized for Dataset Viewer compatibility and integration with the Hugging Face ecosystem.
 
+> [!TIP]
+> If you're an LLM or agent following this guide, check `hf --help` and `hf <command> --help` (e.g. `hf upload --help`, `hf auth --help`) for the exact flags before running upload commands. The CLI matches the installed `huggingface_hub` version, so prefer its output over flags recalled from memory.
+
 ## Overview
 
 Your goal is to help a user upload a dataset to the Hugging Face Hub. Ideally, the dataset should be compatible with the Dataset Viewer (and thus the `load_dataset` function) to ensure easy access and usability. You should aim to meet the following criteria:
@@ -95,10 +98,10 @@ See https://huggingface.co/docs/hub/storage-limits#repository-limitations-and-re
 | **Video files**                     | Use [videofolder](https://huggingface.co/docs/datasets/en/video_load#videofolder) for automatic organization | `load_dataset("videofolder", data_dir="./videos").push_to_hub("username/dataset")`               |
 | **PDF documents**                   | Use [pdffolder](https://huggingface.co/docs/datasets/en/document_dataset#pdffolder) for text extraction      | `load_dataset("pdffolder", data_dir="./pdfs").push_to_hub("username/dataset")`                   |
 | **Very large datasets (100GB+)**    | Use `max_shard_size` to control memory usage                                                                 | `dataset.push_to_hub("username/dataset", max_shard_size="5GB")`                                  |
-| **Many files / directories (>10k)** | Use upload_large_folder to avoid Git limitations                                                             | `api.upload_large_folder(folder_path="./data", repo_id="username/dataset", repo_type="dataset")` |
-| **Streaming large media**           | WebDataset format for efficient streaming                                                                    | Create .tar shards, then `upload_large_folder()`                                                 |
+| **Many files / directories (>10k)** | Use `upload_folder`/`hf upload` (handles large/many-file uploads)                                                             | `hf upload username/dataset ./data --repo-type=dataset` |
+| **Streaming large media**           | WebDataset format for efficient streaming                                                                    | Create .tar shards, then `hf upload`                                                 |
 | **Scientific data (HDF5, NetCDF)**  | Convert to Parquet with Array features                                                                       | See [Scientific Data](#scientific-data) section                                                  |
-| **Custom/proprietary formats**      | Document thoroughly if conversion impossible                                                                 | `upload_large_folder()` with comprehensive README                                                |
+| **Custom/proprietary formats**      | Document thoroughly if conversion impossible                                                                 | `hf upload` with comprehensive README                                                |
 
 ## Upload Workflow
 
@@ -119,7 +122,7 @@ See https://huggingface.co/docs/hub/storage-limits#repository-limitations-and-re
 
    - **Small files (<1GB) with hub-compatible format**: Can use [Hub UI](https://huggingface.co/new-dataset) for quick uploads
    - **Built-in loader available**: Use the loader + `push_to_hub()` (see Quick Reference table)
-   - **Large datasets or many files**: Use `upload_large_folder()` for files >100GB or >10k files
+   - **Large datasets or many files**: Use `upload_folder` / `hf upload` which handles large uploads (auto multi-commit, resumable).
    - **Custom formats**: Convert to hub-compatible format if possible, otherwise document thoroughly
 
 4. ✓ **Test locally** (if using built-in loader):
@@ -363,16 +366,21 @@ Use when you've loaded/converted data using the datasets library
 dataset.push_to_hub("username/dataset", max_shard_size="5GB")
 ```
 
-**Pre-existing files (use upload_large_folder)**:
-Use when you have hub-compatible files (e.g., Parquet files) already prepared and organized
+**Pre-existing files (use `upload_folder` / `hf upload`)**:
+Use when you have hub-compatible files (e.g., Parquet files) already prepared and organized.
 
 ```python
 from huggingface_hub import HfApi
 api = HfApi()
-api.upload_large_folder(folder_path="./data", repo_id="username/dataset", repo_type="dataset", num_workers=16)
+api.upload_folder(folder_path="./data", repo_id="username/dataset", repo_type="dataset")
+```
+
+```bash
+# Or from the CLI:
+hf upload username/dataset ./data --repo-type=dataset
 ```
 
-**Important**: Before using `upload_large_folder`, verify the files meet repository limits:
+**Important**: Before uploading, verify the files meet repository limits:
 
 - Check folder structure if you have file access: ensure no folder contains >10k files
 - Ask the user to confirm: "Are your files in a hub-compatible format (Parquet/CSV/JSON) and organized appropriately?"
@@ -420,7 +428,6 @@ api.upload_large_folder(folder_path="./data", repo_id="username/dataset", repo_t
 | "Repository not found"     | Run `hf auth login`          |
 | Memory errors              | Use `max_shard_size="500MB"`         |
 | Dataset viewer not working | Wait 5-10min, check README.md config |
-| Timeout errors             | Use `multi_commits=True`             |
 | Files >50GB                | Split into smaller files             |
 | "File not found"           | Use relative paths in metadata       |
 
@@ -510,7 +517,7 @@ For streaming large media datasets:
 
 - Create 1-5GB tar shards
 - Consistent internal structure
-- Upload with `upload_large_folder`
+- Upload with `hf upload`
 
 ### Scientific Data
 

docs/hub/storage-limits.md

@@ -73,7 +73,7 @@ We gathered a list of tips and recommendations for structuring your repo. If you
 | Files per repo     | <100k              | merge data into fewer files                            |
 | Entries per folder | <10k               | use subdirectories in repo                             |
 | File size          | <200GB             | split data into chunked files                          |
-| Commit size        | <100 files*        | upload files in multiple commits                       |
+| Commit size        | <100 files*        | `upload_folder`/`hf upload` auto-split large folders into multiple commits                       |
 | Commits per repo   | -                  | upload multiple files per commit and/or squash history |
 
 _\* Not relevant when using `git` CLI directly_
@@ -113,8 +113,7 @@ git operation (addition or delete) is checked by the server. When a hundred Larg
 each file is checked individually to ensure it's been correctly uploaded. When pushing data through HTTP,
 a timeout of 60s is set on the request, meaning that if the process takes more time, an error is raised. However, it can
 happen (in rare cases) that even if the timeout is raised client-side, the process is still
-completed server-side. This can be checked manually by browsing the repo on the Hub. To prevent this timeout, we recommend
-adding around 50-100 files per commit.
+completed server-side. This can be checked manually by browsing the repo on the Hub. The [`upload_folder`](https://huggingface.co/docs/huggingface_hub/main/en/guides/upload) method and `hf upload` command avoid this automatically by splitting large folders into multiple commits. If you commit manually, keep around 50-100 files per commit.
 
 ### Sharing large datasets on the Hub