Align dataset upload guide + storage limits with upload_folder/hf upload multi-commit (huggingface_hub 1.20)

[davanstrien]

Aligning these docs with the new upload best practices. Since huggingface_hub 1.20, upload_folder / hf upload automatically split large folders into multiple resumable commits and are now the recommended path for large uploads to repos (buckets have their own path). The guides still described the old single-commit world, so:

datasets-upload-guide-llm.md

• Default to upload_folder / hf upload throughout; reframe upload_large_folder as a fall-back-if-you-hit-trouble option (it's more limited feature-wise — no custom path_in_repo/commit_message, no PR, no delete-during-upload)
• Drop the removed multi_commits=True recommendation
• Add a short note on the new multi-commit / resumable behavior + the create_pr+revision restriction
• Add a tip pointing at hf --help / hf <command> --help as the source of truth for current commands

storage-limits.md

• The manual "<100 files per commit" splitting is now handled automatically by upload_folder / hf upload

It might be worth deprecating the LLM upload-guide page down the line, but for now I'd rather keep it aligned with the CLI and current best practices than let it drift.

---

Note

Low Risk
Documentation-only changes with no runtime or security impact.

Overview
Updates dataset upload and storage limits documentation to match huggingface_hub 1.20+ behavior, where upload_folder and hf upload are the default path for large or many-file uploads (resumable, auto-split across commits).

In datasets-upload-guide-llm.md, quick-reference tables, workflow steps, upload-method examples, WebDataset appendix, and custom-format guidance now point at hf upload / api.upload_folder instead of upload_large_folder. A new tip tells LLM agents to use hf --help and per-command help for current CLI flags. The troubleshooting table removes the multi_commits=True timeout workaround.

In storage-limits.md, commit-size guidance now states that upload_folder/hf upload split large folders automatically; the operations-per-commit section no longer tells users to manually batch ~50–100 files when using those tools.

^{Reviewed by Cursor Bugbot for commit 21b16b8. Bugbot is set up for automated code reviews on this repo. Configure here.}

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[Wauplin]

Thanks for thinking about this.

To be clear, upload_large_folder will be deprecated/removed so we shouldn't mention it at all

[julien-c]

Thanks for thinking of updating this 🔥