Add dataset upload guide for LLMs (experimental guide)

[davanstrien]

Overview

This PR introduces an experimental guide tailored to help LLMs assist users in uploading datasets to the Hugging Face Hub in a format compatible with the Hub. The goal is to be an llms.txt file, but focused less on the datasets library and more on "how do I get this dataset onto the Hub in the most optimal format"

The primary use case is researchers or practitioners who have existing datasets (research data, domain-specific collections, etc.) that aren't in native Hub formats. They can provide this guide to an LLM (Claude Code, Cursor, or other AI coding assistants) along with a pointer to their local dataset. While advanced agents may handle much of the conversion automatically, even basic LLM + prompting workflows benefit from this structured guidance.

This is a proof of concept to test whether we should consider adding more LLM-focused documentation to the Hub docs for datasets. The guide is structured to work well with LLM context windows and provides a decision-tree approach that LLMs can follow to give users optimal upload recommendations.

Motivation

Without dedicated guidance, LLMs may recommend outdated or suboptimal approaches for dataset uploads based on their training data. This guide ensures LLMs have access to current best practices, including:

• Latest built-in loaders (imagefolder, audiofolder, videofolder, etc.)
• Current repository limits and recommendations
• Modern upload methods like upload_large_folder for large datasets
• Dataset Viewer compatibility requirements

What's included

New guide: datasets-upload-guide-llm.md covering:

• Decision workflow for choosing upload methods
• Built-in loader usage (imagefolder, audiofolder, etc.)
• Custom format conversion strategies
• Validation and troubleshooting steps
• Multi-modal dataset examples
• Added to TOC under the Datasets section

Note: This is experimental, and we can iterate on the format based on feedback and more real-world testing with LLMs. We may not want to merge this straight away (or at all), but I think it could be nice to start pointing people to this guide.

I also have a relatively good agent-based approach for end-to-end conversion of existing datasets to hub-compatible formats. I think it makes sense to start with this approach first, so we can get some sense of how well a more minimal approach like this can work for helping people with this.

cc @lhoestq @julien-c for viz.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lhoestq]

nice! I wonder if we should also include this in the HF datasets MCP as a list of instructions accessible for agents to read ?

edit: as replied on slack, this is more for llm + search than coding agents so it's fine as docs :)

[davanstrien]

Thanks for the feedback @lhoestq! I've addressed the technical issues:

1. ✅ Fixed Array1D → List() with length parameter
2. ✅ Added push_to_hub() calls to the multi-modal examples

Regarding the toctree placement: I initially placed it at the same level as "Uploading Datasets" for better discoverability by LLM agents (as you mentioned, SEO for LLM agents), but I'm happy to move it as a subsection under "Uploading Datasets" if that's preferred. What do you think would be best?

[julien-c]

cool idea. I think it's a "Resource" in MCP speak no? https://modelcontextprotocol.io/docs/concepts/resources

(cc @evalstate too)