Skip to content

Commit bd69f9a

Browse files
egrace479hlapp
egrace479
and
hlapp
committed
HF upload guide update
Pull from Collab Guide [PR 65](Imageomics/Collaborative-distributed-science-guide#65) * Initial re-organization of page, includes integrity checks and more links to docs WIP refining * Add in links, fix typos * Additional notes in sections, refs out * Add updated screenshot with arrow match UI change to 'contribute' * Remove extra outdated content, update URLs for new versions, run linting * Clarify UI upload cases Co-authored-by: Hilmar Lapp <hlapp@drycafe.net> * Clarify CLI upload use case Co-authored-by: Hilmar Lapp <hlapp@drycafe.net> * Add pointer for considerations with large folder uploads --------- Co-authored-by: Hilmar Lapp <hlapp@drycafe.net>
1 parent aec8a5c commit bd69f9a

2 files changed

Lines changed: 78 additions & 49 deletions

File tree

‎docs/wiki-guide/The-Hugging-Face-Dataset-Upload-Guide.md‎

Lines changed: 78 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,50 @@
11
# Hugging Face Dataset Guide
22

3-
## Create a New Dataset Repository
3+
[Hugging Face](https://hf.co/) offers numerous methods for interacting with and creating datasets. This page provides a basic overview with some recommendations specifically targeting image dataset uploads, though the principles are transferrable to other data types. We list these options&mdash;in order of increasing complexity&mdash;with some guidance, recommendations, and links out to the appropriate parts of the Hugging Face docs for the most up-to-date information available.
44

5-
When creating a new dataset repository, you can make the dataset **Public** (accessible to anyone on the internet) or **Private** (accessible only to members of the organization).
5+
1. [Web interface (UI)](#upload-a-dataset-with-the-web-interface): For smaller, simpler uploads.
6+
2. [Hugging Face Command Line Interface (CLI)](#upload-a-dataset-with-the-hugging-face-cli): For most use-cases, easy access from cluster.
7+
3. [Hugging Face API (python package)](#upload-a-dataset-with-hfapi): For when more fine-grained control than is achievable with the CLI is needed.
8+
4. [Git/Git LFS](#upload-a-dataset-with-git): Main use-case is when multiple PRs lead to merge conflicts&mdash;Hugging Face provides no other means for resolution.
69

7-
![New dataset repository interface](images/HF-dataset-upload/346972860-ed0feb0e-529b-4021-b44f-41ac96680bc3.png){ loading=lazy, width=800 }
8-
/// caption
9-
///
10+
!!! info
11+
Some sections of the Hugging Face docs, such as for the `huggingface_hub`, have only version specific links for stable versions. In this case, if the link directs to an older version, there will be a banner to alert you to a newer version available, so keep an eye out for that updated version banner.
12+
13+
Most of the content below is covered in various parts of [Hugging Face's Upload Guide](https://huggingface.co/docs/huggingface_hub/en/guides/upload); this page is provided as a summary reference mainly to determine which method might be best and link to the appropriate docs. Additionally, we include an [integrity check](#integrity-check) to help you ensure that your repo contains all the desired files after uploading through any of these methods.
14+
15+
[HF tips and tricks for large uploads](https://huggingface.co/docs/huggingface_hub/en/guides/upload#tips-and-tricks-for-large-uploads).
16+
17+
## Note on Authentication
18+
19+
All of these methods require authentication to edit datasets, ranging from passwords, to tokens, to SSH authentication, and all support editing **Public** (accessible to anyone on the internet) or **Private** (accessible only to members of the organization) repos. Two key notes on authentication:
20+
21+
1. Private repositories are only visible if you are authenticated.
22+
2. If using tokens for access, be sure to create a [fine-grained token](https://huggingface.co/docs/hub/en/security-tokens#what-are-user-access-tokens), specifically for your needs.
1023

1124
## Upload a Dataset with the Web Interface
1225

13-
In the Files and versions tab of the Dataset card, you can choose to add file in the hugging web interface.
26+
In the Files and Versions tab of the repository, you can select "Contribute" to add or create files or start a pull request directly from the web interface.
1427

1528
![Dataset repository Add file button](images/HF-dataset-upload/346190430-9e6cef9b-18ef-4d4a-84c5-1a3f75ac9336.png){ loading=lazy }
1629

30+
This method is fine for smaller files (<100MB), or data uploads from distributed sources, that have a relatively flat structure with few directories, and/or have few files. If you are uploading existing files, navigate to the target folder first.
31+
32+
## Upload a Dataset with the Hugging Face CLI
33+
34+
Hugging Face provides a comprehensive Command Line Interface (CLI) and corresponding [docs](https://huggingface.co/docs/huggingface_hub/en/guides/cli). Note that this is installed with the `huggingface_hub` python package, but can also be installed directly, then called with `hf <command>`.
35+
36+
The Hugging Face CLI is the ideal method for uploads that are large in volume, have more than a few files, and/or a folder structure with many or nested directories. It works directly from HPC clusters, such as OSC. Under the hood, [`hf upload`](https://huggingface.co/docs/huggingface_hub/en/guides/cli#hf-upload) uses the same upload functions described below, under [Upload a Dataset with HfApi](#upload-a-dataset-with-hfapi). Review [Hugging Face's guidance on large folder uploads](https://huggingface.co/docs/huggingface_hub/v1.10.1/guides/upload#upload-a-large-folder) before selecting a method for uploading large folders to a non-empty repository.
37+
38+
When uploading to a dataset, note that the repo type must be specified (`--repo-type=dataset`); this is also the case for spaces, since Hugging Face treats models as the default.
39+
40+
There are specific [`hf datasets`](https://huggingface.co/docs/huggingface_hub/en/guides/cli#hf-datasets) and [`hf repo`](https://huggingface.co/docs/huggingface_hub/en/guides/cli#hf-repo) commands for more general queries and repo initialization.
41+
1742
## Upload a Dataset with HfApi
1843

44+
When more complex dataset structures are involved or more fine-grained control (not exposed on CLI) over how a repo will be organized on Hugging Face is neeeded, the Hugging Face API may be the answer. For instance, if a glob pattern cannot sufficiently clarify necessary exclusions of subfolders or files, [`HfApi`](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api) is likely the preferred choice. This is a class, accessible through the [`huggingface_hub` package](https://huggingface.co/docs/huggingface_hub/index), that acts as a Python wrapper for the API.
45+
46+
Please see the Hugging Face API docs for the most up-to-date guidance. For quick reference, to [upload by file](https://huggingface.co/docs/huggingface_hub/v1.10.1/package_reference/hf_api#huggingface_hub.HfApi.upload_file) or [upload by folder (structure maintained)](https://huggingface.co/docs/huggingface_hub/v1.10.1/package_reference/hf_api#huggingface_hub.HfApi.upload_folder).
47+
1948
``` py linenums="1"
2049
from huggingface_hub import login
2150

@@ -24,70 +53,70 @@ login()
2453

2554
from huggingface_hub import HfApi
2655
api = HfApi()
56+
repo_id = "<ABC-Center/dataset name>"
2757

28-
api.upload_file (
58+
# Upload by file
59+
api.upload_file(
2960
path_or_fileobj = <the local file path that you would like to upload>,
3061
path_in_repo = <the path in the repo>,
31-
repo_id = <ABC-Center/dataset name>,
62+
repo_id = repo_id,
3263
repo_type = 'dataset'
3364
)
34-
```
35-
36-
## Upload a Dataset with Git
37-
38-
### If the Dataset is Less Than 5GB
39-
40-
Navigate to the folder for the repository:
4165

66+
# Upload by folder (maintain structure)
67+
api.upload_folder(
68+
folder_path="/path/to/local/folder", # should end with folder containing data
69+
path_in_repo="path/to/folder/", # path desired for folder in repo
70+
repo_id= repo_id,
71+
repo_type="dataset",
72+
    token_id = "paste-token-here" # if you're not logged in, HF does **not** recommend this method
73+
)
4274
```
43-
# Clone the repository
44-
git clone https://huggingface.co/datasets/username/repo-name
4575

46-
# Add, commit, and push the files
47-
git add
48-
git commit -m 'comments'
49-
git push
76+
Repos can also be created through the Hugging Face API using the [create_repo method](https://huggingface.co/docs/huggingface_hub/v1.10.1/en/package_reference/hf_api#huggingface_hub.HfApi.create_repo) with the following parameters:
5077

78+
```py linenums="1"
79+
repo_id = "<HF-Org-name>/<dataset-name>"
80+
repo_type = "dataset"
81+
private = True # if you want the repo private
5182
```
5283

53-
### If the Dataset is Larger Than 5GB
84+
See also instructions using the [datasets package](https://huggingface.co/docs/datasets/create_dataset).
5485

55-
#### Install Git LFS
86+
## Upload a Dataset with Git
5687

57-
Follow instructions at <https://git-lfs.com/>
88+
Using Git to interact with Hugging Face requires installation of [Git LFS](https://git-lfs.com/), the [Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/en/guides/cli), and then enabling large file upload for the repo
5889

59-
#### Install the Hugging Face CLI
90+
Hugging Face provides details on [git vs http](https://huggingface.co/docs/huggingface_hub/en/concepts/git_vs_http), really using Git vs HfApi
6091

61-
```
62-
brew install huggingface-cli
63-
pip install -U "huggingface_hub[cli]"
64-
```
92+
Hugging Face has moved away from Git LFS, instead utilizing [Xet](https://huggingface.co/docs/hub/en/xet/index) for data storage and version control; this is [backwards compatible with LFS](https://huggingface.co/docs/hub/en/xet/legacy-git-lfs).
6593

66-
#### Enable the repository to upload large files
94+
## Other Repo Considerations
6795

68-
```
69-
huggingface-cli lfs-enable-largefiles <your local dataset>
70-
```
96+
One time where `git` may be needed, is if one encounters a [merge conflict](https://discuss.huggingface.co/t/how-to-fix-merge-conflicts-in-prs/160090). Unlike GitHub, Hugging Face ***does not*** have conflict resolution UI tools, nor does it provide merge conflict resolution capabilities in the CLI or HfApi. The only means for resolving merge conflicts is to manually update the pull request in a [local clone](The-Hugging-Face-Workflow.md#hugging-face-pull-requests-with-local-edits), pulling `main` into your PR branch and resolving the conflicts.
7197

72-
#### Initialize Git LFS
98+
## Integrity Check
7399

74-
```
75-
git lfs install
76-
```
100+
Sometimes uploads fail partway through, leaving one or more files un-uploaded. Unfortunately, it seems that there is not an easy solution to be alerted to these issues when not uploading through the UI. Additionally, using a glob pattern to set upload without a dry-run[^1] (in `git` terms, this would be running `git status` after adding files) can also lead to accidental exclusion. To catch these issues, we recommend the following integrity check after uploading a dataset[^1].
77101

78-
#### Track large files (e.g., .csv files)
102+
[^1]: The Hugging Face CLI does have a [dry-run mode](https://huggingface.co/docs/huggingface_hub/en/guides/cli#dry-run-mode) for *downloading* datasets. Additionally, if working with Git LFS, there is a [preupload LFS](https://huggingface.co/docs/huggingface_hub/en/guides/upload#preupload-lfs-files-before-commit) option to ensure all files are properly preset and organized before committing. There are additional considerations for sharding noted in the Hugging Face docs.
79103

80-
```
81-
# Adds a line to .gitattributes, which Git uses to determine files managed by LFS
82-
git lfs track "*.csv"
83-
git add .gitattributes
84-
git commit -m "Track large files with Git LFS"
85-
```
104+
```python
105+
import pandas as pd
106+
from huggingface_hub import HfApi
86107

87-
#### Add, commit, and push the files
108+
api = HfApi()
109+
repo_id = "<org-name>/<repo-name>"
110+
repo_type = "dataset"
88111

112+
file_list = api.list_repo_files(repo_id=repo_id, repo_type=repo_type)
113+
file_df = pd.Dataframe(columns = {"filepath": file_list})
114+
metadata = pd.read_csv("path/to/metadata/file")
115+
116+
# assuming you use the same filepath in your system as in the repo
117+
df = pd.merge(file_df, metadata, how = "inner", on = "filepath")
118+
df.shape[0] # this should match the number of expected images
89119
```
90-
git add
91-
git commit -m 'comments'
92-
git push
93-
```
120+
121+
!!! tip "Pro tip"
122+
If you don't have a metadata file for your images, use the [sum-buddy package](Helpful-Tools-for-your-Workflow.md#sum-buddy) to generate one in your local file system. This can also be used as a metadata file for the dataset viewer as needed (see [image datasets docs](https://huggingface.co/docs/hub/en/datasets-image) for more information on setting this up). Similar options are available for [audio](https://huggingface.co/docs/hub/en/datasets-audio) and [video](https://huggingface.co/docs/hub/en/datasets-video) datasets.

‎docs/wiki-guide/images/HF-dataset-upload/346190430-9e6cef9b-18ef-4d4a-84c5-1a3f75ac9336.png‎

-71.3 KB
Loading

0 commit comments

Comments
 (0)