[Datasets] Add PyArrow docs#1839
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
nice! |
| pq.write_table(table_test , "hf://datasets/username/my_dataset/test.parquet", use_content_defined_chunking=True) | ||
| ``` | ||
|
|
||
| We use `use_content_defined_chunking=True` to enable faster uploads and downloads from Hugging Face thanks to Xet deduplication (it requires `pyarrow>=21.0`). |
There was a problem hiding this comment.
do you think we could ask them to turn it on by default?
There was a problem hiding this comment.
Maybe in case of a hf:// URI but that could be fragile to implement since filesystem objects can be passed as well.
I would rather consider the flavor argument https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html
pq.write_table(table_test , "hf://datasets/username/my_dataset/test.parquet", flavor="huggingface")There was a problem hiding this comment.
Also note that CDC has a slight performance overhead, so turning it on by default would make sense if we can infer that the targeted filesystem can use it for deduplication.
davanstrien
left a comment
davanstrien
left a comment
There was a problem hiding this comment.
Very cool! Think this could also be helpful for people already heavily using Arrow for science datasets to understand how to get them on the Hub.
|
|
||
| To load a file from Hugging Face, the path needs to start with `hf://`. For example, the path to the [stanfordnlp/imdb](https://huggingface.co/datasets/stanfordnlp/imdb) dataset repository is `hf://datasets/stanfordnlp/imdb`. The dataset on Hugging Face contains multiple Parquet files. The Parquet file format is designed to make reading and writing data frames efficient, and to make sharing data across data analysis languages easy. Here is how to load the file `plain_text/train-00000-of-00001.parquet` as a pyarrow Table (it requires `pyarrow>=21.0`): | ||
|
|
||
| ```python |
There was a problem hiding this comment.
maybe
| ```python | |
| ```pycon |
(see https://pygments.org/docs/lexers/#pygments.lexers.python.PythonConsoleLexer for example)
There was a problem hiding this comment.
wait what Oo
I might keep it that way for now for consistency and update the whole docs later if it's really a thing :o
Co-authored-by: Daniel van Strien <davanstrien@users.noreply.github.com> Co-authored-by: Julien Chaumond <julien@huggingface.co>
|
thanks for the reviews :) I took your comments into account |
6 participants
pyarrow21 will be out soon and has an official HF integration :)It also includes Parquet CDC for efficient Xet deduplication for datasets
cc @kszucs