From dffa86c937cdb436e042ea401fb446c1b45c6d93 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Tue, 2 Jun 2026 15:39:00 +0100 Subject: [PATCH 01/10] Add Process Large Datasets guide for Jobs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New docs/hub/jobs-large-datasets.md: how to read, filter, and process datasets larger than a Job's ephemeral disk — streaming, mounting + lazy reads, reading/filtering over hf:// with Polars/DuckDB, saving results to a bucket, and a Common Crawl worked example. Links canonical pages rather than restating them; adds an inbound pointer from jobs-configuration#volumes. --- docs/hub/_toctree.yml | 2 + docs/hub/jobs-configuration.md | 3 + docs/hub/jobs-large-datasets.md | 127 ++++++++++++++++++++++++++++++++ 3 files changed, 132 insertions(+) create mode 100644 docs/hub/jobs-large-datasets.md diff --git a/docs/hub/_toctree.yml b/docs/hub/_toctree.yml index 135e60cec1..6d74408be0 100644 --- a/docs/hub/_toctree.yml +++ b/docs/hub/_toctree.yml @@ -452,6 +452,8 @@ title: Popular Images - local: jobs-examples title: Examples & Tutorials + - local: jobs-large-datasets + title: Process Large Datasets - local: jobs-schedule title: Schedule Jobs - local: jobs-webhooks diff --git a/docs/hub/jobs-configuration.md b/docs/hub/jobs-configuration.md index a83bc32e1b..be33e0aeea 100644 --- a/docs/hub/jobs-configuration.md +++ b/docs/hub/jobs-configuration.md @@ -105,6 +105,9 @@ You can pass environment variables to your job using Mount Hugging Face repositories (models, datasets) or [Storage Buckets](./storage-buckets) as volumes in your job container using `-v` or `--volume`. The syntax uses the `hf://` URL scheme: `hf://[TYPE/]SOURCE:/MOUNT_PATH[:ro]`. +> [!TIP] +> Because mounted files are fetched lazily, mounting lets a Job work with datasets far larger than its local disk. See [Process Large Datasets](./jobs-large-datasets) for mounting, streaming, and processing big data on Jobs. + Volume types: | Type | Example | diff --git a/docs/hub/jobs-large-datasets.md b/docs/hub/jobs-large-datasets.md new file mode 100644 index 0000000000..0d21808b52 --- /dev/null +++ b/docs/hub/jobs-large-datasets.md @@ -0,0 +1,127 @@ +# Process large datasets + +Every Job comes with a fixed amount of local disk, set by its [hardware flavor](./jobs-pricing#pricing) (the **Ephemeral Storage** column). That's plenty for many tasks — but you don't need to fit a dataset on disk to work with it. To use datasets larger than the disk, read them directly from the Hub. + +**Which approach?** + +- **Fits on disk** → a plain `load_dataset(...)` works as-is. +- **Iterating over rows, or training** → [stream](#stream-the-dataset) it — no download. +- **Filtered or column-pruned scans** → [mount](#mount-a-dataset-model-or-bucket) the repo and read it lazily, or query it [directly over `hf://`](#read-and-filter-over-hf) with Polars or DuckDB. +- **Persisting results** → write them to a [Storage Bucket](#saving-results) so they survive the Job. + +## Stream the dataset + +Streaming reads examples from the Hub as your code consumes them — no download, no local copy, and it scales to multi-TB datasets. Recent releases made it [up to 100× more efficient](https://huggingface.co/blog/streaming-datasets), reaching performance on par with local SSDs when training across many workers: + +```python +from datasets import load_dataset + +ds = load_dataset("HuggingFaceFW/fineweb-edu", "sample-10BT", split="train", streaming=True) +for example in ds.take(1000): + ... # streams in as you iterate, nothing hits disk +``` + +A streamed dataset is an `IterableDataset` supporting lazy `.filter()`, `.map()`, `.shuffle(buffer_size=...)`, and `.batch()`, and can be passed straight to a PyTorch `DataLoader` or a `Trainer` to train on data larger than disk. See the [Stream guide](/docs/datasets/stream) for the full API, and [Examples & Tutorials](./jobs-examples) for end-to-end training walkthroughs. + +## Mount a dataset, model, or bucket + +Mount a repository or bucket into the Job as a local path with `-v` / `--volume`; files are fetched lazily over the network as your code reads them, so any tool that reads local files just works: + +```bash +hf jobs uv run --flavor cpu-upgrade \ + -v hf://datasets/HuggingFaceFW/fineweb-edu:/mnt/data \ + filter.py +``` + +```python +# filter.py — lazy scan: reads Parquet metadata first, then only the columns/rows the query needs +import polars as pl + +df = ( + pl.scan_parquet("/mnt/data/sample/10BT/*.parquet") + .filter(pl.col("int_score") >= 4) + .select("text", "url", "token_count") + .collect() +) +``` + +Datasets and models mount read-only; buckets are read-write, which makes them a good place to [save results](#saving-results). See [Configuration](./jobs-configuration#volumes) for the full `-v` syntax and [bucket access patterns](./storage-buckets-access#volume-mounts-in-jobs-and-spaces) for details. + +> [!TIP] +> Files read through a mount are cached on the Job's ephemeral disk, so reading lazily (a `scan_parquet`, or one file at a time) keeps the footprint small. When running a local script with `hf jobs uv run`, the script directory is mounted at `/data`, so mount your data elsewhere (e.g. `/mnt/data`). + +## Read and filter over `hf://` + +Many data libraries read datasets and buckets straight from the Hub over `hf://` (via [`HfFileSystem`](/docs/huggingface_hub/guides/hf_file_system)/fsspec) — no mount needed. **Polars**, **DuckDB**, and **pandas** all read Hub Parquet directly, pushing filters and column selection down into the scan, so a single Job can process far more data than fits in memory. For example, DuckDB can filter the source and write the result to a mounted bucket in one query: + +```python +import duckdb +from huggingface_hub import HfFileSystem + +duckdb.register_filesystem(HfFileSystem()) +duckdb.sql( + """ + COPY ( + SELECT text, url, token_count + FROM 'hf://datasets/HuggingFaceFW/fineweb-edu/sample/100BT/*.parquet' + WHERE int_score >= 4 AND token_count >= 4000 + ) TO '/mnt/out/result.parquet' (FORMAT parquet) + """ +) +``` + +See [Python data tools](./storage-buckets-access#python-data-tools) and [bucket integrations](./storage-buckets-integrations) for per-library examples. A large scan is network-bound rather than memory-bound, so to go faster you can split the work across several Jobs running in parallel. + +## Saving results + +Ephemeral disk doesn't survive the Job, so write anything you want to keep to a [Storage Bucket](./storage-buckets) mounted read-write, or push it to the Hub as a dataset: + +```bash +hf jobs uv run --flavor cpu-performance \ + -v hf://datasets/HuggingFaceFW/fineweb-edu:/mnt/data \ + -v hf://buckets/username/my-output:/mnt/out \ + filter.py +``` + +Files written under the bucket mount path persist after the Job ends. To publish a processed dataset instead, use [`Dataset.push_to_hub`](/docs/datasets/upload_dataset). + +## Worked example: query Common Crawl without downloading it + +Common Crawl mirrors its archive to the bucket [`commoncrawl/commoncrawl`](https://huggingface.co/datasets/commoncrawl/commoncrawl) — hundreds of TB. Stream one WET (plaintext) shard straight from `hf://`, parse it, and query it with DuckDB; only a few MB transit because the gzip is read sequentially and stopped early: + +```python +# /// script +# requires-python = ">=3.11" +# dependencies = ["huggingface_hub>=1.0", "fastwarc>=0.15", "duckdb>=1.0"] +# /// +import duckdb +from fastwarc.warc import ArchiveIterator, WarcRecordType +from huggingface_hub import HfFileSystem + +WET = ( + "buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17/" + "segments/1775805908305.14/wet/" + "CC-MAIN-20260410081153-20260410111153-00000.warc.wet.gz" +) + +rows = [] +with HfFileSystem().open(WET, "rb") as f: + for rec in ArchiveIterator(f, record_types=WarcRecordType.conversion): + lang = (rec.headers.get("WARC-Identified-Content-Language", "") or "und").split(",")[0] + rows.append((rec.headers.get("WARC-Target-URI", ""), lang, len(rec.reader.read()))) + if len(rows) >= 5000: + break + +con = duckdb.connect() +con.execute("CREATE TABLE wet(url VARCHAR, lang VARCHAR, n_chars BIGINT)") +con.executemany("INSERT INTO wet VALUES (?,?,?)", rows) +con.sql("SELECT lang, count(*) AS docs FROM wet GROUP BY lang ORDER BY docs DESC LIMIT 10").show() +``` + +Run it with `hf jobs uv run cc_wet.py`. + +## See also + +- [Stream](/docs/datasets/stream) · [Streaming datasets: 100× more efficient](https://huggingface.co/blog/streaming-datasets) +- [Pricing & hardware](./jobs-pricing#pricing) — ephemeral disk per flavor · [Configuration](./jobs-configuration#volumes) — volumes +- [Storage Buckets](./storage-buckets) · [access patterns](./storage-buckets-access) · [integrations](./storage-buckets-integrations) From a379bf9a9754decde362cb29b8a54ca7e19943e5 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Tue, 2 Jun 2026 15:41:46 +0100 Subject: [PATCH 02/10] small edits --- docs/hub/jobs-large-datasets.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/hub/jobs-large-datasets.md b/docs/hub/jobs-large-datasets.md index 0d21808b52..bd149eb420 100644 --- a/docs/hub/jobs-large-datasets.md +++ b/docs/hub/jobs-large-datasets.md @@ -5,7 +5,7 @@ Every Job comes with a fixed amount of local disk, set by its [hardware flavor]( **Which approach?** - **Fits on disk** → a plain `load_dataset(...)` works as-is. -- **Iterating over rows, or training** → [stream](#stream-the-dataset) it — no download. +- **Iterating over rows, or training** → [stream](#stream-the-dataset) it. - **Filtered or column-pruned scans** → [mount](#mount-a-dataset-model-or-bucket) the repo and read it lazily, or query it [directly over `hf://`](#read-and-filter-over-hf) with Polars or DuckDB. - **Persisting results** → write them to a [Storage Bucket](#saving-results) so they survive the Job. @@ -52,7 +52,7 @@ Datasets and models mount read-only; buckets are read-write, which makes them a ## Read and filter over `hf://` -Many data libraries read datasets and buckets straight from the Hub over `hf://` (via [`HfFileSystem`](/docs/huggingface_hub/guides/hf_file_system)/fsspec) — no mount needed. **Polars**, **DuckDB**, and **pandas** all read Hub Parquet directly, pushing filters and column selection down into the scan, so a single Job can process far more data than fits in memory. For example, DuckDB can filter the source and write the result to a mounted bucket in one query: +Many data libraries read datasets and buckets straight from the Hub over `hf://` (via [`HfFileSystem`](/docs/huggingface_hub/guides/hf_file_system)/fsspec) with no mount needed. **Polars**, **DuckDB**, and **pandas** all read Hub Parquet directly, pushing filters and column selection down into the scan, so a single Job can process far more data than fits in memory. For example, DuckDB can filter the source and write the result to a mounted bucket in one query: ```python import duckdb From c5d79ef72f4d5093d7e26e1a894961c17a242a23 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Fri, 12 Jun 2026 12:00:09 +0300 Subject: [PATCH 03/10] Verify snippets on live Jobs; rework hf:// and mount sections from results Every example was run against real Jobs. Key changes from what testing showed: - hf:// section: native Polars scan (verified ~4 min for the 28 GB sample aggregate on the default CPU flavor) replaces the register_filesystem(HfFileSystem()) DuckDB example, which could not finish the same scan within the 30-minute default timeout. Native hf:// stays for datasets; HfFileSystem noted for buckets. - Mount section: reframed for whole-file/local-path access; large Parquet scans pointed at hf:// instead (measured several times slower through the mount). Added the missing PEP 723 header to the script example. - Save results: hosts the DuckDB COPY-to-bucket pattern (out-of-core write), scaled from 100BT to 10BT, with --timeout shown and a note that it transfers the full text column. - Added --timeout guidance (long scans are network-bound), real output for the Common Crawl example (verified verbatim, ~1 min, no token needed), hf jobs hardware mention, and Title Case/heading consistency. Co-Authored-By: Claude Fable 5 --- docs/hub/jobs-large-datasets.md | 106 ++++++++++++++++++++++---------- 1 file changed, 72 insertions(+), 34 deletions(-) diff --git a/docs/hub/jobs-large-datasets.md b/docs/hub/jobs-large-datasets.md index bd149eb420..06c68250e6 100644 --- a/docs/hub/jobs-large-datasets.md +++ b/docs/hub/jobs-large-datasets.md @@ -1,13 +1,14 @@ -# Process large datasets +# Process Large Datasets -Every Job comes with a fixed amount of local disk, set by its [hardware flavor](./jobs-pricing#pricing) (the **Ephemeral Storage** column). That's plenty for many tasks — but you don't need to fit a dataset on disk to work with it. To use datasets larger than the disk, read them directly from the Hub. +Every Job comes with a fixed amount of local disk, set by its [hardware flavor](./jobs-pricing#pricing) (the **Ephemeral Storage** column, also shown by `hf jobs hardware`). That's plenty for many tasks — but you don't need to fit a dataset on disk to work with it. To use datasets larger than the disk, read them directly from the Hub. **Which approach?** -- **Fits on disk** → a plain `load_dataset(...)` works as-is. +- **Fits on disk** → a plain `load_dataset(...)` works as-is (or just pick a bigger flavor — up to 1 TB of ephemeral disk). - **Iterating over rows, or training** → [stream](#stream-the-dataset) it. -- **Filtered or column-pruned scans** → [mount](#mount-a-dataset-model-or-bucket) the repo and read it lazily, or query it [directly over `hf://`](#read-and-filter-over-hf) with Polars or DuckDB. -- **Persisting results** → write them to a [Storage Bucket](#saving-results) so they survive the Job. +- **Filtered or column-pruned scans** → query it [directly over `hf://`](#read-and-filter-over-hf) with Polars or DuckDB. +- **Tools that expect local file paths** → [mount](#mount-a-dataset-model-or-bucket) the repo and read it lazily. +- **Persisting results** → write them to a [Storage Bucket](#save-results) so they survive the Job. ## Stream the dataset @@ -23,6 +24,26 @@ for example in ds.take(1000): A streamed dataset is an `IterableDataset` supporting lazy `.filter()`, `.map()`, `.shuffle(buffer_size=...)`, and `.batch()`, and can be passed straight to a PyTorch `DataLoader` or a `Trainer` to train on data larger than disk. See the [Stream guide](/docs/datasets/stream) for the full API, and [Examples & Tutorials](./jobs-examples) for end-to-end training walkthroughs. +## Read and filter over `hf://` + +Many data libraries read Hub datasets directly over `hf://` paths — **Polars**, **DuckDB**, and **pandas** all scan Hub Parquet natively, pushing filters and column selection down into the scan, so a single Job can process far more data than fits in memory or on disk. This query summarizes ~28 GB of Parquet in about four minutes on the default CPU flavor: + +```python +import polars as pl + +agg = ( + pl.scan_parquet("hf://datasets/HuggingFaceFW/fineweb-edu/sample/10BT/*.parquet") + .filter(pl.col("int_score") >= 4) + .group_by("int_score") + .agg(pl.len().alias("docs"), pl.col("token_count").sum().alias("tokens")) + .sort("int_score") + .collect() +) +print(agg) +``` + +Scans like this are network-bound rather than memory-bound, and engines differ in how aggressively they parallelize remote reads, so timings vary by library. Two practical consequences: set `--timeout` above the default 30 minutes for long scans, and split the work across several Jobs running in parallel to go faster. See [Polars](./datasets-polars), [DuckDB](./datasets-duckdb), and [pandas](./datasets-pandas) for per-library examples, and [Python data tools](./storage-buckets-access#python-data-tools) for reading **buckets** over `hf://` (which goes through [`HfFileSystem`](/docs/huggingface_hub/guides/hf_file_system)). + ## Mount a dataset, model, or bucket Mount a repository or bucket into the Job as a local path with `-v` / `--volume`; files are fetched lazily over the network as your code reads them, so any tool that reads local files just works: @@ -30,59 +51,58 @@ Mount a repository or bucket into the Job as a local path with `-v` / `--volume` ```bash hf jobs uv run --flavor cpu-upgrade \ -v hf://datasets/HuggingFaceFW/fineweb-edu:/mnt/data \ - filter.py + process.py ``` ```python -# filter.py — lazy scan: reads Parquet metadata first, then only the columns/rows the query needs +# /// script +# dependencies = ["polars"] +# /// +# process.py — the mounted repo is just a directory of files +from pathlib import Path + import polars as pl -df = ( - pl.scan_parquet("/mnt/data/sample/10BT/*.parquet") - .filter(pl.col("int_score") >= 4) - .select("text", "url", "token_count") - .collect() -) +for path in Path("/mnt/data/sample/10BT").glob("*.parquet"): + shard = pl.read_parquet(path) + ... # process one shard at a time, write results out ``` -Datasets and models mount read-only; buckets are read-write, which makes them a good place to [save results](#saving-results). See [Configuration](./jobs-configuration#volumes) for the full `-v` syntax and [bucket access patterns](./storage-buckets-access#volume-mounts-in-jobs-and-spaces) for details. +Mounting is the natural fit when files are consumed whole — model weights, audio or image files, archives — or when a tool only accepts file paths. For large multi-file Parquet scans, querying [directly over `hf://`](#read-and-filter-over-hf) is typically several times faster than scanning through a mount. + +Datasets and models mount read-only; buckets are read-write, which makes them a good place to [save results](#save-results). See [Configuration](./jobs-configuration#volumes) for the full `-v` syntax and [bucket access patterns](./storage-buckets-access#volume-mounts-in-jobs-and-spaces) for details. > [!TIP] -> Files read through a mount are cached on the Job's ephemeral disk, so reading lazily (a `scan_parquet`, or one file at a time) keeps the footprint small. When running a local script with `hf jobs uv run`, the script directory is mounted at `/data`, so mount your data elsewhere (e.g. `/mnt/data`). +> Files read through a mount are cached on the Job's ephemeral disk, so reading lazily (one file at a time) keeps the footprint small. When running a local script with `hf jobs uv run`, the script directory is mounted at `/data`, so mount your data elsewhere (e.g. `/mnt/data`). -## Read and filter over `hf://` +## Save results -Many data libraries read datasets and buckets straight from the Hub over `hf://` (via [`HfFileSystem`](/docs/huggingface_hub/guides/hf_file_system)/fsspec) with no mount needed. **Polars**, **DuckDB**, and **pandas** all read Hub Parquet directly, pushing filters and column selection down into the scan, so a single Job can process far more data than fits in memory. For example, DuckDB can filter the source and write the result to a mounted bucket in one query: +Ephemeral disk doesn't survive the Job, so write anything you want to keep to a [Storage Bucket](./storage-buckets) mounted read-write, or push it to the Hub as a dataset. DuckDB can filter the source over `hf://` and persist the matches to a mounted bucket in one query — and because it writes out-of-core, the result doesn't need to fit in memory. A filter like this reads the full `text` column, so expect it to take a while (it transfers most of the ~28 GB): + +```bash +hf jobs uv run --flavor cpu-upgrade --timeout 1h \ + -v hf://buckets/username/my-output:/mnt/out \ + filter.py +``` ```python +# /// script +# dependencies = ["duckdb"] +# /// +# filter.py — scan ~28 GB of Parquet, keep only the matching rows import duckdb -from huggingface_hub import HfFileSystem -duckdb.register_filesystem(HfFileSystem()) duckdb.sql( """ COPY ( SELECT text, url, token_count - FROM 'hf://datasets/HuggingFaceFW/fineweb-edu/sample/100BT/*.parquet' + FROM 'hf://datasets/HuggingFaceFW/fineweb-edu/sample/10BT/*.parquet' WHERE int_score >= 4 AND token_count >= 4000 ) TO '/mnt/out/result.parquet' (FORMAT parquet) """ ) ``` -See [Python data tools](./storage-buckets-access#python-data-tools) and [bucket integrations](./storage-buckets-integrations) for per-library examples. A large scan is network-bound rather than memory-bound, so to go faster you can split the work across several Jobs running in parallel. - -## Saving results - -Ephemeral disk doesn't survive the Job, so write anything you want to keep to a [Storage Bucket](./storage-buckets) mounted read-write, or push it to the Hub as a dataset: - -```bash -hf jobs uv run --flavor cpu-performance \ - -v hf://datasets/HuggingFaceFW/fineweb-edu:/mnt/data \ - -v hf://buckets/username/my-output:/mnt/out \ - filter.py -``` - Files written under the bucket mount path persist after the Job ends. To publish a processed dataset instead, use [`Dataset.push_to_hub`](/docs/datasets/upload_dataset). ## Worked example: query Common Crawl without downloading it @@ -118,7 +138,25 @@ con.executemany("INSERT INTO wet VALUES (?,?,?)", rows) con.sql("SELECT lang, count(*) AS docs FROM wet GROUP BY lang ORDER BY docs DESC LIMIT 10").show() ``` -Run it with `hf jobs uv run cc_wet.py`. +Run it with `hf jobs uv run cc_wet.py` — it completes in about a minute on the default CPU flavor and prints: + +``` +┌─────────┬───────┐ +│ lang │ docs │ +│ varchar │ int64 │ +├─────────┼───────┤ +│ eng │ 1974 │ +│ zho │ 586 │ +│ rus │ 434 │ +│ jpn │ 244 │ +│ spa │ 239 │ +│ fra │ 194 │ +│ deu │ 179 │ +│ por │ 145 │ +│ pol │ 98 │ +│ ita │ 79 │ +└─────────┴───────┘ +``` ## See also From e952f0238ffffc0636f2d0f219c369febabc1a6c Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Tue, 7 Jul 2026 08:57:11 +0100 Subject: [PATCH 04/10] Fix Common Crawl link: use buckets path, not datasets MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit commoncrawl/commoncrawl is a Storage Bucket, not a dataset — the datasets/ URL returns 401. Matches the buckets/ path already used in the worked example just below. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/hub/jobs-large-datasets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/jobs-large-datasets.md b/docs/hub/jobs-large-datasets.md index 06c68250e6..4e9956fc05 100644 --- a/docs/hub/jobs-large-datasets.md +++ b/docs/hub/jobs-large-datasets.md @@ -107,7 +107,7 @@ Files written under the bucket mount path persist after the Job ends. To publish ## Worked example: query Common Crawl without downloading it -Common Crawl mirrors its archive to the bucket [`commoncrawl/commoncrawl`](https://huggingface.co/datasets/commoncrawl/commoncrawl) — hundreds of TB. Stream one WET (plaintext) shard straight from `hf://`, parse it, and query it with DuckDB; only a few MB transit because the gzip is read sequentially and stopped early: +Common Crawl mirrors its archive to the bucket [`commoncrawl/commoncrawl`](https://huggingface.co/buckets/commoncrawl/commoncrawl) — hundreds of TB. Stream one WET (plaintext) shard straight from `hf://`, parse it, and query it with DuckDB; only a few MB transit because the gzip is read sequentially and stopped early: ```python # /// script From ae648574824fe626798079e9c32f4ea269ae8e67 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Tue, 7 Jul 2026 12:18:39 +0100 Subject: [PATCH 05/10] Tighten intro + Save results wording, trim Common Crawl table Editorial polish after review: rewrite the intro (cut repetition, name the read paths), tighten the Save-results paragraph (drop a caveat already made in the Read/filter section), and truncate the Common Crawl output table to the top languages. Every code example re-verified against live Jobs. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/hub/jobs-large-datasets.md | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/docs/hub/jobs-large-datasets.md b/docs/hub/jobs-large-datasets.md index 4e9956fc05..e91464fc07 100644 --- a/docs/hub/jobs-large-datasets.md +++ b/docs/hub/jobs-large-datasets.md @@ -1,6 +1,6 @@ # Process Large Datasets -Every Job comes with a fixed amount of local disk, set by its [hardware flavor](./jobs-pricing#pricing) (the **Ephemeral Storage** column, also shown by `hf jobs hardware`). That's plenty for many tasks — but you don't need to fit a dataset on disk to work with it. To use datasets larger than the disk, read them directly from the Hub. +Every Job comes with a fixed amount of local disk, set by its [hardware flavor](./jobs-pricing#pricing) (the **Ephemeral Storage** column, also shown by `hf jobs hardware`). You don't need to fit a whole dataset on that disk to work with it: datasets and [Storage Buckets](./storage-buckets) can be read directly from the Hub — streamed, queried, or mounted — so a single Job can process data far larger than its disk. This page covers the options and when to reach for each. **Which approach?** @@ -77,7 +77,7 @@ Datasets and models mount read-only; buckets are read-write, which makes them a ## Save results -Ephemeral disk doesn't survive the Job, so write anything you want to keep to a [Storage Bucket](./storage-buckets) mounted read-write, or push it to the Hub as a dataset. DuckDB can filter the source over `hf://` and persist the matches to a mounted bucket in one query — and because it writes out-of-core, the result doesn't need to fit in memory. A filter like this reads the full `text` column, so expect it to take a while (it transfers most of the ~28 GB): +Ephemeral disk doesn't survive the Job, so write anything you want to keep to a [Storage Bucket](./storage-buckets) mounted read-write, or push it to the Hub as a dataset. DuckDB can filter the source over `hf://` and write the matches straight to a mounted bucket in one out-of-core query, so the result never has to fit in memory: ```bash hf jobs uv run --flavor cpu-upgrade --timeout 1h \ @@ -149,12 +149,7 @@ Run it with `hf jobs uv run cc_wet.py` — it completes in about a minute on the │ zho │ 586 │ │ rus │ 434 │ │ jpn │ 244 │ -│ spa │ 239 │ -│ fra │ 194 │ -│ deu │ 179 │ -│ por │ 145 │ -│ pol │ 98 │ -│ ita │ 79 │ +│ … │ … │ └─────────┴───────┘ ``` From 8c304ce2dfe3005e1f7daa86ed7f9aab9fca6f64 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Tue, 7 Jul 2026 12:29:41 +0100 Subject: [PATCH 06/10] Use a heading for 'Which approach?' to match house style Sibling Jobs pages use ## headings, not bold pseudo-headings. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/hub/jobs-large-datasets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/jobs-large-datasets.md b/docs/hub/jobs-large-datasets.md index e91464fc07..9e79527ad0 100644 --- a/docs/hub/jobs-large-datasets.md +++ b/docs/hub/jobs-large-datasets.md @@ -2,7 +2,7 @@ Every Job comes with a fixed amount of local disk, set by its [hardware flavor](./jobs-pricing#pricing) (the **Ephemeral Storage** column, also shown by `hf jobs hardware`). You don't need to fit a whole dataset on that disk to work with it: datasets and [Storage Buckets](./storage-buckets) can be read directly from the Hub — streamed, queried, or mounted — so a single Job can process data far larger than its disk. This page covers the options and when to reach for each. -**Which approach?** +## Which approach? - **Fits on disk** → a plain `load_dataset(...)` works as-is (or just pick a bigger flavor — up to 1 TB of ephemeral disk). - **Iterating over rows, or training** → [stream](#stream-the-dataset) it. From fa60db1fcd3a0cf92e0fa0bbcc3588ff434a8b1a Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Thu, 9 Jul 2026 10:01:16 +0100 Subject: [PATCH 07/10] Update docs/hub/jobs-large-datasets.md Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com> --- docs/hub/jobs-large-datasets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/jobs-large-datasets.md b/docs/hub/jobs-large-datasets.md index 9e79527ad0..126508b544 100644 --- a/docs/hub/jobs-large-datasets.md +++ b/docs/hub/jobs-large-datasets.md @@ -5,7 +5,7 @@ Every Job comes with a fixed amount of local disk, set by its [hardware flavor]( ## Which approach? - **Fits on disk** → a plain `load_dataset(...)` works as-is (or just pick a bigger flavor — up to 1 TB of ephemeral disk). -- **Iterating over rows, or training** → [stream](#stream-the-dataset) it. +- **Iterating over rows, processing or training** → [stream](#stream-the-dataset) it. - **Filtered or column-pruned scans** → query it [directly over `hf://`](#read-and-filter-over-hf) with Polars or DuckDB. - **Tools that expect local file paths** → [mount](#mount-a-dataset-model-or-bucket) the repo and read it lazily. - **Persisting results** → write them to a [Storage Bucket](#save-results) so they survive the Job. From e38d6bbe4c63c87b42537bfb14eacef289183427 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Thu, 9 Jul 2026 10:02:07 +0100 Subject: [PATCH 08/10] Update docs/hub/jobs-large-datasets.md Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com> --- docs/hub/jobs-large-datasets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/jobs-large-datasets.md b/docs/hub/jobs-large-datasets.md index 126508b544..716669c30e 100644 --- a/docs/hub/jobs-large-datasets.md +++ b/docs/hub/jobs-large-datasets.md @@ -125,7 +125,7 @@ WET = ( ) rows = [] -with HfFileSystem().open(WET, "rb") as f: +with hffs.open(WET, "rb") as f: for rec in ArchiveIterator(f, record_types=WarcRecordType.conversion): lang = (rec.headers.get("WARC-Identified-Content-Language", "") or "und").split(",")[0] rows.append((rec.headers.get("WARC-Target-URI", ""), lang, len(rec.reader.read()))) From 839d5b03576ec52e0f69580ced5640c45ae7878e Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Thu, 9 Jul 2026 10:02:20 +0100 Subject: [PATCH 09/10] Update docs/hub/jobs-large-datasets.md Co-authored-by: Quentin Lhoest <42851186+lhoestq@users.noreply.github.com> --- docs/hub/jobs-large-datasets.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hub/jobs-large-datasets.md b/docs/hub/jobs-large-datasets.md index 716669c30e..c988d57f75 100644 --- a/docs/hub/jobs-large-datasets.md +++ b/docs/hub/jobs-large-datasets.md @@ -116,7 +116,7 @@ Common Crawl mirrors its archive to the bucket [`commoncrawl/commoncrawl`](https # /// import duckdb from fastwarc.warc import ArchiveIterator, WarcRecordType -from huggingface_hub import HfFileSystem +from huggingface_hub import hffs WET = ( "buckets/commoncrawl/commoncrawl/crawl-data/CC-MAIN-2026-17/" From 7cd203e257e683523c46a6419a7069b723b61b62 Mon Sep 17 00:00:00 2001 From: Daniel van Strien Date: Thu, 9 Jul 2026 10:06:11 +0100 Subject: [PATCH 10/10] Apply remaining review suggestions: Spark distributed streaming, hffs version floor Co-Authored-By: Claude Fable 5 --- docs/hub/jobs-large-datasets.md | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/docs/hub/jobs-large-datasets.md b/docs/hub/jobs-large-datasets.md index c988d57f75..9c51f7624c 100644 --- a/docs/hub/jobs-large-datasets.md +++ b/docs/hub/jobs-large-datasets.md @@ -22,7 +22,17 @@ for example in ds.take(1000): ... # streams in as you iterate, nothing hits disk ``` -A streamed dataset is an `IterableDataset` supporting lazy `.filter()`, `.map()`, `.shuffle(buffer_size=...)`, and `.batch()`, and can be passed straight to a PyTorch `DataLoader` or a `Trainer` to train on data larger than disk. See the [Stream guide](/docs/datasets/stream) for the full API, and [Examples & Tutorials](./jobs-examples) for end-to-end training walkthroughs. +A streamed dataset is an `IterableDataset` supporting lazy `.filter()`, `.map()`, `.shuffle(buffer_size=...)`, and `.batch()`, and can be passed straight to a PyTorch `DataLoader` or a `Trainer` to train on data larger than disk. See the [Stream guide](/docs/datasets/stream) for the full API, and [Examples & Tutorials](./jobs-examples) for end-to-end training walkthroughs for single-node or distributed setups. + +Streaming also works in distributed setups, using Spark: + +```python +import pyspark_huggingface + +df = spark.read.format("huggingface").option("config", "sample-10BT").load("HuggingFaceFW/fineweb-edu") +``` + +The resulting Spark dataframe is distributed: each worker streams from its own subset of files. See the [Spark documentation](./datasets-spark) for examples of reading, writing, and efficiently filtering rows and columns. ## Read and filter over `hf://` @@ -112,7 +122,7 @@ Common Crawl mirrors its archive to the bucket [`commoncrawl/commoncrawl`](https ```python # /// script # requires-python = ">=3.11" -# dependencies = ["huggingface_hub>=1.0", "fastwarc>=0.15", "duckdb>=1.0"] +# dependencies = ["huggingface_hub>=1.9", "fastwarc>=0.15", "duckdb>=1.0"] # /// import duckdb from fastwarc.warc import ArchiveIterator, WarcRecordType