Add Jobs guide: Process Large Datasets

[davanstrien]

What

Adds a new Jobs documentation page — Process Large Datasets (docs/hub/jobs-large-datasets.md) — under the Jobs section of the nav, plus an inbound pointer from jobs-configuration's Volumes section.

It's a task-oriented guide for working with datasets larger than a Job's ephemeral disk, opening with a short decision rule and then covering each approach:

• Stream with datasets (streaming=True) — links the Stream guide and the 100× more efficient blog
• Read & filter over hf:// directly with Polars/DuckDB/pandas (native readers, no mount)
• Mount a dataset/model/bucket for tools that expect local file paths
• Save results to a Storage Bucket (DuckDB COPY straight from hf:// to the mount) or push_to_hub
• A Common Crawl worked example: stream a WET shard straight from hf:// → DuckDB, no download

It links the canonical reference pages (jobs-pricing, jobs-configuration#volumes, storage-buckets-access, datasets/stream, per-library dataset pages) rather than restating them.

Verification

Every snippet was run against live Jobs. Timings that shaped the page (same aggregate query, same ~28 GB / 14-file Parquet glob):

Read path	Time
Polars native hf:// scan	~4 min (in the doc)
DuckDB native hf://	~18 min
DuckDB via register_filesystem(HfFileSystem())	did not finish (killed at 30-min default timeout)
Polars over a -v repo mount	~6 min per file (~85 min extrapolated)

Hence the page's structure: hf:// native readers for scans, mounts for whole-file/local-path access, explicit --timeout guidance for network-bound scans. The Common Crawl example runs verbatim in ~1 min on the default CPU flavor with no token; its printed output is included on the page.

Notes

• Add Serve Models guide for Jobs (vLLM + llama.cpp via exposed ports) #2554 (jobs-serving) touched the same _toctree.yml region and has since merged; this branch is mergeable and rebases cleanly on main (no conflicts).

---

Note

Low Risk
Documentation-only changes with no runtime or API impact.

Overview
Adds a new Jobs doc page Process Large Datasets and registers it in the Hub nav (after Examples & Tutorials).

The guide explains how to work with data bigger than a Job’s ephemeral disk: a short “which approach?” decision tree, then sections on streaming (datasets + Spark), native hf:// scans (Polars/DuckDB/pandas) with --timeout and parallel-job guidance, volume mounts for path-based tools, persisting output to Storage Buckets or push_to_hub, and a Common Crawl WET end-to-end example. It points to existing pricing, configuration, bucket, and per-library docs instead of duplicating them.

Configuration gains a Volumes tip that lazy mounts enable oversized datasets and links to this page.

^{Reviewed by Cursor Bugbot for commit 7cd203e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[davanstrien]

IMO, we can keep iterating on this as libraries mature, but nice to document some patterns already.

[lhoestq]

lgtm ! I added some minor suggestions