docs: cross-link Nemotron-CC recipe as a production NeMo Curator example

[arhamm1]

Summary

• Adds a Real-World Recipe callout in README.md under the "Why NeMo Curator?" section, linking to the Nemotron-CC pipeline
• Adds a Recipes row to the Learn More table in README.md
• Adds a new Production Recipes section in tutorials/README.md
• Closes Docs - Add how Curator is used to build Nemotron datasets #1548.

Context

The Nemotron-CC curation pipeline is a complete end-to-end example of NeMo Curator used at production scale — covering Common Crawl extraction, exact/fuzzy/substring deduplication, ensemble quality classification, and LLM-based synthetic data generation. Cross-linking it here helps users discover a real-world reference implementation of the library.

🤖 Generated with Claude Code

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[greptile-apps]

Greptile Summary

This documentation-only PR cross-links the Nemotron-CC production curation pipeline from both README.md and tutorials/README.md, adding a "Real-World Recipe" callout, a Recipes row in the Learn More table, and a new "Production Recipes" section. The previously raised concern about missing in-repo tutorial links has been addressed — all new entries now point to both the external NVIDIA-NeMo/Nemotron pipeline and the in-repo tutorials/synthetic/nemotron_cc/ tutorial.

Confidence Score: 5/5

Safe to merge — documentation-only changes with correct relative paths and no code impact.

All changes are documentation (markdown only). Relative links to tutorials/synthetic/nemotron_cc/ resolve correctly from both files. The previously flagged concern about missing in-repo links is fully addressed. No logic, security, or runtime issues are possible in this change.

No files require special attention.

Important Files Changed

Filename	Overview
README.md	Adds a "Real-World Recipe" callout and a Recipes row to the Learn More table, both linking to the external Nemotron pipeline and the in-repo SDG tutorial; relative paths are correct.
tutorials/README.md	Adds a "Production Recipes" section with a table row linking to the external Nemotron repo and the in-repo synthetic/nemotron_cc/ tutorial; relative path resolves correctly from the tutorials/ directory.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[README.md<br/>Why NeMo Curator?] -->|Real-World Recipe callout| B[External: NVIDIA-NeMo/Nemotron<br/>nemotron-cc pipeline]
    A -->|Real-World Recipe callout| C[In-repo: tutorials/synthetic/nemotron_cc/]

    D[README.md<br/>Learn More table — Recipes row] -->|link| B
    D -->|link| C

    E[tutorials/README.md<br/>Production Recipes section] -->|Nemotron-CC link| B
    E -->|SDG tutorial link| C

    C --> F[nemotron_cc_pipelines.py<br/>nemotron_cc_sdg_*.py]

Loading

_{Reviews (6): Last reviewed commit: "Merge branch 'main' into docs/crosslink-..." | Re-trigger Greptile}

[greptile-apps]

New section misses in-repo Nemotron-CC content

The new "Production Recipes" row links exclusively to NVIDIA-NeMo/Nemotron (an external repo), but this repository already contains the Nemotron-CC synthetic data generation pipeline at tutorials/synthetic/nemotron_cc/. The NVIDIA technical blog states the pipeline was merged into NeMo Curator, and tutorials/synthetic/nemotron_cc/nemotron_cc_pipelines.py exists in this repo.

Pointing only at the external recipe means users browsing tutorials/README.md will miss the SDG portion that lives here. Consider adding a link to tutorials/synthetic/nemotron_cc/ alongside (or instead of) the external recipe link, and clarifying which parts of the end-to-end pipeline live where:

Suggested change

	| [Nemotron-CC](https://github.com/NVIDIA-NeMo/Nemotron/tree/main/src/nemotron/recipes/data_curation/nemotron-cc) | Curate Common Crawl snapshots into an LLM-ready dataset, reproducing the [Nemotron-CC datasets](https://huggingface.co/datasets/nvidia/Nemotron-CC-v2) | `CommonCrawlDownloadExtractStage` • Exact/Fuzzy/Substring Dedup • `FineWebNemotronEduClassifier` • Synthetic Data Generation |
	| [Nemotron-CC (SDG)](tutorials/synthetic/nemotron_cc/) | Synthetic data generation stage of the Nemotron-CC pipeline | `nemotron_cc_pipelines.py` • Diverse QA • Knowledge List |
	| [Nemotron-CC (full curation)](https://github.com/NVIDIA-NeMo/Nemotron/tree/main/src/nemotron/recipes/data_curation/nemotron-cc) | Curate Common Crawl snapshots into an LLM-ready dataset, reproducing the [Nemotron-CC datasets](https://huggingface.co/datasets/nvidia/Nemotron-CC-v2) | `CommonCrawlDownloadExtractStage` • Exact/Fuzzy/Substring Dedup • `FineWebNemotronEduClassifier` • Synthetic Data Generation |

[greptile-apps]

Learn More table link points to external repo, not canonical in-repo location

The Recipes row links to NVIDIA-NeMo/Nemotron (external), but this repo already hosts the Nemotron-CC SDG pipeline at tutorials/synthetic/nemotron_cc/. Users clicking from the main README.md learn-more table would be sent off-site when a directly relevant tutorial lives in the same repo. Consider adding a link to the internal tutorial alongside the external recipe:

Suggested change

	| **Recipes** | [Nemotron-CC: end-to-end web data curation](https://github.com/NVIDIA-NeMo/Nemotron/tree/main/src/nemotron/recipes/data_curation/nemotron-cc) |
	| **Recipes** | [Nemotron-CC SDG (in-repo)](tutorials/synthetic/nemotron_cc/) • [Nemotron-CC full pipeline](https://github.com/NVIDIA-NeMo/Nemotron/tree/main/src/nemotron/recipes/data_curation/nemotron-cc) |

[greptile-apps]

Tip:

Greploops — Automatically fix all review issues by running /greploops in Claude Code. It iterates: fix, push, re-review, repeat until 5/5 confidence.

Use the Greptile plugin for Claude Code to query reviews, search comments, and manage custom context directly from your terminal.

[sarahyurick]

Not sure how detailed you want this description to be, but technically all of the stages are:

1. Download and extract Common Crawl
2. Language identification and filtering
3. Exact deduplication
4. Fuzzy deduplication
5. Substring deduplication
6. Ensemble quality classification using 3 classifiers (1 fasttext and 2 FineWeb-based)
7. SDG (4 generation tasks)

[sarahyurick]

/ok to test d504b64

[sarahyurick]

/ok to test 14b7b7e