Skip to content

CHANGELOG.md

Latest commit

 

History

History
352 lines (272 loc) · 23.5 KB

CHANGELOG.md

File metadata and controls

352 lines (272 loc) · 23.5 KB

Changelog

NVIDIA NeMo Curator 1.0.0

This major release represents a fundamental architecture shift from Dask to Ray, expanding NeMo Curator to support multimodal data curation with new video and audio capabilities. This refactor enables unified backend processing, better heterogeneous computing support, and enhanced autoscaling for dynamic workloads.

Installation Updates

  • New Docker container: Updated Docker infrastructure with CUDA 12.8.1 and Ubuntu 24.04 base; obtainable through the NGC Catalog (nvcr.io/nvidia/nemo-curator:25.09)

  • Docker file to build own image: Simplified Dockerfile structure for custom container builds with FFmpeg support

  • UV source installations: Integrated UV package manager (v0.8.22) for faster dependency management

  • PyPI improvements: Enhanced PyPI installation with modular extras for targeted functionality:

    Extra Installation Command Description
    All Modalities nemo-curator[all] Complete installation with all modalities and GPU support
    Text Curation nemo-curator[text_cuda12] GPU-accelerated text processing with RAPIDS
    Image Curation nemo-curator[image_cuda12] Image processing with NVIDIA DALI
    Audio Curation nemo-curator[audio_cuda12] Speech recognition with NeMo ASR models
    Video Curation nemo-curator[video_cuda12] Video processing with GPU acceleration
    Basic GPU nemo-curator[cuda12] CUDA utilities without modality-specific dependencies

    All GPU installations require the NVIDIA PyPI index:

    uv pip install --extra-index-url https://pypi.nvidia.com nemo-curator[EXTRA]

New Modalities

Video

NeMo Curator now supports comprehensive video data curation with distributed processing capabilities:

Audio

New audio curation capabilities for speech data processing:

Modality Refactors

Text

  • Ray backend migration: Complete transition from Dask to Ray for distributed text processing
  • Improved model-based classifier throughput: Better overlapping of compute between tokenization and inference through length-based sequence sorting for optimal GPU memory utilization
  • Task-centric architecture: New Task-based processing model for finer-grained control
  • Pipeline redesign: Updated ProcessingStage and Pipeline architecture with resource specification

Image

  • Pipeline-based architecture: Transitioned from legacy ImageTextPairDataset to modern stage-based processing with ImageReaderStage, ImageEmbeddingStage, and filter stages
  • DALI-based image loading: New ImageReaderStage uses NVIDIA DALI for high-performance WebDataset tar shard processing with GPU/CPU fallback
  • Modular processing stages: Separate stages for embedding generation, aesthetic filtering, and NSFW filtering
  • Task-based data flow: Images processed as ImageBatch tasks containing ImageObject instances with metadata, embeddings, and classification scores

Learn more about image curation.

Deduplication Improvements

Enhanced deduplication capabilities across all modalities with improved performance and flexibility:

  • Exact and Fuzzy deduplication: Updated rapidsmpf-based shuffle backend for more efficient GPU-to-GPU data transfer and better spilling capabilities
  • Semantic deduplication: Support for deduplicating text, image, and video datasets using unified embedding-based workflows
  • New ranking strategies: Added RankingStrategy which allows you to rank elements within cluster centers to decide which point to prioritize during duplicate removal, supporting metadata-based ranking to prioritize specific datasets or inputs

Core Refactors

The architecture refactor introduces a layered system with unified interfaces and multiple execution backends:

User Layer: Pipeline → ProcessingStage X→Y → ProcessingStage Y→Z → ProcessingStage Z→W
           ↓
Orchestration Layer: BaseExecutor Interface
           ↓
Backend Layer: XennaExecutor (Production Ready) | RayActorPoolExecutor (Experimental) | RayDataExecutor (Experimental)
           ↓
Adaptation Layer: Xenna Adapter | Ray Actor Adapter | Ray Data Adapter
           ↓
Execution Layer: Cosmos-Xenna (Streaming/Batch) | Ray Actor Pool (Load Balancing) | Ray Data API (Dataset Processing)

Pipelines

  • New Pipeline API: Ray-based pipeline execution with BaseExecutor interface
  • Multiple backends: Support for Xenna, Ray Actor Pool, and Ray Data execution backends
  • Resource specification: Configurable CPU and GPU memory requirements per stage
  • Stage composition: Improved stage validation and execution orchestration

Stages

  • ProcessingStage redesign: Generic ProcessingStage[X, Y] base class with type safety
  • Resource requirements: Built-in resource specification for CPU and GPU memory
  • Backend adapters: Stage adaptation layer for different Ray orchestration systems
  • Input/output validation: Enhanced type checking and data validation

Tutorials

For all tutorial content, refer to the tutorials directory in the NeMo Curator GitHub repository.

Known Limitations

(Pending Refactor in Future Release)

Generation

  • Synthetic data generation: Synthetic text generation features are being refactored for Ray compatibility
  • Hard negative mining: Retrieval-based data generation workflows under development

PII

  • PII processing: Personal Identifiable Information removal tools are being updated for Ray backend
  • Privacy workflows: Enhanced privacy-preserving data curation capabilities in development

Blending & Shuffling

  • Data blending: Multi-source dataset blending functionality being refactored
  • Dataset shuffling: Large-scale data shuffling operations under development

Docs Refactor

  • Local preview capability: Improved documentation build system with local preview support
  • Modality-specific guides: Comprehensive documentation for each supported modality (text, image, audio, video)
  • API reference: Complete API documentation with type annotations and examples

What's Next

The next release will focus on completing the refactor of Generation, PII, and Blending & Shuffling features, along with additional performance optimizations and new modality support.

NVIDIA NeMo Curator 0.9.0

Major Features and Enhancements

  • New How-to Data Recipes (Tutorials)
    • Multimodal DAPT Curation w/ PDF Extraction
    • Llama Nemotron Data Curation
    • LLM NIM - PII Redaction
  • Performance and Code Optimizations
    • Simplified Clustering Logic: Significantly improved semantic deduplication clustering performance
    • Removed convoluted backend switching logic that caused performance issues
    • Eliminated expensive length assertions that could cause timeouts on large datasets
    • Improved GPU utilization during KMeans clustering operations
    • Tested on 37M embedding dataset (80GB) across 7 GPUs with substantial performance gains

Bug Fixes

  • FastText Download URL Fix
    • Corrected the fasttext model download URL in nemotron-cc tutorial
    • Changed from dl.fbaipublicfiles.com/fastText/ to dl.fbaipublicfiles.com/fasttext/
    • Ensures reliable model downloads for language identification
  • NeMo Retriever Tutorial Bug Fix
    • Fixed lambda function bug in RetrieverEvalSetGenerator
    • Corrected score assignment from df["question"].apply(lambda: 1) to df["score"] = 1
  • API Usage Updates
    • Updated examples and tutorials to use correct DocumentDataset API
    • Replaced deprecated write_to_disk(result, output_dir, output_type="parquet") with result.to_parquet(output_dir)
    • Updated exact deduplication workflows: deduplicator.remove() now returns DocumentDataset directly

NVIDIA NeMo Curator 0.8.0

  • Llama Based PII Redaction
  • Trafilatura Text Extractor
  • Chinese & Japanese Stopwords for Text Extractors
  • Writing gzip compressed jsonl datasets
  • Training dataset curation for retriever customization using hard-negative mining
  • Implemented a memory efficient pairwise similarity in Semantic Deduplication

NVIDIA NeMo Curator 0.7.1

  • Fix Transformers + Cuda Context bug
  • Fix rate limit in SDG Retriever Eval Tutorial

NVIDIA NeMo Curator 0.7.0

  • Python 3.12 Support
  • Curator on Blackwell
  • Nemotron-CC Dataset Recipe
  • Performant S3 for Fuzzy Deduplication

NVIDIA NeMo Curator 0.6.0

  • Synthetic Data Generation for Text Retrieval
    • LLM-based Filters
      • Easiness
      • Answerability
    • Q&A Retrieval Generation Pipeline

NVIDIA NeMo Curator 0.5.0

Highlights

  • Image Curation
    • Image Embedding Creation
    • Aesthetic Classifier
    • NSFW Classifier
    • Semantic Deduplication
  • Text Curation
    • Quality Classifier
    • Aegis Classifier
    • FineWeb-Edu Classifier

Full Changelog: https://github.com/NVIDIA/NeMo-Curator/commits/v0.5.0

NVIDIA NeMo Curator 0.4.1

What's Changed

Full Changelog: https://github.com/NVIDIA/NeMo-Curator/compare/v0.4.0...v0.4.1

NVIDIA NeMo Curator 0.4.0

Highlights

  • Semantic Deduplication
  • Resiliparse for Text Extraction
  • Improve Distributed Data Classification - Domain classifier is 1.55x faster through intelligent batching
  • Synthetic data generation for fine-tuning

What's Changed

New Contributors

Full Changelog: https://github.com/NVIDIA/NeMo-Curator/commits/v0.4.0s

NeMo Curator 0.3.0

What's Changed

New Contributors

Full Changelog: https://github.com/NVIDIA/NeMo-Curator/commits/v0.3.0

PyPi

https://pypi.org/project/nemo-curator/0.3.0/