Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

Refine high-quality datasets and visual AI models

A Doctor for your data

fastdup is a powerful, free tool designed to rapidly generate valuable insights from image and video datasets. It helps enhance the quality of both images and labels, while significantly reducing data operation costs, all with unmatched scalability.

Scalable data pre processing and curation toolkit for LLMs

Interactively explore unstructured datasets from your dataframe.

A curated, but incomplete, list of data-centric AI resources.

一个管理电脑内资源，并提供游戏化的综合管理器

Curated list of open source tooling for data-centric AI on unstructured data.

🏭 Mega Scale Multimodal DataPipeline for SOTA Foundation Models

Streamline on-policy/off-policy distillation workflows in a few lines of code

A simple yet versatile context engineered for scalable online data collection

[ICLR 2025] Official implementation of paper "Improving Data Efficiency via Curating LLM-Driven Rating Systems"

Metamapper is a data discovery and documentation platform for improving how teams understand and interact with their data.

PyTorch dataset debugger for computer vision — pause training, mine live loss signals to surface mislabels, class imbalance & outliers, then curate your image, video & LiDAR data without restarting

A library for detecting problematic data segments in structured and unstructured data with few lines of code.

Learn2Clean: Optimizing the Sequence of Tasks for Data Preparation and Cleaning

Lesson guide and textbook for "Data as a Science" course.

A tool for downloading from public image boards (which allow scraping) / preview your images & tags / edit your images & tags. Additional tabs for downloading other desired code repositories as well as S.O.T.A. diffusion and auto-tag/caption models for your purposes. Custom datasets can be added!

[NeurIPS 2024] 🧼🔎 A holistic self-supervised data cleaning strategy to detect irrelevant samples, near duplicates and label errors.