Merge branch 'main' of github.com:SeregaCodit/DataForge into main

Author: Serhii Naumenko

.gitignore

@@ -289,3 +289,4 @@ pyrightconfig.json
 /cache/
 /tst_commands.py
 /.idea/DataForge.iml
+/reports/

README.MD

@@ -1,32 +1,49 @@
 # DataForge 
-**DataForge** is a high-performance CLI tool designed to automate the preparation and management of machine learning datasets. It helps you transform raw and dirty data (like videos and unsorted images) into clean, balanced, and ready-to-train datasets with minimal effort.
 
-`📖 [Read the full documentation here](https://seregacodit.github.io/DataForge/)`
+**DataForge** is a high-performance CLI tool designed to automate the preparation and management of machine learning datasets. It helps you transform raw data (like videos and unsorted images) into clean, balanced, and ready-to-train datasets with minimal effort.
+
+[Read the full documentation here](https://seregacodit.github.io/DataForge)
+
 ### Key Features
-* **Parallel Processing:** uses multiprocessing to handle thousands of files quickly.
-* **Vectorized Calculations:** employs NumPy for ultra-fast image comparison.
-* **Smart Caching:** incremental caching (MD5-based) allows working with large datasets on NAS without re-calculating everything.
-* **Config:** Built with Pydantic v2 for safe and flexible settings via JSON or CLI.
+* **Parallel Processing:** Uses multiprocessing to handle thousands of files quickly.
+* **Vectorized Calculations:** Employs NumPy for ultra-fast image comparison and hashing.
+* **Smart Caching:** Incremental caching (MD5-based) allows working with large datasets on NAS or local storage without re-calculating existing data.
+* **Flexible Configuration:** Built with Pydantic v2 for safe settings via `config.json` or CLI arguments.
 
 ---
 
 ### Available Commands
 
-* **`move`** — move files from source to target directory based on patterns.
-* **`slice`** — convert video files into sequences of images. Use `--remove` to delete the source video after a successful slice.
-* **`delete`** — safely remove files matching specific patterns.
-* **`dedup`** — find and remove visual duplicates using **dHash**.
-    * *Threshold:* information similarity limit (0-100%).
-    * *Core Size:* higher values (e.g., 32, 64) detect small changes (like a moving car), lower values (e.g., 8) ignore noise.
-* **`clean-annotations`** — automatically find and delete "orphan" annotation files (XML/TXT) that no longer have a corresponding image.
-* **`convert-annotations`** — convert dataset labels between formats (e.g., **Pascal VOC** to **YOLO**).
+* **`move`** — Move files from source to target directory based on specific patterns.
+* **`slice`** — Convert video files into sequences of images. Use `--remove` to delete the source video after a successful slice.
+* **`delete`** — Safely remove files matching specific patterns.
+* **`dedup`** — Find and remove visual duplicates using **dHash**.
+    * *Threshold:* Similarity limit (0-100%).
+    * *Core Size:* Higher values (e.g., 32) detect small changes; lower values (e.g., 8) ignore noise.
+* **`clean-annotations`** — Automatically find and delete "orphan" annotation files (XML/TXT) that do not have a corresponding image.
+* **`convert-annotations`** — Convert dataset labels between formats (e.g., **Pascal VOC** to **YOLO**).
+* **`stats` — Advanced Dataset Analytics & Health Check**
+This command performs a deep-dive into your dataset to identify biases and feature correlations before you start training.
+    * **Analytics Highlights:**
+        * **Class Distribution:** Visualizes object counts to detect imbalances.
+        * **Spatial Density Heatmaps:** Identifies "positional bias" for each class using 3x3 grids.
+        * **Correlation Analysis:** Global and per-class matrices showing relationships between features.
+        * **Dataset Manifold (UMAP):** 2D projection to identify "representation gaps" and object clusters.
+        * **Quality Metrics:** Analysis of object areas, aspect ratios, brightness, contrast, and blur.
+        * **Outlier Detection:** Automatically marks extreme data points using the IQR method.
+    * **Outputs:** Technical console summary, high-resolution PNG plots, and unified PDF reports.
+
+**Usage Example:**
+```bash
+python data_forge.py stats --src ./data/train --target_format yolo --report_path ./reports/v1
+```
 
 ---
 
 ### Automation & Intervals
-By default, commands run once. If you want to monitor a folder and process files as they appear, use the repeat flag:
-* Use **`-r`** to run the command in a cycle.
-* Set the delay between cycles with **`-s`** (seconds).
+By default, commands run once. To monitor a folder and process files as they appear, use these flags:
+* **`-r`**: Run the command in a continuous cycle.
+* **`-s`**: Set the delay (in seconds) between cycles.
 
 ---
 
@@ -47,14 +64,14 @@ pip install -r requirements.txt
 
 3. **Check usage:**
 ```bash
-python data_forge.py --help             # See all commands
+python data_forge.py --help             # See all available commands
 python data_forge.py {command} --help   # See arguments for a specific command
 ```
 
 ---
 
-### Workflow
-For multiple tasks, you can modify `start_all_tasks.sh` and run them all in the background:
+### Workflow Optimization
+For multiple tasks, you can modify `start_all_tasks.sh` and run them in the background:
 ```bash
 bash start_all_tasks.sh
 ```
@@ -63,6 +80,6 @@ To stop all running DataForge processes:
 pkill -f data_forge.py
 ```
 
-### Configuration
-You can manage all default settings in `config.json`. DataForge follows this priority:
+### Configuration Priority
+You can manage default settings in `config.json`. DataForge follows this priority:
 **CLI Arguments > config.json > Internal Defaults.**

config.json

@@ -7,14 +7,67 @@
   "step_sec": 5,
   "log_path": "./log",
   "log_level": "INFO",
-  "filetype": "image",
+  "report_path": "./reports",
+  "datatype": "image",
   "method": "dhash",
   "hash_threshold": 10,
-  "confirm_choice": ["delete", "вудуеу", "yes", "y", "true", "t", "1"],
+  "confirm_choice": [
+    "yes"
+  ],
   "core_size": 16,
-  "n_jobs":  20,
+  "n_jobs": 20,
   "cache_file_path": "./cache",
   "cache_name": null,
-  "a_suffix": [".xml"],
-  "a_source": null
+  "a_suffix": [
+    ".xml"
+  ],
+  "a_source": null,
+  "img_dataset_report_schema": [
+    {
+      "title": "GEOMETRY",
+      "type": "numeric",
+      "columns": [
+        "object_area",
+        "object_relative_area",
+        "object_width",
+        "object_height",
+        "object_aspect_ratio"
+      ]
+    },
+    {
+      "title": "IMAGE QUALITY",
+      "type": "numeric",
+      "columns": [
+        "im_brightness",
+        "im_contrast",
+        "im_blur_score"
+      ]
+    },
+    {
+      "title": "SPATIAL BIAS",
+      "type": "binary",
+      "columns": [
+        "object_in_center",
+        "object_in_top_side",
+        "object_in_bottom_side",
+        "object_in_left_side",
+        "object_in_right_side",
+        "object_in_left_top",
+        "object_in_right_top",
+        "object_in_left_bottom",
+        "object_in_right_bottom"
+      ]
+    },
+    {
+      "title": "TRUNCATION",
+      "type": "binary",
+      "columns": [
+        "truncated_top",
+        "truncated_bottom",
+        "truncated_left",
+        "truncated_right"
+      ]
+    }
+
+  ]
 }

const_utils/arguments.py

@@ -19,7 +19,7 @@ class Arguments:
     rm: str = "-rm"
     log_level: str = "--log_level"
     log_path: str = "--log_path"
-    filetype: str = "--filetype"
+    datatype: str = "--datatype"
     method: str = "--method"
     m: str = "-m"
     action: str = "--action"
@@ -33,3 +33,5 @@ class Arguments:
     destination_type: str = "--destination-type"
     img_path: str = "--img_path"
     extensions: str = "--ext"
+    margin: str = "--margin"
+    report_path: str = "--report_path"

const_utils/commands.py

@@ -8,4 +8,5 @@ class Commands:
     delete: str = "delete"
     dedup: str = "dedup"
     clean_annotations: str = "clean-annotations"
-    convert_annotations: str = "convert-annotations"
+    convert_annotations: str = "convert-annotations"
+    stats: str = "stats"

const_utils/default_values.py

@@ -1,13 +1,14 @@
 import json
 import multiprocessing
 
-from typing import Union, Tuple, Optional, List
+from typing import Union, Tuple, Optional, List, Dict, Any
 
 from pydantic import Field, field_validator
 from pydantic_settings import BaseSettings, SettingsConfigDict
 from pathlib import Path
 
 from const_utils.copmarer import Constants
+from const_utils.stats_constansts import ImageStatsKeys
 from logger.log_level_mapping import LevelMapping
 
 
@@ -30,7 +31,7 @@ class AppSettings(BaseSettings):
         step_sec (float): Time interval in seconds for video slicing.
         log_path (Path): Directory where log files are stored.
         log_level (str): Verbosity level of the logger (e.g., INFO, DEBUG).
-        filetype (str): The category of files being processed (e.g., image).
+        datatype (str): The category of files being processed (e.g., image).
         method (str): The algorithm name for hashing or comparison.
         hash_threshold (int): Distance threshold for identifying duplicates (0-100).
         confirm_choice (tuple): Keywords used to confirm interactive deletion.
@@ -40,7 +41,7 @@ class AppSettings(BaseSettings):
         cache_name (Optional[Path]): Custom name for the cache file.
         a_suffix (Tuple[str, ...]): File patterns specific to annotations.
         a_source (Optional[Path]): Directory where annotation files are located.
-        destination_type (Optional[str]): Target format for annotation conversion.
+        destination_type (Optional[str]): Target format for annotations.
         extensions (Tuple[str, ...]): Supported image file extensions.
     """
     max_percentage: int = 100
@@ -58,10 +59,10 @@ class AppSettings(BaseSettings):
     step_sec: float = Field(default=1.0, ge=0.1)
     log_path: Path = Field(default=Path("./log"))
     log_level: str = Field(default=LevelMapping.info)
-    filetype: str = Field(default=Constants.image)
+    datatype: str = Field(default=Constants.image)
     method: str = Field(default=Constants.dhash)
     hash_threshold: int = Field(default=10, ge=0, le=100)
-    confirm_choice: tuple = Field(default=("delete",))
+    confirm_choice: tuple = Field(default=("yes",))
     core_size: int = Field(default=8, ge=8)
     n_jobs: int = Field(default=2, ge=1, le=multiprocessing.cpu_count())
     cache_file_path: Path = Field(default=Path("./cache"))
@@ -70,6 +71,55 @@ class AppSettings(BaseSettings):
     a_source: Optional[Path] = Field(default=None)
     destination_type: Optional[str] = Field(default=None)
     extensions: Tuple[str, ...] = Field(default=(".jpg", ".jpeg,", ".png"))
+    margin_threshold: int = Field(default=5, ge=0, le=100)
+    report_path: Path = Field(default=Path("./reports"))
+    img_dataset_report_schema: List[Dict[str, Any]] = Field(default=[
+        {
+            "title": "GEOMETRY",
+            "type": "numeric",
+            "columns": [
+                ImageStatsKeys.object_area,
+                ImageStatsKeys.object_relative_area,
+                ImageStatsKeys.object_width,
+                ImageStatsKeys.object_height,
+                ImageStatsKeys.object_aspect_ratio
+            ]
+        },
+        {
+            "title": "SPATIAL BIAS",
+            "type": "binary",
+            "columns": [
+                ImageStatsKeys.object_in_center,
+                ImageStatsKeys.object_in_top_side,
+                ImageStatsKeys.object_in_bottom_side,
+                ImageStatsKeys.object_in_left_side,
+                ImageStatsKeys.object_in_right_side,
+                ImageStatsKeys.object_in_left_top,
+                ImageStatsKeys.object_in_right_top,
+                ImageStatsKeys.object_in_left_bottom,
+                ImageStatsKeys.object_in_right_bottom
+            ]
+        },
+        {
+            "title": "TRUNCATION",
+            "type": "binary",
+            "columns": [
+                ImageStatsKeys.truncated_top,
+                ImageStatsKeys.truncated_bottom,
+                ImageStatsKeys.truncated_left,
+                ImageStatsKeys.truncated_right
+            ]
+        },
+        {
+            "title": "IMAGE QUALITY",
+            "type": "numeric",
+            "columns": [
+                ImageStatsKeys.im_brightness,
+                ImageStatsKeys.im_contrast,
+                ImageStatsKeys.im_blur_score
+            ]
+        }
+    ])
 
 
     @field_validator('core_size')
@@ -92,7 +142,7 @@ def check_power_of_two(cls, value: int) -> int:
         return value
 
 
-    @field_validator("log_path", "cache_file_path", "a_source", mode='before')
+    @field_validator("report_path", "log_path", "cache_file_path", "a_source", mode='before')
     @classmethod
     def ensure_path(cls, value: Union[str, Path]) -> Path:
         """

const_utils/parser_help.py

@@ -18,7 +18,7 @@ class HelpStrings:
     remove: str = "remove files after processing"
     log_path: str = "path to log directory"
     log_level: str = f"A level of logging matches mapping: {str(LevelMapping.mapping())}"
-    filetype: str = "Type of file. Currently this parameter only supports 'image'"
+    datatype: str = "Type of data. Currently this parameter only supports 'image'"
     method: str = "Default: dhash. A method of comparing images. It's can be ['phash, dhash, ahash, cnn]"
     threshold: str = ("A minimal difference between files that means the files"
                       f" have a different information. Using Hemming distance for *hash methods")
@@ -36,4 +36,7 @@ class HelpStrings:
     destination_type: str = "A type of destination annotation format"
     img_path: str = "Path to dataset images directory"
     extensions: str = ("A tuple of file extensions that will be used as pattern for building file whitelists for "
-                       "converting from yolo to other formats")
+                       "converting from yolo to other formats")
+    margin: str = ("A threshold value of margin from any image border. If any side of object bbox cloaser that this"
+                   "value to image boarder - object will be defined as truncated")
+    report_path: str = "A path to directory where reports will be stored"

const_utils/stats_constansts.py

@@ -0,0 +1,38 @@
+from dataclasses import dataclass
+
+
+@dataclass
+class ImageStatsKeys:
+    """Constants for stats dictionary keys and default values."""
+    path: str = "path"
+    im_path: str = "im_path"
+    mtime: str = "mtime"
+    class_name: str = "class_name"
+    objects_count: str = "objects_count"
+    im_width: str = "im_width"
+    im_height: str = "im_height"
+    im_depth: str = "im_depth"
+    im_brightness: str = "im_brightness"
+    im_contrast: str = "im_contrast"
+    im_blur_score: str = "im_blur_score"
+    has_neighbors: str = "has_neighbors"
+    object_width: str = "object_width"
+    object_height: str = "object_height"
+    object_aspect_ratio: str = "object_aspect_ratio"
+    object_area: str = "object_area"
+    object_relative_area: str = "object_relative_area"
+    object_in_center: str = "object_in_center"
+    object_in_right_side: str = "object_in_right_side"
+    object_in_left_side: str = "object_in_left_side"
+    object_in_top_side: str = "object_in_top_side"
+    object_in_bottom_side: str = "object_in_bottom_side"
+    object_in_left_top: str = "object_in_left_top"
+    object_in_right_top: str = "object_in_right_top"
+    object_in_left_bottom: str = "object_in_left_bottom"
+    object_in_right_bottom: str = "object_in_right_bottom"
+    full_size: str = "full_size"
+    truncated_left: str = "truncated_left"
+    truncated_right: str = "truncated_right"
+    truncated_top: str = "truncated_top"
+    truncated_bottom: str = "truncated_bottom"
+    outlier_any: str = "outlier_any"

const_utils/xml_names.py

@@ -0,0 +1,26 @@
+from dataclasses import dataclass
+
+@dataclass
+class XMLNames:
+    """Constants for XML tag and attribute names."""
+    annotation: str = "annotation"
+    folder: str = "folder"
+    filename: str = "filename"
+    path: str = "path"
+    source: str = "source"
+    database: str = "database"
+    size: str = "size"
+    width: str = "width"
+    height: str = "height"
+    depth: str = "depth"
+    segmented: str = "segmented"
+    object: str = "object"
+    name: str = "name"
+    pose: str = "pose"
+    truncated: str = "truncated"
+    difficult: str = "difficult"
+    bndbox: str = "bndbox"
+    xmin: str = "xmin"
+    ymin: str = "ymin"
+    xmax: str = "xmax"
+    ymax: str = "ymax"