feat(core): Add class group-aware metric analysis with multi-config support

CHANGELOG.md

@@ -14,6 +14,25 @@ No changes to highlight.
 
 ## Other Changes:
 
+No changes to highlight.
+
+# v1.4.1
+
+## New Features:
+
+- Add yolov9 pretrained weights by `@illian01` in [PR 631](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/631)
+- Add EXIR exporting feature by `@illian01` in [PR 632](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/632)
+
+## Bug Fixes:
+
+No changes to highlight.
+
+## Breaking Changes:
+
+No changes to highlight.
+
+## Other Changes:
+
 Fix/add data params mlflow by `@hglee98` in [PR 629](https://github.com/Nota-NetsPresso/netspresso-trainer/pull/629)
 
 # 1.4.0

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.8.16
+FROM python:3.10
 
 ARG TORCH_VERSION="2.0.1"
 ARG TORCHVISION_VERSION="0.15.2"

README.md

@@ -33,8 +33,8 @@ _____
 
 ### Prerequisites
 
-- Python `3.8` | `3.9` | `3.10`
-- PyTorch `2.0.1` (recommended) (compatible with: `1.11.x` - `2.0.1`)
+- Python `>=3.10`
+- PyTorch `>=2.0.1`
 
 ### Install with pypi
 

config/benchmark_examples/detection-coco2017-yolov9_tiny/augmentation.yaml

@@ -0,0 +1,56 @@
+augmentation:
+  train:
+    - 
+      name: mosaicdetection
+      size: [640, 640]
+      mosaic_prob: 1.0
+      affine_scale: [0.1, 1.9]
+      degrees: 0.0
+      translate: 0.1
+      shear: 0.0
+      enable_mixup: True
+      mixup_prob: 0.15
+      mixup_scale: [0.1, 2.0]
+      fill: 0
+      mosaic_off_duration: 15
+    -
+      name: hsvjitter
+      h_mag: 5
+      s_mag: 30
+      v_mag: 30
+    -
+      name: randomhorizontalflip
+      p: 0.5
+    -
+      name: resize
+      size: 640
+      interpolation: bilinear
+      max_size: ~
+      resize_criteria: long
+    -
+      name: pad
+      size: 640
+      fill: 0
+    -
+      name: randomresize
+      base_size: [640, 640]
+      stride: 32
+      random_range: 5
+      interpolation: bilinear
+    -
+      name: totensor
+      pixel_range: 1.0
+  inference:
+    - 
+      name: resize
+      size: 640
+      interpolation: bilinear
+      max_size: ~
+      resize_criteria: long
+    - 
+      name: pad
+      size: 640
+      fill: 0
+    -
+      name: totensor
+      pixel_range: 1.0

config/benchmark_examples/detection-coco2017-yolov9_tiny/data.yaml

@@ -0,0 +1,21 @@
+data:
+  name: coco2017
+  task: detection
+  format: local # local, huggingface
+  path:
+    root: ./data/coco2017 # dataset root
+    train:
+      image: images/train # directory for training images
+      label: labels/train # directory for training labels
+    valid:
+      image: images/valid # directory for valid images
+      label: labels/valid # directory for valid labels
+    test:
+      image: ~
+      label: ~
+    pattern:
+      image: ~
+      label: ~
+  id_mapping: id_mapping.json
+  # id_mapping: ['person', 'bicycle', 'car', 'motorcycle', 'airplane', 'bus', 'train', 'truck', 'boat', 'traffic light', 'fire hydrant', 'stop sign', 'parking meter', 'bench', 'bird', 'cat', 'dog', 'horse', 'sheep', 'cow', 'elephant', 'bear', 'zebra', 'giraffe', 'backpack', 'umbrella', 'handbag', 'tie', 'suitcase', 'frisbee', 'skis', 'snowboard', 'sports ball', 'kite', 'baseball bat', 'baseball glove', 'skateboard', 'surfboard', 'tennis racket', 'bottle', 'wine glass', 'cup', 'fork', 'knife', 'spoon', 'bowl', 'banana', 'apple', 'sandwich', 'orange', 'broccoli', 'carrot', 'hot dog', 'pizza', 'donut', 'cake', 'chair', 'couch', 'potted plant', 'bed', 'dining table', 'toilet', 'tv', 'laptop', 'mouse', 'remote', 'keyboard', 'cell phone', 'microwave', 'oven', 'toaster', 'sink', 'refrigerator', 'book', 'clock', 'vase', 'scissors', 'teddy bear', 'hair drier', 'toothbrush']
+  pallete: ~

config/benchmark_examples/detection-coco2017-yolov9_tiny/environment.yaml

@@ -0,0 +1,6 @@
+environment: 
+  seed: 1
+  num_workers: 4
+  gpus: 0, 1, 2, 3, 4, 5, 6, 7
+  batch_size: 16 # Batch size per gpu
+  cache_data: False

config/benchmark_examples/detection-coco2017-yolov9_tiny/logging.yaml

@@ -0,0 +1,18 @@
+logging:
+  project_id: ~
+  output_dir: ./outputs
+  tensorboard: true
+  mlflow: false
+  stdout: true
+  num_save_samples: 16 # num_save_samples should be >= 0 or None
+  model_save_options:
+    save_optimizer_state: true
+    save_best_only: false
+    best_model_criterion: loss # metric
+    sample_input_size: [640, 640] # Used for flops and onnx export
+    onnx_export_opset: 13 # Recommend in range [13, 17]
+    validation_epoch: &validation_epoch 10
+    save_checkpoint_epoch: *validation_epoch  # Multiplier of `validation_epoch`.
+  metrics:
+    classwise_analysis: False
+    metric_names: ~ # None for default settings

config/benchmark_examples/detection-coco2017-yolov9_tiny/model.yaml

@@ -0,0 +1,72 @@
+model:
+  task: detection
+  name: yolov9_tiny
+  checkpoint:
+    use_pretrained: false
+    load_head: false
+    path: ~
+    optimizer_path: ~
+  freeze_backbone: false
+  architecture:
+    full: ~ # auto
+    backbone:
+      name: gelan
+      params:
+        stem_out_channels: 16
+        stem_kernel_size: 3
+        stem_stride: 2
+        return_stage_idx: [1, 2, 3]
+        act_type: &act_type silu
+      stage_params:
+        # Conv2D: ['conv', out_channels, kernel_size, stride]
+        # ELAN: ['elan', out_channels, part_channels, use_identity]
+        # RepNCSPELAN: ['repncspelan', out_channels, part_channels, use_identity, depth]
+        # AConv: ['aconv', out_channels]
+        # ADown: ['adown', out_channels]
+        -
+          - ['conv', 32, 3, 2]
+          - ['elan', 32, 32, false]
+        -
+          - ['aconv', 64]
+          - ['repncspelan', 64, 64, false, 3]
+        -
+          - ['aconv', 96]
+          - ['repncspelan', 96, 96, false, 3]
+        - 
+          - ['aconv', 128]
+          - ['repncspelan', 128, 128, false, 3]
+    neck:
+      name: yolov9fpn
+      params:
+        repeat_num: 3
+        act_type: *act_type
+        use_aux_loss: &use_aux_loss false
+        bu_type: aconv
+        spp_channels: 128
+        n4_channels: 96
+        p3_channels: 64
+        p3_to_p4_channels: 48
+        p4_channels: 96
+        p4_to_p5_channels: 64
+        p5_channels: 128
+    head:
+      name: yolo_detection_head
+      params:
+        version: v9
+        num_anchors: ~
+        use_group: true
+        reg_max: &reg_max 16
+        act_type: *act_type
+        use_aux_loss: *use_aux_loss
+  postprocessor: 
+    params: 
+      # postprocessor - decode
+      reg_max: *reg_max
+      score_thresh: 0.01
+      # postprocessor - nms
+      nms_thresh: 0.65
+      class_agnostic: false
+  losses:
+    - criterion: yolov9_loss
+      weight: ~
+      l1_activate_epoch: ~

config/benchmark_examples/detection-coco2017-yolov9_tiny/training.yaml

@@ -0,0 +1,23 @@
+training:
+  epochs: 500
+  mixed_precision: True
+  max_norm: ~
+  ema:
+    name: exp_decay
+    decay: 0.9999
+    beta: 2000
+  optimizer:
+    name: sgd
+    lr: 0.01
+    momentum: 0.937
+    weight_decay: 0.0005 # No bias and norm decay
+    nesterov: True
+    no_bias_decay: True
+    no_norm_weight_decay: True
+    overwrite: ~
+  scheduler:
+    name: cosine_no_sgdr
+    warmup_epochs: 3
+    warmup_bias_lr: 0.001
+    min_lr: 0.0001
+    end_epoch: 485

config/logging.yaml

@@ -15,4 +15,5 @@ logging:
     save_checkpoint_epoch: *validation_epoch  # Multiplier of `validation_epoch`.
   metrics:
     classwise_analysis: False
-    metric_names: ~ # None for default settings
+    metric_names: ~ # None for default settings
+    class_groups: ~ # Example: {vehicle: [car, truck, bus], person: [person, rider]}

config/model/mobilenetv4/mobilenetv4-conv-medium-classification.yaml

@@ -21,7 +21,7 @@ model:
         norm_type: batch_norm
         act_type: relu
         return_stage_idx: ~
-        layer_scale: 0.1
+        layer_scale: ~
       stage_params:
         # Conv2D: ['conv', out_channels, kernel_size, stride]
         # FusedIB: ['fi', out_channels, hidden_channels, kernel_size, stride]

config/model/mobilenetv4/mobilenetv4-conv-small-classification.yaml

@@ -21,7 +21,7 @@ model:
         norm_type: batch_norm
         act_type: relu
         return_stage_idx: ~
-        layer_scale: 0.1
+        layer_scale: ~
       stage_params:
         # Conv2D: ['conv', out_channels, kernel_size, stride]
         # FusedIB: ['fi', out_channels, hidden_channels, kernel_size, stride]

config/model/mobilenetv4/mobilenetv4-hybrid-large-classification.yaml

@@ -21,7 +21,7 @@ model:
         norm_type: batch_norm
         act_type: gelu
         return_stage_idx: ~
-        layer_scale: ~
+        layer_scale: 0.1
       stage_params:
         # Conv2D: ['conv', out_channels, kernel_size, stride]
         # FusedIB: ['fi', out_channels, hidden_channels, kernel_size, stride]

config/model/yolo/yolov9_c-detection.yaml → config/model/yolov9/yolov9_c-detection.yaml

@@ -2,7 +2,7 @@ model:
   task: detection
   name: yolov9_c
   checkpoint:
-    use_pretrained: false
+    use_pretrained: true
     load_head: false
     path: ~
     optimizer_path: ~

config/model/yolo/yolov9_m-detection.yaml → config/model/yolov9/yolov9_m-detection.yaml

@@ -2,7 +2,7 @@ model:
   task: detection
   name: yolov9_m
   checkpoint:
-    use_pretrained: false
+    use_pretrained: true
     load_head: false
     path: ~
     optimizer_path: ~

config/model/yolo/yolov9_s-detection.yaml → config/model/yolov9/yolov9_s-detection.yaml

@@ -2,7 +2,7 @@ model:
   task: detection
   name: yolov9_s
   checkpoint:
-    use_pretrained: false
+    use_pretrained: true
     load_head: false
     path: ~
     optimizer_path: ~

config/model/yolov9/yolov9_tiny-detection.yaml

@@ -0,0 +1,72 @@
+model:
+  task: detection
+  name: yolov9_tiny
+  checkpoint:
+    use_pretrained: true
+    load_head: false
+    path: ~
+    optimizer_path: ~
+  freeze_backbone: false
+  architecture:
+    full: ~ # auto
+    backbone:
+      name: gelan
+      params:
+        stem_out_channels: 16
+        stem_kernel_size: 3
+        stem_stride: 2
+        return_stage_idx: [1, 2, 3]
+        act_type: &act_type silu
+      stage_params:
+        # Conv2D: ['conv', out_channels, kernel_size, stride]
+        # ELAN: ['elan', out_channels, part_channels, use_identity]
+        # RepNCSPELAN: ['repncspelan', out_channels, part_channels, use_identity, depth]
+        # AConv: ['aconv', out_channels]
+        # ADown: ['adown', out_channels]
+        -
+          - ['conv', 32, 3, 2]
+          - ['elan', 32, 32, false]
+        -
+          - ['aconv', 64]
+          - ['repncspelan', 64, 64, false, 3]
+        -
+          - ['aconv', 96]
+          - ['repncspelan', 96, 96, false, 3]
+        - 
+          - ['aconv', 128]
+          - ['repncspelan', 128, 128, false, 3]
+    neck:
+      name: yolov9fpn
+      params:
+        repeat_num: 3
+        act_type: *act_type
+        use_aux_loss: &use_aux_loss false
+        bu_type: aconv
+        spp_channels: 128
+        n4_channels: 96
+        p3_channels: 64
+        p3_to_p4_channels: 48
+        p4_channels: 96
+        p4_to_p5_channels: 64
+        p5_channels: 128
+    head:
+      name: yolo_detection_head
+      params:
+        version: v9
+        num_anchors: ~
+        use_group: true
+        reg_max: &reg_max 16
+        act_type: *act_type
+        use_aux_loss: *use_aux_loss
+  postprocessor: 
+    params: 
+      # postprocessor - decode
+      reg_max: *reg_max
+      score_thresh: 0.01
+      # postprocessor - nms
+      nms_thresh: 0.65
+      class_agnostic: false
+  losses:
+    - criterion: yolov9_loss
+      weight: ~
+      l1_activate_epoch: ~

docs/benchmarks/benchmarks.md

@@ -40,6 +40,10 @@ If you have a better recipe, please share with us anytime. We appreciate all eff
 |---|---|---|---|---|---|---|---|---|---|---|
 | COCO-val | [RT-DETR_res18*](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/rtdetr/rtdetr-res18-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/rtdetr/rtdetr_res18_coco.safetensors?versionId=uu9v49NI6rQx8wOY6bJbEXUFOG_R9xqH) | (640, 640) | 65.77 | 52.75 | 48.49 | 20.18M | 40.36G | Supported | No input z-norm, [lyuwenyu/RT-DETR](https://github.com/lyuwenyu/RT-DETR/tree/main/rtdetr_pytorch) |
 | COCO-val | [RT-DETR_res50*](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/rtdetr/rtdetr-res50-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/rtdetr/rtdetr_res50_coco.safetensors?versionId=JHmnjY13BEflpnDCYPFJ1c17UwpqDrLQ) | (640, 640) | 72.64 | 59.50 | 54.73 | 42.94M | 138.36G | Supported | No input z-norm, [lyuwenyu/RT-DETR](https://github.com/lyuwenyu/RT-DETR/tree/main/rtdetr_pytorch) |
+| COCO-val | [yolov9-tiny](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/yolov9/yolov9_tiny-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/yolov9/yolov9_tiny_coco.safetensors) | (640, 640) | 50.03 | 38.63 | 36.02 | 2.44M | 9.99G | Supported | No input z-norm |
+| COCO-val | [yolov9-s*](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/yolov9/yolov9_s-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/yolov9/yolov9_s_coco.safetensors) | (640, 640) | 62.63 | 51.13 | 47.13 | 7.23M | 26.87G | Supported | No input z-norm, [YOLO](https://yolo-docs.readthedocs.io/en/latest/2_model_zoo/0_object_detection.html) |
+| COCO-val | [yolov9-m*](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/yolov9/yolov9_m-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/yolov9/yolov9_m_coco.safetensors) | (640, 640) | 67.43 | 56.13 | 51.72 | 20.12M | 77.08G | Supported | No input z-norm, [YOLO](https://yolo-docs.readthedocs.io/en/latest/2_model_zoo/0_object_detection.html) |
+| COCO-val | [yolov9-c*](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/yolov9/yolov9_c-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/yolov9/yolov9_c_coco.safetensors) | (640, 640) | 69.16 | 57.90 | 53.28 | 25.50M | 103.17G | Supported | No input z-norm, [YOLO](https://yolo-docs.readthedocs.io/en/latest/2_model_zoo/0_object_detection.html) |
 | COCO-val | [YOLOX-nano*](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/yolox/yolox-nano-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/yolox/yolox_nano_coco.safetensors?versionId=JCXugDTwGegx9Kl6Jc5AMJpIkA.WlNVP) | (416, 416) | 41.30 | 27.90 | 26.33 | 0.91M | 1.08G | Supported | [Megvii-BaseDetection/YOLOX](https://github.com/Megvii-BaseDetection/YOLOX?tab=readme-ov-file#benchmark), conf_thresh=0.01, nms_thresh=0.65 |
 | COCO-val | [YOLOX-tiny*](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/yolox/yolox-tiny-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/yolox/yolox_tiny_coco.safetensors?versionId=lJp1bCEToD_6IaL9kRCqcYIwVZ.QQ.1P) | (416, 416) | 50.69 | 36.18 | 34.00 | 5.06M | 6.45G | Supported | [Megvii-BaseDetection/YOLOX](https://github.com/Megvii-BaseDetection/YOLOX?tab=readme-ov-file#benchmark), conf_thresh=0.01, nms_thresh=0.65 |
 | COCO-val | [YOLOX-s](https://github.com/Nota-NetsPresso/netspresso-trainer/blob/master/config/model/yolox/yolox-s-detection.yaml) | [download](https://netspresso-trainer-public.s3.ap-northeast-2.amazonaws.com/checkpoint/yolox/yolox_s_coco.safetensors?versionId=QRLqHKqhv8TSYBrmsQ3M8lCR8w7HEZyA) | (640, 640) | 58.56 | 44.10 | 40.63 | 8.97M | 26.81G | Supported | conf_thresh=0.01, nms_thresh=0.65 |