Performance metrics (#401)

Author: mgumowsk

.github/workflows/pre_commit.yml

@@ -55,4 +55,4 @@ jobs:
           uv sync --locked --extra tests --extra ovms
       - name: Run python unit tests
         run: |
-          uv run pytest tests/unit
+          uv run pytest tests/unit --cov

docs/source/guides/index.md

@@ -5,4 +5,5 @@
 :hidden:
 
 ./model-configuration
+./performance_metrics
 ```

docs/source/guides/performance_metrics.md

@@ -0,0 +1,269 @@
+# Performance Metrics
+
+The Model API provides comprehensive performance monitoring capabilities through the `PerformanceMetrics` class. This allows to measure and analyze the performance of model inference pipeline, including detailed timing information for each stage of the inference process.
+
+## Overview
+
+Performance metrics are automatically collected during model inference and include information for:
+
+- **Model loading time**: Time spent loading the model to the inference device
+- **Preprocessing time**: Time spent on input data preprocessing
+- **Inference time**: Time spent on actual model inference on the device
+- **Postprocessing time**: Time spent on output data postprocessing
+- **Total time**: Overall time for the complete inference pipeline
+- **Total minimal time**: Overall minimum time for the complete inference pipeline
+- **Total maxmium time**: Overall maximum time for the complete inference pipeline
+- **Total frames**: Total number of inferences
+- **FPS**: Frames Per Second
+
+Each metric provides statistical information including mean, standard deviation, and individual measurements.
+
+## Basic Usage
+
+### Accessing Performance Metrics
+
+Every model instance automatically collects performance metrics. You can access them using the `get_performance_metrics()` method:
+
+```python
+from model_api.models import Model
+import cv2
+
+# Create a model
+model = Model.create_model("path/to/your/model.xml")
+
+# Perform inference
+image = cv2.imread("path/to/image.jpg")
+result = model(image)
+
+# Get performance metrics
+metrics = model.get_performance_metrics()
+```
+
+### Logging Performance Metrics
+
+The simplest way to view performance metrics is to use the built-in logging method:
+
+```python
+# Configure logging
+logging.basicConfig(level=logging.INFO, format='%(message)s')
+
+# Log all performance metrics to console
+metrics.log_metrics()
+```
+
+This will output detailed performance information:
+
+```bash
+============================================================
+               🚀 PERFORMANCE METRICS REPORT 🚀
+============================================================
+
+📊 Model Loading:
+   Load Time: 2.497s
+
+⚙️  Processing Times (mean ± std):
+   Preprocess:  0.001s ± 0.000s
+   Inference:   0.570s ± 0.020s
+   Postprocess: 0.001s ± 0.000s
+
+📈 Total Time Statistics:
+   Mean:  0.572s ± 0.020s
+   Min:   0.556s
+   Max:   0.642s
+
+🎯 Performance Summary:
+   Total Frames: 100
+   FPS:          1.75
+============================================================
+```
+
+## Detailed Metrics Access
+
+### Individual Timing Statistics
+
+You can access individual timing statistics for more detailed analysis:
+
+```python
+# Get specific timing statistics
+load_time = metrics.get_load_time()
+preprocess_time = metrics.get_preprocess_time()
+inference_time = metrics.get_inference_time()
+postprocess_time = metrics.get_postprocess_time()
+total_time = metrics.get_total_time()
+total_min_time = metrics.get_total_time_min()
+total_max_time = metrics.get_total_time_max()
+
+# Access statistical information
+print(f"Mean inference time: {inference_time.mean():.3f} seconds")
+print(f"Standard deviation: {inference_time.stddev():.3f} seconds")
+print(f"Total inference time: {inference_time.time:.3f} seconds")
+print(f"Number of inferences: {inference_time.count}")
+```
+
+### Frame Rate and Throughput
+
+```python
+# Get frames per second and total frame count
+fps = metrics.get_fps()
+total_frames = metrics.get_total_frames()
+
+print(f"Processed {total_frames} frames at {fps:.2f} FPS")
+```
+
+## Advanced Usage
+
+### Batch Processing Performance
+
+When processing multiple inputs, performance metrics accumulate across all inferences:
+
+```python
+import cv2
+from model_api.models import DetectionModel
+
+model = DetectionModel.create_model("path/to/detection/model.xml")
+
+# Process multiple images
+images = ["image1.jpg", "image2.jpg", "image3.jpg"]
+for image_path in images:
+    image = cv2.imread(image_path)
+    result = model(image)
+
+# Get accumulated metrics for all inferences
+metrics = model.get_performance_metrics()
+metrics.log_metrics()
+```
+
+### Performance Monitoring During Inference
+
+```python
+import cv2
+from model_api.models import ClassificationModel
+
+model = ClassificationModel.create_model("efficientnet-b0-pytorch")
+image = cv2.imread("test_image.jpg")
+
+# Run multiple inferences and monitor performance
+for i in range(100):
+    result = model(image)
+
+    # Check performance every 10 inferences
+    if (i + 1) % 10 == 0:
+        metrics = model.get_performance_metrics()
+        print(f"After {i + 1} inferences:")
+        print(f"  Mean inference time: {metrics.get_inference_time().mean():.3f}s")
+        print(f"  Current FPS: {metrics.get_fps():.2f}")
+```
+
+## Performance Optimization Tips
+
+### Analyzing Bottlenecks
+
+Use performance metrics to identify bottlenecks in inference pipeline:
+
+```python
+metrics = model.get_performance_metrics()
+
+preprocess_time = metrics.get_preprocess_time().mean()
+inference_time = metrics.get_inference_time().mean()
+postprocess_time = metrics.get_postprocess_time().mean()
+
+print("Time breakdown:")
+print(f"  Preprocessing: {preprocess_time:.3f}s ({preprocess_time/total:.1%})")
+print(f"  Inference:     {inference_time:.3f}s ({inference_time/total:.1%})")
+print(f"  Postprocessing: {postprocess_time:.3f}s ({postprocess_time/total:.1%})")
+
+total = preprocess_time + inference_time + postprocess_time
+```
+
+### Warm-up Considerations
+
+The first few inferences may be slower due to system warm-up. Consider excluding them from performance analysis:
+
+```python
+# Warm-up inferences
+for _ in range(5):
+    model(image)
+
+# Reset metrics after warm-up
+model.get_performance_metrics().reset()
+
+# Now measure actual performance
+for _ in range(100):
+    model(image)
+
+metrics = model.get_performance_metrics()
+metrics.log_metrics()
+```
+
+## Best Practices
+
+1. **Warm-up Period**: Always include a warm-up period before measuring performance for production benchmarks.
+
+2. **Multiple Runs**: Collect metrics over multiple inference runs to get statistically significant results.
+
+3. **Reset Between Tests**: Reset metrics when comparing different configurations or models.
+
+4. **Monitor All Stages**: Pay attention to all pipeline stages (preprocessing, inference, postprocessing) to identify bottlenecks.
+
+5. **Environment Consistency**: Ensure consistent testing conditions (device state, background processes, etc.) when comparing performance.
+
+## Example: Complete Performance Analysis
+
+```python
+import cv2
+from model_api.models import DetectionModel
+
+def analyze_model_performance(model_path, test_images, warmup_runs=5, test_runs=100):
+    """Complete performance analysis example."""
+
+    # Load model
+    model = DetectionModel.create_model(model_path)
+
+    # Load test image
+    image = cv2.imread(test_images[0])
+
+    print("Starting warm-up...")
+    # Warm-up runs
+    for _ in range(warmup_runs):
+        model(image)
+
+    # Reset metrics after warm-up
+    model.get_performance_metrics().reset()
+
+    print(f"Running {test_runs} test inferences...")
+    # Performance measurement runs
+    for i, image_path in enumerate(test_images[:test_runs]):
+        image = cv2.imread(image_path)
+        result = model(image)
+
+        # Log progress
+        if (i + 1) % 10 == 0:
+            print(f"  Completed {i + 1}/{test_runs}")
+
+    # Analyze results
+    metrics = model.get_performance_metrics()
+
+    print("\n" + "="*50)
+    print("PERFORMANCE ANALYSIS RESULTS")
+    print("="*50)
+
+    metrics.log_metrics()
+
+    # Additional analysis
+    inference_time = metrics.get_inference_time()
+    print(f"\nInference time analysis:")
+    print(f"  Minimum: {min(inference_time.durations):.3f}s")
+    print(f"  Maximum: {max(inference_time.durations):.3f}s")
+    print(f"  Median: {sorted(inference_time.durations)[len(inference_time.durations)//2]:.3f}s")
+
+    return metrics
+
+# Usage
+if __name__ == "__main__":
+    model_path = "path/to/your/model.xml"
+    test_images = ["image1.jpg", "image2.jpg", "image3.jpg"]  # Add more images
+
+    metrics = analyze_model_performance(model_path, test_images)
+```
+
+This comprehensive performance monitoring system helps optimize model inference pipeline and ensure optimal performance in production deployments.

examples/metrics/README.md

@@ -0,0 +1,99 @@
+# Benchmark - a metrics API example
+
+This example demonstrates how to use the Python API of OpenVINO Model API for performance analysis and metrics collection during model inference. This tutorial includes the following features:
+
+- Model performance measurement
+- Configurable device selection (CPU, GPU, etc.)
+- Automatic image dataset discovery
+- Warm-up and test runs with customizable parameters
+- Detailed inference time analysis
+- Metrics logging and reporting
+- Performance statistics calculation
+
+## Prerequisites
+
+Install Model API from source. Please refer to the main [README](../../../README.md) for details.
+
+## Run example
+
+To run the example, please execute the following command:
+
+```bash
+python benchmark.py <model_path> <dataset_path> [options]
+```
+
+### Required Arguments
+
+- `model_path` - Path to the model file (.xml)
+- `dataset_path` - Path to the dataset directory containing test images
+
+### Optional Arguments
+
+- `--device` - Device to run the model on (default: CPU)
+- `--warmup-runs` - Number of warmup runs (default: 5)
+- `--test-runs` - Number of test runs (default: 100)
+
+### Examples
+
+```bash
+# Basic usage with CPU
+python benchmark.py /path/to/model.xml /path/to/images
+
+# Use GPU with custom parameters
+python benchmark.py /path/to/model.xml /path/to/images --device GPU --warmup-runs 10 --test-runs 50
+
+# Show help
+python benchmark.py --help
+```
+
+## Expected Output
+
+The example will display:
+
+- Number of images found in the dataset directory
+- Progress updates during warm-up and test phases
+- Comprehensive performance analysis results including timing statistics
+- Detailed metrics about the model's inference performance on the specified device
+
+Example output
+
+```bash
+OpenVINO Runtime
+   build: 2025.2.0-19140-c01cd93e24d-releases/2025/2
+Reading model model.xml
+The model model.xml is loaded to CPU
+   Number of model infer requests: 2
+Starting warm-up...
+Running 100 test inferences...
+  Completed 10/100
+  Completed 20/100
+  Completed 30/100
+  Completed 40/100
+  Completed 50/100
+  Completed 60/100
+  Completed 70/100
+  Completed 80/100
+  Completed 90/100
+  Completed 100/100
+============================================================
+               🚀 PERFORMANCE METRICS REPORT 🚀
+============================================================
+
+📊 Model Loading:
+   Load Time: 2.497s
+
+⚙️  Processing Times (mean ± std):
+   Preprocess:  0.001s ± 0.000s
+   Inference:   0.570s ± 0.020s
+   Postprocess: 0.001s ± 0.000s
+
+📈 Total Time Statistics:
+   Mean:  0.572s ± 0.020s
+   Min:   0.556s
+   Max:   0.642s
+
+🎯 Performance Summary:
+   Total Frames: 100
+   FPS:          1.75
+============================================================
+```