Merge branch 'sam2.1_hiera_small' of https://github.com/CodeLinaro/olive-recipes into sam_vit_base

Combining recipes for facebook/sam-vit-base and facebook/sam2.1-hiera-small

Author: Shiva Chilukamari

sam2.1-hiera-small/LICENSE

@@ -0,0 +1,22 @@
+Microsoft.
+Copyright (c) Microsoft Corporation.
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sam2.1-hiera-small/QNN/README.md

@@ -0,0 +1,61 @@
+# SAM2 Model Conversion
+
+This repository demonstrates the optimization of the [sam2.1-hiera-small](https://github.com/facebookresearch/sam2) model using **post-training quantization (PTQ)** techniques.
+
+
+### Quantization Python Environment Setup
+Quantization is resource-intensive and requires GPU acceleration
+
+For GPU Environment Setup:
+```bash
+pip install -r requirements.txt
+```
+
+### AOT Compilation Python Environment Setup
+Model compilation using QNN Execution Provider requires a Python environment with onnxruntime-qnn installed. In a separate Python environment, install the required packages:
+
+```bash
+# Install Olive & ORT-QNN
+pip install olive-ai onnxruntime-qnn torch torchvision transformers
+```
+
+Replace `/path/to/qnn/env/bin` in [sam21_vision_encoder_qnn_ctx.json](sam21_vision_encoder_qnn_ctx.json) and [sam21_mask_decoder_qnn_ctx.json](sam21_mask_decoder_qnn_ctx.json) with the path to the directory containing your QNN environment's Python executable. This path can be found by running the following command in the environment:
+
+```bash
+# Linux
+command -v python
+# Windows
+# where python
+```
+
+This command will return the path to the Python executable. Set the parent directory of the executable as the `/path/to/qnn/env/bin` in the config file.
+
+### Generate ONNX Model
+Activate the **Quantization Python Environment** and run command to generate encoder and decoder models:
+
+```bash
+python generate_model.py
+```
+
+### Run the Quantization + Compilation Config
+Activate the **Quantization Python Environment** and run the workflow:
+
+For Encoder Model:
+```bash
+olive run --config sam21_vision_encoder_qnn_ctx.json
+```
+
+For Decoder Model:
+```bash
+olive run --config sam21_mask_decoder_qnn_ctx.json
+```
+
+> ⚠️ If optimization fails during context binary generation, rerun the command. The process will resume from the last completed step.
+
+### Model ORT Execution
+
+Execute SAM model in **AOT Compilation Python Environment** using following command:
+
+```bash
+python sam2_mask_generator.py --model_ve path/to/encoder_model.onnx --model_md path/to/decoder_model.onnx --image_path car.png --box_x 40 --box_y 235 --box_w 940 --box_h 490 --output_path car_mask.png
+```

sam2.1-hiera-small/QNN/config.py

@@ -0,0 +1,14 @@
+# -------------------------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# Licensed under the MIT License.
+# --------------------------------------------------------------------------
+
+
+class ModelConfig:
+    model_name = "facebook/sam2.1-hiera-small"
+    data_dir = "quantization_dataset_100"
+    ve_input_name = "pixel_values"
+    ve_channel_size = 3
+    ve_sample_size = 1024
+    md_input_names = ("image_embeddings", "high_res_features1", "high_res_features2", "coords.1", "labels")
+    ms_input_shapes = ((1, 256, 64, 64), (1, 32, 256, 256), (1, 64, 128, 128), (1, 5, 2), (1, 5))