04.SLMOps.Deployment.md

Section 4: Deployment - Production-Ready Model Implementation

Overview

This comprehensive tutorial will guide you through the complete process of deploying fine-tuned quantized models using Foundry Local. We'll cover model conversion, quantization optimization, and deployment configuration from start to finish.

Prerequisites

Before getting started, ensure you have the following:

• ✅ A fine-tuned onnx model ready for deployment
• ✅ Windows or Mac computer
• ✅ Python 3.10 or higher
• ✅ At least 8GB available RAM
• ✅ Foundry Local installed on your system

Part 1: Environment Setup

Installing Required Tools

Open your terminal (Command Prompt on Windows, Terminal on Mac) and run the following commands in sequence:

# Update transformers library to the latest version
pip install transformers -U

# Install Microsoft Olive (model conversion tool)
pip install git+https://github.com/microsoft/Olive.git

# Install ONNX Runtime GenAI
git clone https://github.com/microsoft/onnxruntime-genai
cd onnxruntime-genai && python build.py --config Release
pip install {Your build release path}/onnxruntime_genai-0.9.0.dev0-cp311-cp311-linux_x86_64.whl

# Install additional dependencies
pip install onnx onnxruntime numpy

⚠️ Important Note: You'll also need CMake version 3.31 or newer, which can be downloaded from cmake.org.

Part 2: Model Conversion and Quantization

Choosing the Right Format

For fine-tuned small language models, we recommend using ONNX format because it offers:

• 🚀 Better performance optimization
• 🔧 Hardware-agnostic deployment
• 🏭 Production-ready capabilities
• 📱 Cross-platform compatibility

Method 1: One-Command Conversion (Recommended)

Use the following command to directly convert your fine-tuned model:

olive auto-opt \
    --model_name_or_path /path/to/your/finetuned/model \
    --device cpu \
    --provider CPUExecutionProvider \
    --use_model_builder \
    --precision int4 \
    --output_path models/your-model-name/onnx \
    --log_level 1

Parameter Explanation:

• --model_name_or_path: Path to your fine-tuned model
• --device cpu: Use CPU for optimization
• --precision int4: Use INT4 quantization (approximately 75% size reduction)
• --output_path: Output path for the converted model

Method 2: Configuration File Approach (Advanced Users)

Create a configuration file named finetuned_conversion_config.json:

{
    "input_model": {
        "type": "HfModel",
        "model_path": "/path/to/your/finetuned/model",
        "task": "text-generation"
    },
    "systems": {
        "local_system": {
            "type": "LocalSystem",
            "accelerators": [
                {
                    "execution_providers": [
                        "CPUExecutionProvider"
                    ]
                }
            ]
        }
    },
    "passes": {
        "builder": {
            "type": "ModelBuilder",
            "config": {
                "precision": "int4",
                "quantization_config": {
                    "block_size": 128,
                    "is_symmetric": false
                }
            }
        }
    },
    "host": "local_system",
    "target": "local_system",
    "cache_dir": "cache",
    "output_dir": "models/output/your-finetuned-model-onnx"
}

Then run:

olive run --config ./finetuned_conversion_config.json

Quantization Options Comparison

Precision	File Size	Inference Speed	Model Quality	Recommended Use
FP16	Baseline × 0.5	Fast	Best	High-end hardware
INT8	Baseline × 0.25	Very Fast	Good	Balanced choice
INT4	Baseline × 0.125	Fastest	Acceptable	Resource-limited

💡 Recommendation: Start with INT4 quantization for your first deployment. If quality isn't satisfactory, try INT8 or FP16.

Part 3: Foundry Local Deployment Configuration

Creating Model Configuration

Navigate to the Foundry Local models directory:

foundry cache cd ./models/

Create your model directory structure:

mkdir -p ./models/custom/your-finetuned-model

Create the inference_model.json configuration file in your model directory:

{
  "Name": "your-finetuned-model-int4",
  "Description": "Fine-tuned quantized model",
  "Version": "1.0",
  "PromptTemplate": {
    "system": "<|im_start|>system\n{Content}<|im_end|>",
    "user": "<|im_start|>user\n{Content}<|im_end|>",
    "assistant": "<|im_start|>assistant\n{Content}<|im_end|>",
    "prompt": "<|im_start|>user\n{Content}<|im_end|>\n<|im_start|>assistant"
  },
  "ModelConfig": {
    "max_length": 2048,
    "temperature": 0.7,
    "top_p": 0.9,
    "repetition_penalty": 1.1
  }
}

Model-Specific Template Configurations

For Qwen Series Models:

{
  "Name": "qwen-finetuned-int4",
  "PromptTemplate": {
    "system": "<|im_start|>system\n{Content}<|im_end|>",
    "user": "<|im_start|>user\n{Content}<|im_end|>",
    "assistant": "<|im_start|>assistant\n{Content}<|im_end|>",
    "prompt": "<|im_start|>user\n{Content}<|im_end|>\n<|im_start|>assistant"
  }
}

Part 4: Model Testing and Optimization

Verifying Model Installation

Check if Foundry Local can recognize your model:

foundry cache ls

You should see your-finetuned-model-int4 in the list.

Starting Model Testing

foundry model run your-finetuned-model-int4

Performance Benchmarking

Monitor key metrics during testing:

1. Response Time: Measure average time per response
2. Memory Usage: Monitor RAM consumption
3. CPU Utilization: Check processor load
4. Output Quality: Evaluate response relevance and coherence

Quality Validation Checklist

• ✅ Model responds appropriately to fine-tuned domain queries
• ✅ Response format matches expected output structure
• ✅ No memory leaks during extended usage
• ✅ Consistent performance across different input lengths
• ✅ Proper handling of edge cases and invalid inputs

Summary

Congratulations! You have successfully completed:

• ✅ Fine-tuned model format conversion
• ✅ Model quantization optimization
• ✅ Foundry Local deployment configuration
• ✅ Performance tuning and troubleshooting