Skip to content

windows_developer_guide.md

Latest commit

 

History

History
139 lines (113 loc) · 5.49 KB

windows_developer_guide.md

File metadata and controls

139 lines (113 loc) · 5.49 KB

OpenVINO™ Model Server Developer Guide for Windows

This document describes windows development and compilation guide for ovms.exe binary. This instruction was tested on Windows 11 and Windows 10 OS.

List of disabled features on Windows model server:

  • cloud storage for the models - to be added in next releases
  • C-API interface - to be added in next releases
  • DAG pipelines - legacy feature

Install prerequisites

Following the steps below requires 40GB of free disk space.

VISUAL BUILD TOOLS

Install build tools for VS:

https://aka.ms/vs/17/release/vs_BuildTools.exe

Mark required options for installation:

  • C++ Desktop development with C++
  • Windows 11 SDK (10.0.26210.0)
  • MSVC v143 CPP - VS 2022 C++ platform toolset.
  • C++ CMake tools for Windows platform toolset.
  • MSVC v142 CPP - VS 2022 C++ platform toolset.
  • Optional Windows 11 SDK (10.0.26100.0) for Windows 10 compilation

Build Tools options

Power shell settings

Set Execution Policy to RemoteSigned Open PowerShell as an administrator: Right-click on the Start button and select “Windows PowerShell (Admin)”. Run the command: Set-ExecutionPolicy Unrestricted -Scope CurrentUser -Force

Enable Developer mode in windows system settings

Follow instructions in the link below: https://learn.microsoft.com/en-us/windows/apps/get-started/enable-your-device-for-development

Run Command Prompt

Press Windows Start and run the cmd.exe terminal as Administrator. Run commands in this prompt is not stated otherwise.

Pull OpenVINO Model Server source

Git is required to complete this step. If you don't have it on your system, download it from https://git-scm.com/downloads/win and install before you continue. Run below commands in terminal to clone model server repository:

mkdir C:\git
cd C:\git\
git clone https://github.com/openvinotoolkit/model_server.git
cd model_server

Install dependencies

Run windows_install_build_dependencies.bat This will install around 6 GB dependencies in the c:\opt directory:

  • wet.exe, msys2 tools, Openvinotoolkit for GenAI, OpenCL headers, BoringSSL, bazel, Python 3.12.10, OpenCV, Curl

If error occurs during the script execution, please fix the error and rerun the script.

windows_install_build_dependencies.bat

Optionally, you add parameter to the windows_install_build_dependencies.bat script

windows_install_build_dependencies.bat my_dir_on_c 1 1

[arg1] - This way you can change default dependency install directory to c:\my_dir_on_c [arg2] - Set the clean flag to 0 or 1 - this will cleanup the installation directories and reinstall all dependencies [arg3] - Add the compilation integrity flag to 0 or 1 - set the additional integritycheck and Qspectre compilation flag when compiling dependencies

COMPILE

[WARNING] This step consumes up to 13GB of disk space. It can take up to 1h depending on host CPU and internet connection speed. This default command compiles ovms.exe without python dependencies, just C++ binary with limited support for chat template processing.

windows_build.bat

Optionally, you add parameter to the windows_build.bat script

windows_install_build_dependencies.bat my_dir_on_c --with_python --with_tests --integrity

[arg1] This way you can change default dependency location directory to c:\my_dir_on_c [arg2] --with_python - this will build the ovms.exe with python dependency and support for python chat templates for GENAI LLM [arg3] --with_tests - this will also build ovms_test.exe target [arg4] --integrity - Add the compilation integrity flag to 0 or 1 - set the additional integritycheck compilation flag when compiling dependencies

Running unit tests - optional

The script compiles ovms_test binary with C++ only, downloads and converts test LLM models (src\tests\llm_testing).

windows_test.bat

The optional script compiles ovms_test binary with python support, downloads and converts test LLM models (src\tests\llm_testing) and installs Python torch and optimum.

windows_test.bat opt --with_python

Creating deployment package

This step prepares ovms.zip deployment package from the build artifacts in the dist\windows\ directory. Run this script after successful compilation. The default version creates C++ only version without Python dependency.

windows_create_package.bat

Optionally you can create a package with Python dependency. Note that to create valid package with Python, you need to build using --with_python flag in the previous step as well.

windows_create_package.bat opt --with_python

Test the Deployment

You can follow the baremetal deployment guide for information how to deploy and use the ovms.zip package.

Developer Command Prompt

For building ovms.exe and running ovms_test.exe with manual bazel commands you must setup proper environment variables. Run the batch script in new "Developer Command Prompt for VS 2022" terminal:

cd c:\git\model_server
windows_setupvars.bat

Rebuild unit tests:

bazel --output_user_root=c:\opt build --config=windows --action_env OpenVINO_DIR=c:\opt\genai/runtime/cmake --jobs=%NUMBER_OF_PROCESSORS% --verbose_failures //src:ovms_test 2>&1 | tee win_build_test.log

Download LLMs

%cd%\windows_prepare_llm_models.bat %cd%\src\test\llm_testing

Change tests configs to windows:

python windows_change_test_configs.py

Run specific unit tests by setting gtest_filter:

%cd%\bazel-bin\src\ovms_test.exe --gtest_filter=* 2>&1 | tee win_full_test.log