Skip to content

Commit 1523624

Browse files
authored
Changing whisper device to GPU (#4154)
### 🛠 Summary Changing whisper device to GPU ### 🧪 Checklist - [ ] Unit tests added. - [ ] The documentation updated. - [ ] Change follows security best practices. ``
1 parent 431527d commit 1523624

1 file changed

Lines changed: 4 additions & 4 deletions

File tree

demos/integration_with_OpenWebUI/README.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ There are other options to fulfill the prerequisites like [OpenVINO Model Server
2525

2626
This demo can be followed without changes on Panther Lake host with 64GB RAM and VRAM allocation to GPU extended using Intel Graphics Software. That way all the mentioned models can be loaded simultaneously. It's also possible to use [llama-swap](https://github.com/openvinotoolkit/model_server/blob/main/extras/llama_swap/README.md) integration to reload the models automatically. On hosts with less VRAM available, use a subset of the models, apply other models or configure different target device like CPU or NPU. Check this list of [preconfigured OpenVINO models](https://huggingface.co/OpenVINO).
2727

28-
## Step 1: Pull model and start the OVMS sever
28+
## Step 1: Pull model and start the OVMS server
2929
::::{tab-set}
3030
:::{tab-item} Windows
3131
:sync: Windows
@@ -363,7 +363,7 @@ mcpo --port 9000 -- python -m mcp_weather_server
363363

364364
## Using Web Search
365365

366-
### Step 1: Configure WebSearch
366+
### Step 1: Configure Web Search
367367

368368
1. Go to **Admin Panel****Settings****Web Search**
369369
2. Enable **Web Search**
@@ -474,14 +474,14 @@ Next, download and add to config model for transcription:
474474
:::{tab-item} Windows
475475
:sync: Windows
476476
```bat
477-
ovms.exe --pull --source_model OpenVINO/whisper-base-fp16-ov --model_repository_path models --task speech2text
477+
ovms.exe --pull --source_model OpenVINO/whisper-base-fp16-ov --model_repository_path models --task speech2text --target_device GPU
478478
ovms.exe --add_to_config --config_path models\config.json --model_path OpenVINO\whisper-base-fp16-ov --model_name OpenVINO/whisper-base-fp16-ov
479479
```
480480
:::
481481
:::{tab-item} Linux (using Docker)
482482
:sync: Linux
483483
```bash
484-
docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --pull --source_model OpenVINO/whisper-base-fp16-ov --model_repository_path /models --task speech2text
484+
docker run --rm -u $(id -u):$(id -g) --device /dev/dri --group-add=$(stat -c "%g" /dev/dri/render* | head -n 1) -v $PWD/models:/models openvino/model_server:weekly --pull --source_model OpenVINO/whisper-base-fp16-ov --model_repository_path /models --task speech2text --target_device GPU
485485
docker run --rm -u $(id -u):$(id -g) -v $PWD/models:/models openvino/model_server:weekly --add_to_config --config_path /models/config.json --model_path OpenVINO/whisper-base-fp16-ov --model_name OpenVINO/whisper-base-fp16-ov
486486
```
487487
:::

0 commit comments

Comments
 (0)