diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/1_overview.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/1_overview.md index ff033f06c3..4106035d48 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/1_overview.md +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/1_overview.md @@ -5,7 +5,7 @@ layout: learningpathall --- Voice-based LLM applications often rely primarily on transcribed text from speech input, such as in interactions with non-player characters in games or voice assistants. This approach can overlook vocal cues—like tone, pitch, and emotion—present in a speaker’s voice. As a result, responses may feel less natural and may not fully capture the user’s underlying intent. -To address this, voice-based sentiment classification analyzes audio input to determine the user’s emotional state, which is then incorporated into the LLM prompt to enable more context-aware responses. In this Learning Path, we will build a sentiment-aware voice assistant that runs entirely on-device. The application records audio, performs transcription—converting speech into written text—using Whisper, classifies sentiment directly from the voice signal, and combines the transcript and voice-based sentiment to guide responses from a local LLM running with llama.cpp. +To address this, voice-based sentiment classification analyzes audio input to determine the user’s emotional state, which is then incorporated into the LLM prompt to enable more context-aware responses. In this Learning Path, you will build a sentiment-aware voice assistant that runs entirely on-device. The application records audio, performs transcription—converting speech into written text—using Whisper, classifies sentiment directly from the voice signal, and combines the transcript and voice-based sentiment to guide responses from a local LLM running with llama.cpp. ![Voice sentiment classification pipeline#center](1_vsapipeline2.png "Voice sentiment classification pipeline") diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/2_set-up-your-environment.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/2_set-up-your-environment.md index e59320dca9..a0d0ac63f5 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/2_set-up-your-environment.md +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/2_set-up-your-environment.md @@ -10,43 +10,69 @@ Before building the voice assistant, create a project workspace and set up an is These instructions support Ubuntu, macOS, and Windows, with Python 3.9 or later and a working microphone. -Check your Python version before continuing: +Install required system tools first. `ffmpeg` is required by Whisper for audio decoding. + +{{< tabpane code=true >}} + {{< tab header="Ubuntu/macOS" language="bash">}} +# Ubuntu +sudo apt update +sudo apt install -y ffmpeg git cmake + +# macOS +brew install ffmpeg git cmake + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +# Install via WinGet +winget install -e --id Gyan.FFmpeg +winget install -e --id Git.Git +winget install -e --id Kitware.CMake + {{< /tab >}} +{{< /tabpane >}} -**Ubuntu or macOS** +Check your Python version before continuing: -```bash +{{< tabpane code=true >}} + {{< tab header="Ubuntu/macOS" language="bash">}} python3 --version -``` - -**Windows PowerShell** - -```powershell + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} py -3 --version -``` + {{< /tab >}} +{{< /tabpane >}} ## Set up the Python environment with UV -Install `UV` first from PyPI using `pip`. `UV` is a fast Python package and environment manager that we will use throughout this Learning Path to create the project environment and install dependencies. - -**Ubuntu or macOS** - -```bash +Install `UV` first so the `uv` command is available in your terminal. `UV` is a fast Python package and environment manager that you will use throughout this Learning Path to create the project environment and install dependencies. + +{{< tabpane code=true >}} + {{< tab header="Ubuntu/macOS" language="bash">}} +curl -LsSf https://astral.sh/uv/install.sh | sh +# Start a new shell, or source your shell rc file so `uv` is on PATH. +uv --version + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} +powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex" +# Open a new PowerShell window so `uv` is on PATH. +uv --version + {{< /tab >}} +{{< /tabpane >}} + +Create and activate the project virtual environment: + +{{< tabpane code=true >}} + {{< tab header="Ubuntu/macOS" language="bash">}} mkdir -p ~/voice-sentiment-assistant cd ~/voice-sentiment-assistant -python3 -m pip install uv uv venv .venv source .venv/bin/activate -``` - -**Windows PowerShell** - -```powershell + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} mkdir $HOME\voice-sentiment-assistant -Force cd $HOME\voice-sentiment-assistant -py -3 -m pip install uv uv venv .venv .\.venv\Scripts\Activate.ps1 -``` + {{< /tab >}} +{{< /tabpane >}} Keep this virtual environment activated while you complete the rest of the Learning Path. @@ -63,6 +89,7 @@ numpy librosa scikit-learn onnx +onnxscript onnxruntime ``` @@ -78,41 +105,50 @@ This installs the libraries needed for the Gradio interface, Whisper transcripti Next, clone the [llama.cpp GitHub repository](https://github.com/ggml-org/llama.cpp), build the local inference server, and start it. This server exposes an OpenAI-compatible API that the Python application will call later in the Learning Path. -**Ubuntu or macOS** +{{% notice Pre-built llama.cpp %}} +If you prefer not to build from source, you can use pre-built binaries from [llama.cpp releases](https://github.com/ggml-org/llama.cpp/releases). Download the package for your platform, extract it, and use the `llama-server` executable from that package in the run commands later in this section. +{{% /notice %}} + -```bash +{{< tabpane code=true >}} + {{< tab header="Ubuntu/macOS" language="bash">}} git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build cmake --build build --config Release -``` - -**Windows PowerShell** - -```powershell + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build cmake --build build --config Release + {{< /tab >}} +{{< /tabpane >}} + +When the build completes, the `llama-server` executable should be available in the build output directory. + +``` +ls ./build/bin/llama-server ``` -When the build completes, the `llama-server` executable should be available in the build output directory. This Learning Path uses a quantized [Gemma 3 1B instruction-tuned model](https://huggingface.co/google/gemma-3-1b-it) served locally through `llama.cpp`. +{{% notice Note %}} +The file verification commands in this Learning Path uses syntax for Ubuntu and macOS. If you're on Windows, adjust the commands to use PowerShell equivalents like `Test-Path .\build\bin\llama-server.exe` or `dir .\build\bin\`. The file location remains the same across all platforms. +{{% /notice %}} -The first time you run this command, `llama.cpp` will download the model from Hugging Face. This can take several minutes depending on your network connection. +This Learning Path uses a quantized [Gemma 3 1B instruction-tuned model](https://huggingface.co/google/gemma-3-1b-it) served locally through `llama.cpp`. -**Ubuntu or macOS** +The first time you run this command, `llama.cpp` will download the model from Hugging Face. This can take several minutes depending on your network connection. Run the following command from the `llama.cpp` directory: -```bash +{{< tabpane code=true >}} + {{< tab header="Ubuntu/macOS" language="bash">}} ./build/bin/llama-server -hf ggml-org/gemma-3-1b-it-GGUF -``` - -**Windows PowerShell** - -```powershell + {{< /tab >}} + {{< tab header="Windows PowerShell" language="powershell">}} .\build\bin\Release\llama-server.exe -hf ggml-org/gemma-3-1b-it-GGUF -``` + {{< /tab >}} +{{< /tabpane >}} Leave this terminal running while you test the application in later steps. The server listens on a local OpenAI-compatible endpoint that your app will call to generate responses. diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/3_build-baseline-voice-to-llm-pipeline.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/3_build-baseline-voice-to-llm-pipeline.md index 051cc919a2..c0f7fc56ae 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/3_build-baseline-voice-to-llm-pipeline.md +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/3_build-baseline-voice-to-llm-pipeline.md @@ -24,7 +24,14 @@ Gradio is a Python library for building simple browser-based interfaces. Here, y This is a good first step because it lets you confirm that microphone capture works before you add transcription and model inference. -Create a file called `app.py`: +Using a new terminal, navigate to the project directory you created in the last step and make sure the virtual environment is active: + +``` +cd $HOME/voice-sentiment-assistant +source .venv/bin/activate +``` + +Create a file called `app.py` with the following contents: ```python import gradio as gr diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/4_train-voice-sentiment-model.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/4_train-voice-sentiment-model.md index e6d0251c48..4acb7b70ac 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/4_train-voice-sentiment-model.md +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/4_train-voice-sentiment-model.md @@ -11,12 +11,22 @@ Before you begin, make sure you have completed the environment setup from the pr This example uses a simple per-sample training loop for clarity. On CPU, training can take 10 to 30 minutes or more depending on your machine. Apple Silicon `mps` and CUDA-enabled GPUs are often faster. +All code snippets in Steps 2.1 to 2.5 are meant to be added to a single Python file: `train_sentiment_model.py` in your project root (`$HOME/voice-sentiment-assistant`). + +Make sure you are located in the project directory: + +```bash +cd $HOME/voice-sentiment-assistant +``` + ### Step 2.1 - Prepare the dataset -This step downloads the RAVDESS speech dataset, extracts it locally, and builds a dataframe that maps each audio file to a sentiment label. RAVDESS encodes emotion labels in the filename format, which we use to select the classes for this demo. +This step downloads the RAVDESS speech dataset, extracts it locally, and builds a dataframe that maps each audio file to a sentiment label. RAVDESS encodes emotion labels in the filename format, which you use to select the classes for this demo. In practice, this turns a folder of audio files into a structured table that your training code can work with. Each row is one example, and each column stores details such as the file path, actor ID, and sentiment label. +Add this to a new file called `train_sentiment_model.py`: + ```python import os import urllib.request @@ -25,25 +35,24 @@ import pandas as pd # Download and unpack the dataset into the local project directory. RAVDESS_URL = "https://zenodo.org/records/1188976/files/Audio_Speech_Actors_01-24.zip?download=1" -output_dir = "data/ravdess" -os.makedirs(output_dir, exist_ok=True) +OUTPUT_DIR = "data/ravdess" +ZIP_PATH = f"{OUTPUT_DIR}/Audio_Speech_Actors_01-24.zip" +EXTRACT_DIR = f"{OUTPUT_DIR}/Audio_Speech_Actors_01-24" -zip_path = f"{output_dir}/Audio_Speech_Actors_01-24.zip" -extract_dir = f"{output_dir}/Audio_Speech_Actors_01-24" +os.makedirs(OUTPUT_DIR, exist_ok=True) -if not os.path.exists(zip_path): +if not os.path.exists(ZIP_PATH): print("Downloading dataset...") - urllib.request.urlretrieve(RAVDESS_URL, zip_path) + urllib.request.urlretrieve(RAVDESS_URL, ZIP_PATH) -if not os.path.exists(extract_dir): - zipfile.ZipFile(zip_path).extractall(extract_dir) +if not os.path.exists(EXTRACT_DIR): + zipfile.ZipFile(ZIP_PATH).extractall(EXTRACT_DIR) -DATASET_PATH = "data/ravdess/Audio_Speech_Actors_01-24" emotion_map = {1: "neutral", 3: "happy", 5: "angry"} data = [] -for actor in os.listdir(DATASET_PATH): - actor_path = os.path.join(DATASET_PATH, actor) +for actor in os.listdir(EXTRACT_DIR): + actor_path = os.path.join(EXTRACT_DIR, actor) if not os.path.isdir(actor_path): continue @@ -73,6 +82,8 @@ Next, split the dataset by speaker so the same speaker does not appear in both t This matters because a speech model can accidentally learn speaker-specific traits instead of sentiment if the same speakers appear in both splits. Separating speakers gives you a better measure of how well the model generalizes. The training split is the data the model learns from. The validation split is held back and used only to test how well the model performs on unseen examples. +Append this to the same `train_sentiment_model.py` file: + ```python import random @@ -97,6 +108,8 @@ print(f"Validation samples: {len(val_df)}") This step loads the HuBERT feature extractor and initializes a sequence classification model. The pre-trained HuBERT base model provides a strong starting point for transfer learning on speech sentiment classification. The feature extractor prepares raw audio in the format expected by the model. The HuBERT model then uses those inputs to predict one of the target sentiment classes. +Append this below the previous code in `train_sentiment_model.py`: + ```python from transformers import AutoFeatureExtractor, HubertForSequenceClassification @@ -122,6 +135,8 @@ During training, the model compares its predictions with the correct labels and For better performance in a production training pipeline, you would typically use batching and a PyTorch `DataLoader`. +Append this below the previous code in `train_sentiment_model.py`: + ```python import librosa import torch @@ -200,23 +215,27 @@ At the end of this step, you will have a trained model and a simple validation a Save both the trained model and the feature extractor so they can be reused later for inference and ONNX export. Keeping them together helps avoid preprocessing mismatches. This is an important final step because the model weights alone are not enough. You also need the matching feature extractor so inference uses the same preprocessing as training. -```python -import os +Append this at the end of `train_sentiment_model.py`: +```python SAVE_DIR = "models/hubert_vsa_ravdess" os.makedirs(SAVE_DIR, exist_ok=True) # Save both the trained model and the preprocessing configuration. model.save_pretrained(SAVE_DIR) feature_extractor.save_pretrained(SAVE_DIR) + +print("Training complete. Model saved successfully.") +print(f"Saved model to {SAVE_DIR}") ``` After this step, the trained model should be saved in `models/hubert_vsa_ravdess`. ## Full training script -The following script combines the previous steps into one runnable example. Save it as `train_sentiment_model.py` and run it from the root of your project. -Use this script if you want to run the full workflow without copying each block separately. +The following script combines the previous steps into one runnable example. +If you already built `train_sentiment_model.py` using Steps 2.1 to 2.5, you can skip this section. +If not, copy this full script into `train_sentiment_model.py` and run it from the root of your project. ```python import os diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/5_compress-and-convert-the-model.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/5_compress-and-convert-the-model.md index eae611c416..0a4bb77a74 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/5_compress-and-convert-the-model.md +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/5_compress-and-convert-the-model.md @@ -40,13 +40,24 @@ dummy = feature_extractor( # Export model graph and IO signatures to ONNX. torch.onnx.export( - model, - (dummy["input_values"], dummy["attention_mask"]), - os.path.join(ONNX_DIR, "hubert_vsa_ravdess.onnx"), - input_names=["input_values", "attention_mask"], - output_names=["logits"], - opset_version=18 -) + model, + (dummy["input_values"], dummy["attention_mask"]), + os.path.join(ONNX_DIR, "hubert_vsa_ravdess.onnx"), + input_names=["input_values", "attention_mask"], + output_names=["logits"], + dynamic_axes={ + "input_values": {0: "batch_size", 1: "sequence_length"}, + "attention_mask": {0: "batch_size", 1: "sequence_length"}, + "logits": {0: "batch_size"}, + }, + opset_version=18, + ) +``` + +Run the script: + +``` +python convert_and_quantize_model.py ``` After this step, you should have `hubert_vsa_ravdess.onnx` in the ONNX output directory. @@ -63,18 +74,44 @@ ls models/hubert_vsa_ravdess_onnx This step applies dynamic INT8 quantization to the ONNX model. The model weights are quantized to integer 8 bits ahead of time, while activations are quantized dynamically during inference. -Quantization typically reduces model size by around 3x to 4x and often improves CPU inference speed, depending on the hardware and model. +Quantization typically reduces model size by around 3x to 4x and often improves CPU inference speed, depending on the hardware and model. + +To your `convert_and_quantize_model.py`, add the following snippet to the end of the file: ```python +import onnx from onnxruntime.quantization import quantize_dynamic, QuantType +def sanitize_onnx_for_dynamic_quantization(input_path: str, output_path: str) -> None: + # Remove intermediate value_info entries to avoid shape-inference conflicts + # triggered by ONNX Runtime's internal Gemm->MatMul rewrite. + model = onnx.load(input_path) + while len(model.graph.value_info): + model.graph.value_info.pop() + onnx.save(model, output_path) + +SANITIZED_ONNX_PATH = os.path.join(ONNX_DIR, "hubert_vsa_ravdess.sanitized.onnx") +INT8_ONNX_PATH = os.path.join(ONNX_DIR, "hubert_vsa_ravdess_int8.onnx") + +sanitize_onnx_for_dynamic_quantization( + os.path.join(ONNX_DIR, "hubert_vsa_ravdess.onnx"), + SANITIZED_ONNX_PATH, +) + quantize_dynamic( - model_input="models/hubert_vsa_ravdess_onnx/hubert_vsa_ravdess.onnx", - model_output="models/hubert_vsa_ravdess_onnx/hubert_vsa_ravdess_int8.onnx", - weight_type=QuantType.QInt8 + model_input=SANITIZED_ONNX_PATH, + model_output=INT8_ONNX_PATH, + weight_type=QuantType.QInt8, + op_types_to_quantize=["MatMul", "Gemm"], ) ``` +Re-run the script: + +``` +python convert_and_quantize_model.py +``` + After this step, you should have a smaller quantized model file for deployment. You can verify the quantized model file with: @@ -89,6 +126,9 @@ To compare file sizes more easily, you can also run: ls -lh models/hubert_vsa_ravdess_onnx ``` +- `hubert_vsa_ravdess.onnx` + `hubert_vsa_ravdess.onnx.data` is your original FP32 model split into metadata + external weights +- `hubert_vsa_ravdess_int8.onnx` is the quantized model, where you should be able to observe a smaller model size compared to the original model. + ### Step 3.3 - Run ONNX inference This step validates that the quantized model can run inference with ONNX Runtime. @@ -156,7 +196,7 @@ models/ ### Model compression results -We can also compare the model metrics before and after quantization. +You can also compare the model metrics before and after quantization. ![Model ONNX conversion and int-8 quantization results#center](4_modelconversionandcompression.png "Model ONNX conversion and int-8 quantization results") diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/6_integrate-voice-sentiment-classification-model.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/6_integrate-voice-sentiment-classification-model.md index 3e56280ad9..e17d092531 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/6_integrate-voice-sentiment-classification-model.md +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/6_integrate-voice-sentiment-classification-model.md @@ -9,12 +9,16 @@ End-to-end response time may take a few seconds because the application runs Whi ![Integrated voice-to-LLM pipeline#center](6_vsapipeline4.png "Integrated voice-to-LLM pipeline") - +In Steps 4.1 to 4.4, continue editing the same `app.py` file you created in the baseline pipeline section (`~/voice-sentiment-assistant/app.py`). ### Step 4.1 - Load the voice sentiment classification model This step initializes the trained and quantized ONNX model, then loads the matching preprocessing components used during training. You will reuse these objects every time you run sentiment inference on new audio. +Add these imports and model-loading definitions to `app.py`: +- add the imports with your other imports at the top of the file +- add `MODEL_DIR`, `ONNX_PATH`, `session`, `feature_extractor`, and `id2label` near `LOCAL_LLM_URL` and your other global setup values + ```python import onnxruntime as ort import numpy as np @@ -80,7 +84,10 @@ def handle_audio(audio_path): json={ "model": "local-model", "messages": [ - {"role": "system", "content": "Acknowledge the user sentiment. Respond with a joke on the sentiment."}, + { + "role": "system", + "content": "Respond helpfully and acknowledge the user's emotional tone." + }, {"role": "user", "content": prompt}, ], }, @@ -99,8 +106,6 @@ def handle_audio(audio_path): In this step, update the interface so it displays three outputs: the transcript, the predicted sentiment, and the final LLM response. For consistency with the earlier page, connect the microphone input with `mic.change(...)`. ```python -import gradio as gr - with gr.Blocks() as demo: mic = gr.Audio(sources="microphone", type="filepath") diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/_index.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/_index.md index 22cbb71d11..c5eb522085 100644 --- a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/_index.md +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/_index.md @@ -21,8 +21,6 @@ learning_objectives: prerequisites: - Python 3.9 or later for programming. - A working microphone for voice input. - - "ffmpeg installed on your system (required by Whisper for audio decoding)." - - "Build tools to compile llama.cpp (git, make, gcc/g++)." - Basic Python and command-line knowledge. author: Bhanu Arya diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/_next-steps.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/_next-steps.md new file mode 100644 index 0000000000..c3db0de5a2 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 # Set to always be larger than the content in this path to be at the end of the navigation. +title: "Next Steps" # Always the same, html page title. +layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing. +---