Merge pull request #3177 from annietllnd/main

Technical review of Voice Sentiment with LLM

Author: pareenaverma

content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/1_overview.md

@@ -5,7 +5,7 @@ layout: learningpathall
 ---
 Voice-based LLM applications often rely primarily on transcribed text from speech input, such as in interactions with non-player characters in games or voice assistants. This approach can overlook vocal cues—like tone, pitch, and emotion—present in a speaker’s voice. As a result, responses may feel less natural and may not fully capture the user’s underlying intent.
 
-To address this, voice-based sentiment classification analyzes audio input to determine the user’s emotional state, which is then incorporated into the LLM prompt to enable more context-aware responses. In this Learning Path, we will build a sentiment-aware voice assistant that runs entirely on-device. The application records audio, performs transcription—converting speech into written text—using Whisper, classifies sentiment directly from the voice signal, and combines the transcript and voice-based sentiment to guide responses from a local LLM running with llama.cpp.
+To address this, voice-based sentiment classification analyzes audio input to determine the user’s emotional state, which is then incorporated into the LLM prompt to enable more context-aware responses. In this Learning Path, you will build a sentiment-aware voice assistant that runs entirely on-device. The application records audio, performs transcription—converting speech into written text—using Whisper, classifies sentiment directly from the voice signal, and combines the transcript and voice-based sentiment to guide responses from a local LLM running with llama.cpp.
 
 ![Voice sentiment classification pipeline#center](1_vsapipeline2.png "Voice sentiment classification pipeline")
 

content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/2_set-up-your-environment.md

@@ -10,43 +10,69 @@ Before building the voice assistant, create a project workspace and set up an is
 
 These instructions support Ubuntu, macOS, and Windows, with Python 3.9 or later and a working microphone.
 
-Check your Python version before continuing:
+Install required system tools first. `ffmpeg` is required by Whisper for audio decoding.
+
+{{< tabpane code=true >}}
+  {{< tab header="Ubuntu/macOS" language="bash">}}
+# Ubuntu
+sudo apt update
+sudo apt install -y ffmpeg git cmake
+
+# macOS
+brew install ffmpeg git cmake
+  {{< /tab >}}
+  {{< tab header="Windows PowerShell" language="powershell">}}
+# Install via WinGet
+winget install -e --id Gyan.FFmpeg
+winget install -e --id Git.Git
+winget install -e --id Kitware.CMake
+  {{< /tab >}}
+{{< /tabpane >}}
 
-**Ubuntu or macOS**
+Check your Python version before continuing:
 
-```bash
+{{< tabpane code=true >}}
+  {{< tab header="Ubuntu/macOS" language="bash">}}
 python3 --version
-```
-
-**Windows PowerShell**
-
-```powershell
+  {{< /tab >}}
+  {{< tab header="Windows PowerShell" language="powershell">}}
 py -3 --version
-```
+  {{< /tab >}}
+{{< /tabpane >}}
 
 ## Set up the Python environment with UV
 
-Install `UV` first from PyPI using `pip`. `UV` is a fast Python package and environment manager that we will use throughout this Learning Path to create the project environment and install dependencies.
-
-**Ubuntu or macOS**
-
-```bash
+Install `UV` first so the `uv` command is available in your terminal. `UV` is a fast Python package and environment manager that you will use throughout this Learning Path to create the project environment and install dependencies.
+
+{{< tabpane code=true >}}
+  {{< tab header="Ubuntu/macOS" language="bash">}}
+curl -LsSf https://astral.sh/uv/install.sh | sh
+# Start a new shell, or source your shell rc file so `uv` is on PATH.
+uv --version
+  {{< /tab >}}
+  {{< tab header="Windows PowerShell" language="powershell">}}
+powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
+# Open a new PowerShell window so `uv` is on PATH.
+uv --version
+  {{< /tab >}}
+{{< /tabpane >}}
+
+Create and activate the project virtual environment:
+
+{{< tabpane code=true >}}
+  {{< tab header="Ubuntu/macOS" language="bash">}}
 mkdir -p ~/voice-sentiment-assistant
 cd ~/voice-sentiment-assistant
-python3 -m pip install uv
 uv venv .venv
 source .venv/bin/activate
-```
-
-**Windows PowerShell**
-
-```powershell
+  {{< /tab >}}
+  {{< tab header="Windows PowerShell" language="powershell">}}
 mkdir $HOME\voice-sentiment-assistant -Force
 cd $HOME\voice-sentiment-assistant
-py -3 -m pip install uv
 uv venv .venv
 .\.venv\Scripts\Activate.ps1
-```
+  {{< /tab >}}
+{{< /tabpane >}}
 
 Keep this virtual environment activated while you complete the rest of the Learning Path.
 
@@ -63,6 +89,7 @@ numpy
 librosa
 scikit-learn
 onnx
+onnxscript
 onnxruntime
 ```
 
@@ -78,41 +105,50 @@ This installs the libraries needed for the Gradio interface, Whisper transcripti
 
 Next, clone the [llama.cpp GitHub repository](https://github.com/ggml-org/llama.cpp), build the local inference server, and start it. This server exposes an OpenAI-compatible API that the Python application will call later in the Learning Path.
 
-**Ubuntu or macOS**
+{{% notice Pre-built llama.cpp %}}
+If you prefer not to build from source, you can use pre-built binaries from [llama.cpp releases](https://github.com/ggml-org/llama.cpp/releases). Download the package for your platform, extract it, and use the `llama-server` executable from that package in the run commands later in this section.
+{{% /notice %}}
+
 
-```bash
+{{< tabpane code=true >}}
+  {{< tab header="Ubuntu/macOS" language="bash">}}
 git clone https://github.com/ggml-org/llama.cpp
 cd llama.cpp
 cmake -B build
 cmake --build build --config Release
-```
-
-**Windows PowerShell**
-
-```powershell
+  {{< /tab >}}
+  {{< tab header="Windows PowerShell" language="powershell">}}
 git clone https://github.com/ggml-org/llama.cpp
 cd llama.cpp
 cmake -B build
 cmake --build build --config Release
+  {{< /tab >}}
+{{< /tabpane >}}
+
+When the build completes, the `llama-server` executable should be available in the build output directory. 
+
+```
+ls ./build/bin/llama-server
 ```
 
-When the build completes, the `llama-server` executable should be available in the build output directory. This Learning Path uses a quantized [Gemma 3 1B instruction-tuned model](https://huggingface.co/google/gemma-3-1b-it) served locally through `llama.cpp`.
+{{% notice Note %}}
+The file verification commands in this Learning Path uses syntax for Ubuntu and macOS. If you're on Windows, adjust the commands to use PowerShell equivalents like `Test-Path .\build\bin\llama-server.exe` or `dir .\build\bin\`. The file location remains the same across all platforms.
+{{% /notice %}}
 
-The first time you run this command, `llama.cpp` will download the model from Hugging Face. This can take several minutes depending on your network connection.
+This Learning Path uses a quantized [Gemma 3 1B instruction-tuned model](https://huggingface.co/google/gemma-3-1b-it) served locally through `llama.cpp`.
 
-**Ubuntu or macOS**
+The first time you run this command, `llama.cpp` will download the model from Hugging Face. This can take several minutes depending on your network connection.
 
 Run the following command from the `llama.cpp` directory:
 
-```bash
+{{< tabpane code=true >}}
+  {{< tab header="Ubuntu/macOS" language="bash">}}
 ./build/bin/llama-server -hf ggml-org/gemma-3-1b-it-GGUF
-```
-
-**Windows PowerShell**
-
-```powershell
+  {{< /tab >}}
+  {{< tab header="Windows PowerShell" language="powershell">}}
 .\build\bin\Release\llama-server.exe -hf ggml-org/gemma-3-1b-it-GGUF
-```
+  {{< /tab >}}
+{{< /tabpane >}}
 
 Leave this terminal running while you test the application in later steps. The server listens on a local OpenAI-compatible endpoint that your app will call to generate responses.
 

content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/3_build-baseline-voice-to-llm-pipeline.md

@@ -24,7 +24,14 @@ Gradio is a Python library for building simple browser-based interfaces. Here, y
 
 This is a good first step because it lets you confirm that microphone capture works before you add transcription and model inference.
 
-Create a file called `app.py`:
+Using a new terminal, navigate to the project directory you created in the last step and make sure the virtual environment is active:
+
+```
+cd $HOME/voice-sentiment-assistant
+source .venv/bin/activate
+```
+
+Create a file called `app.py` with the following contents:
 
 ```python
 import gradio as gr

content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/4_train-voice-sentiment-model.md

@@ -11,12 +11,22 @@ Before you begin, make sure you have completed the environment setup from the pr
 
 This example uses a simple per-sample training loop for clarity. On CPU, training can take 10 to 30 minutes or more depending on your machine. Apple Silicon `mps` and CUDA-enabled GPUs are often faster.
 
+All code snippets in Steps 2.1 to 2.5 are meant to be added to a single Python file: `train_sentiment_model.py` in your project root (`$HOME/voice-sentiment-assistant`).
+
+Make sure you are located in the project directory:
+
+```bash
+cd $HOME/voice-sentiment-assistant
+```
+
 ### Step 2.1 - Prepare the dataset
 
-This step downloads the RAVDESS speech dataset, extracts it locally, and builds a dataframe that maps each audio file to a sentiment label. RAVDESS encodes emotion labels in the filename format, which we use to select the classes for this demo.
+This step downloads the RAVDESS speech dataset, extracts it locally, and builds a dataframe that maps each audio file to a sentiment label. RAVDESS encodes emotion labels in the filename format, which you use to select the classes for this demo.
 
 In practice, this turns a folder of audio files into a structured table that your training code can work with. Each row is one example, and each column stores details such as the file path, actor ID, and sentiment label.
 
+Add this to a new file called `train_sentiment_model.py`:
+
 ```python
 import os
 import urllib.request
@@ -25,25 +35,24 @@ import pandas as pd
 
 # Download and unpack the dataset into the local project directory.
 RAVDESS_URL = "https://zenodo.org/records/1188976/files/Audio_Speech_Actors_01-24.zip?download=1"
-output_dir = "data/ravdess"
-os.makedirs(output_dir, exist_ok=True)
+OUTPUT_DIR = "data/ravdess"
+ZIP_PATH = f"{OUTPUT_DIR}/Audio_Speech_Actors_01-24.zip"
+EXTRACT_DIR = f"{OUTPUT_DIR}/Audio_Speech_Actors_01-24"
 
-zip_path = f"{output_dir}/Audio_Speech_Actors_01-24.zip"
-extract_dir = f"{output_dir}/Audio_Speech_Actors_01-24"
+os.makedirs(OUTPUT_DIR, exist_ok=True)
 
-if not os.path.exists(zip_path):
+if not os.path.exists(ZIP_PATH):
     print("Downloading dataset...")
-    urllib.request.urlretrieve(RAVDESS_URL, zip_path)
+    urllib.request.urlretrieve(RAVDESS_URL, ZIP_PATH)
 
-if not os.path.exists(extract_dir):
-    zipfile.ZipFile(zip_path).extractall(extract_dir)
+if not os.path.exists(EXTRACT_DIR):
+    zipfile.ZipFile(ZIP_PATH).extractall(EXTRACT_DIR)
 
-DATASET_PATH = "data/ravdess/Audio_Speech_Actors_01-24"
 emotion_map = {1: "neutral", 3: "happy", 5: "angry"}
 data = []
 
-for actor in os.listdir(DATASET_PATH):
-    actor_path = os.path.join(DATASET_PATH, actor)
+for actor in os.listdir(EXTRACT_DIR):
+    actor_path = os.path.join(EXTRACT_DIR, actor)
     if not os.path.isdir(actor_path):
         continue
 
@@ -73,6 +82,8 @@ Next, split the dataset by speaker so the same speaker does not appear in both t
 This matters because a speech model can accidentally learn speaker-specific traits instead of sentiment if the same speakers appear in both splits. Separating speakers gives you a better measure of how well the model generalizes.
 The training split is the data the model learns from. The validation split is held back and used only to test how well the model performs on unseen examples.
 
+Append this to the same `train_sentiment_model.py` file:
+
 ```python
 import random
 
@@ -97,6 +108,8 @@ print(f"Validation samples: {len(val_df)}")
 This step loads the HuBERT feature extractor and initializes a sequence classification model. The pre-trained HuBERT base model provides a strong starting point for transfer learning on speech sentiment classification.
 The feature extractor prepares raw audio in the format expected by the model. The HuBERT model then uses those inputs to predict one of the target sentiment classes.
 
+Append this below the previous code in `train_sentiment_model.py`:
+
 ```python
 from transformers import AutoFeatureExtractor, HubertForSequenceClassification
 
@@ -122,6 +135,8 @@ During training, the model compares its predictions with the correct labels and
 
 For better performance in a production training pipeline, you would typically use batching and a PyTorch `DataLoader`.
 
+Append this below the previous code in `train_sentiment_model.py`:
+
 ```python
 import librosa
 import torch
@@ -200,23 +215,27 @@ At the end of this step, you will have a trained model and a simple validation a
 Save both the trained model and the feature extractor so they can be reused later for inference and ONNX export. Keeping them together helps avoid preprocessing mismatches.
 This is an important final step because the model weights alone are not enough. You also need the matching feature extractor so inference uses the same preprocessing as training.
 
-```python
-import os
+Append this at the end of `train_sentiment_model.py`:
 
+```python
 SAVE_DIR = "models/hubert_vsa_ravdess"
 os.makedirs(SAVE_DIR, exist_ok=True)
 
 # Save both the trained model and the preprocessing configuration.
 model.save_pretrained(SAVE_DIR)
 feature_extractor.save_pretrained(SAVE_DIR)
+
+print("Training complete. Model saved successfully.")
+print(f"Saved model to {SAVE_DIR}")
 ```
 
 After this step, the trained model should be saved in `models/hubert_vsa_ravdess`.
 
 ## Full training script
 
-The following script combines the previous steps into one runnable example. Save it as `train_sentiment_model.py` and run it from the root of your project.
-Use this script if you want to run the full workflow without copying each block separately.
+The following script combines the previous steps into one runnable example.
+If you already built `train_sentiment_model.py` using Steps 2.1 to 2.5, you can skip this section.
+If not, copy this full script into `train_sentiment_model.py` and run it from the root of your project.
 
 ```python
 import os