diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/1_overview.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/1_overview.md new file mode 100644 index 0000000000..ff033f06c3 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/1_overview.md @@ -0,0 +1,12 @@ +--- +title: Overview +weight: 2 +layout: learningpathall +--- +Voice-based LLM applications often rely primarily on transcribed text from speech input, such as in interactions with non-player characters in games or voice assistants. This approach can overlook vocal cues—like tone, pitch, and emotion—present in a speaker’s voice. As a result, responses may feel less natural and may not fully capture the user’s underlying intent. + +To address this, voice-based sentiment classification analyzes audio input to determine the user’s emotional state, which is then incorporated into the LLM prompt to enable more context-aware responses. In this Learning Path, we will build a sentiment-aware voice assistant that runs entirely on-device. The application records audio, performs transcription—converting speech into written text—using Whisper, classifies sentiment directly from the voice signal, and combines the transcript and voice-based sentiment to guide responses from a local LLM running with llama.cpp. + +![Voice sentiment classification pipeline#center](1_vsapipeline2.png "Voice sentiment classification pipeline") + +You will start by building a baseline voice-to-LLM pipeline—capturing audio, transcribing it into text, and using it to generate responses with an LLM. You will then extend this pipeline with a voice-based sentiment classification model. This involves training the model, optimizing it for efficient on-device inference, and integrating it into a unified application. diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/1_vsapipeline2.png b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/1_vsapipeline2.png new file mode 100644 index 0000000000..413bb43434 Binary files /dev/null and b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/1_vsapipeline2.png differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/2_set-up-your-environment.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/2_set-up-your-environment.md new file mode 100644 index 0000000000..e59320dca9 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/2_set-up-your-environment.md @@ -0,0 +1,119 @@ +--- +title: Set up your environment +weight: 3 + +### FIXED, DO NOT MODIFY +layout: learningpathall +--- + +Before building the voice assistant, create a project workspace and set up an isolated `UV` environment. This keeps project dependencies separate from your system installation and makes it easier to reproduce the steps in the rest of the Learning Path. + +These instructions support Ubuntu, macOS, and Windows, with Python 3.9 or later and a working microphone. + +Check your Python version before continuing: + +**Ubuntu or macOS** + +```bash +python3 --version +``` + +**Windows PowerShell** + +```powershell +py -3 --version +``` + +## Set up the Python environment with UV + +Install `UV` first from PyPI using `pip`. `UV` is a fast Python package and environment manager that we will use throughout this Learning Path to create the project environment and install dependencies. + +**Ubuntu or macOS** + +```bash +mkdir -p ~/voice-sentiment-assistant +cd ~/voice-sentiment-assistant +python3 -m pip install uv +uv venv .venv +source .venv/bin/activate +``` + +**Windows PowerShell** + +```powershell +mkdir $HOME\voice-sentiment-assistant -Force +cd $HOME\voice-sentiment-assistant +py -3 -m pip install uv +uv venv .venv +.\.venv\Scripts\Activate.ps1 +``` + +Keep this virtual environment activated while you complete the rest of the Learning Path. + +Create a `requirements.txt` file for the packages used across the rest of the Learning Path: + +```txt +gradio +openai-whisper +requests +torch +transformers +pandas +numpy +librosa +scikit-learn +onnx +onnxruntime +``` + +Install the dependencies into your active `UV` virtual environment: + +```console +uv pip install -r requirements.txt +``` + +This installs the libraries needed for the Gradio interface, Whisper transcription, model training, and ONNX Runtime inference. Some packages in this list are used later in the Learning Path when you optimize and export the sentiment model. + +## Download, build, and run llama.cpp + +Next, clone the [llama.cpp GitHub repository](https://github.com/ggml-org/llama.cpp), build the local inference server, and start it. This server exposes an OpenAI-compatible API that the Python application will call later in the Learning Path. + +**Ubuntu or macOS** + +```bash +git clone https://github.com/ggml-org/llama.cpp +cd llama.cpp +cmake -B build +cmake --build build --config Release +``` + +**Windows PowerShell** + +```powershell +git clone https://github.com/ggml-org/llama.cpp +cd llama.cpp +cmake -B build +cmake --build build --config Release +``` + +When the build completes, the `llama-server` executable should be available in the build output directory. This Learning Path uses a quantized [Gemma 3 1B instruction-tuned model](https://huggingface.co/google/gemma-3-1b-it) served locally through `llama.cpp`. + +The first time you run this command, `llama.cpp` will download the model from Hugging Face. This can take several minutes depending on your network connection. + +**Ubuntu or macOS** + +Run the following command from the `llama.cpp` directory: + +```bash +./build/bin/llama-server -hf ggml-org/gemma-3-1b-it-GGUF +``` + +**Windows PowerShell** + +```powershell +.\build\bin\Release\llama-server.exe -hf ggml-org/gemma-3-1b-it-GGUF +``` + +Leave this terminal running while you test the application in later steps. The server listens on a local OpenAI-compatible endpoint that your app will call to generate responses. + +At this point, your development environment is ready. You have installed the required audio and build tools, created a `UV` environment, installed the Python dependencies, and started a local `llama.cpp` server. In the next section, you will use this setup to build the baseline voice-to-LLM pipeline by creating a simple Gradio interface, transcribing microphone input with Whisper, and sending the transcript to the local LLM. diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/3_build-baseline-voice-to-llm-pipeline.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/3_build-baseline-voice-to-llm-pipeline.md new file mode 100644 index 0000000000..051cc919a2 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/3_build-baseline-voice-to-llm-pipeline.md @@ -0,0 +1,187 @@ +--- +title: Build the voice-to-LLM pipeline +weight: 4 +layout: learningpathall +--- + +In this section, you will build an end-to-end pipeline that: + +1. Records audio from your microphone +2. Transcribes it to text using Whisper +3. Sends the text to a locally hosted LLM +4. Displays the model's response + +This forms the foundation of your voice assistant. + +![Baseline voice-to-LLM pipeline#center](3_vsapipeline1.png "Baseline voice-to-LLM pipeline") + +Before you begin, make sure you have completed the environment setup in the previous section and that your `llama-server` is still running. + +### Step 1.1 - Create a basic Gradio UI + +Start by creating a simple web interface that captures microphone input. +Gradio is a Python library for building simple browser-based interfaces. Here, you use it to create a small front end that records audio from your microphone. + +This is a good first step because it lets you confirm that microphone capture works before you add transcription and model inference. + +Create a file called `app.py`: + +```python +import gradio as gr + +with gr.Blocks() as demo: + mic = gr.Audio(sources="microphone", type="filepath") + +demo.launch() +``` + +Run the app: + +```bash +python app.py +``` + +Open your browser at: + +`http://127.0.0.1:7860` + +You should now see a simple interface that allows you to record audio. +At this stage, the app only captures audio. It does not yet transcribe speech or send anything to the LLM. + +### Step 1.2 - Add speech-to-text with Whisper + +Next, add transcription using the Whisper model. +Whisper is a speech-to-text model. It takes audio as input and returns a text transcript. In this pipeline, it converts spoken input into text before anything is sent to the LLM. + +Update `app.py` with the following code: + +```python +import whisper + +# Load a small Whisper model for local transcription +model = whisper.load_model("base") + +def transcribe(audio): + return model.transcribe(audio)["text"] +``` + +The first time you run this, Whisper will download the model, which may take a few minutes. + +At this stage, your app can convert recorded audio into text. +The output of this step is a text transcript that represents what the user said. + +### Step 1.3 - Connect to the local LLM + +Define the OpenAI-compatible endpoint exposed by `llama-server`. +An endpoint is the URL your program uses to talk to another service. In this case, `llama-server` exposes a local API on your machine, and your app sends the transcript there to get a response. + +Because the server is OpenAI-compatible, the request format looks like a standard chat completions API. + +Update `app.py` with the following import and endpoint definition: + +```python +import requests + +LOCAL_LLM_URL = "http://127.0.0.1:8080/v1/chat/completions" +``` + +Make sure your `llama-server` from the previous section is running before continuing. +Without the local server running, the next step will not be able to generate an answer. + +### Step 1.4 - Build the full pipeline + +Now combine transcription and LLM interaction into a single function. +This function becomes the core of the application. Audio goes in, text is extracted, that text is sent to the model, and the response comes back out. + +Keeping the logic in one function makes it easier to connect the pipeline to the user interface in the next step. + +Update `app.py` by adding the following function: + +```python +def handle_audio(audio): + # Step 1: Transcribe audio + text = transcribe(audio) + + # Step 2: Send transcript to local LLM + response = requests.post( + LOCAL_LLM_URL, + json={ + "model": "local-model", + "messages": [{"role": "user", "content": text}], + }, + ) + + if response.status_code != 200: + return text, "Error: LLM request failed" + + data = response.json() + answer = data["choices"][0]["message"]["content"] + + return text, answer +``` + +### Step 1.5 - Connect the UI to the pipeline + +Update your UI so that recorded audio triggers the full pipeline and displays results. +This is the final integration step. You now connect the interface, transcription, and model request so the app behaves like a real voice assistant. + +When the user records audio, Gradio calls your pipeline function. The app then shows both the transcript and the assistant response in the browser. + +Update `app.py` so it contains the following complete version: + +```python +import gradio as gr +import whisper +import requests + +model = whisper.load_model("base") + +LOCAL_LLM_URL = "http://127.0.0.1:8080/v1/chat/completions" + +def transcribe(audio): + return model.transcribe(audio)["text"] + +def handle_audio(audio): + text = transcribe(audio) + + response = requests.post( + LOCAL_LLM_URL, + json={ + "model": "local-model", + "messages": [{"role": "user", "content": text}], + }, + ) + + if response.status_code != 200: + return text, "Error: LLM request failed" + + data = response.json() + answer = data["choices"][0]["message"]["content"] + + return text, answer + +with gr.Blocks() as demo: + mic = gr.Audio(sources="microphone", type="filepath") + transcript = gr.Textbox(label="Transcript") + response = gr.Textbox(label="LLM Response") + + mic.change(fn=handle_audio, inputs=mic, outputs=[transcript, response]) + +demo.launch() +``` + +## What you should see + +After recording audio in the browser: + +- Your speech is transcribed into text +- The transcript is sent to the local LLM +- The LLM response is displayed in the interface + +## Troubleshooting + +- No response from LLM: ensure `llama-server` is still running. +- Whisper is slow on first run: this is expected due to model download and initialization. +- Microphone not working: check browser permissions for microphone access. + +At this point, you have a working voice-to-LLM pipeline. In the next section, you will extend this pipeline by adding a voice sentiment classification model. diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/3_vsapipeline1.png b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/3_vsapipeline1.png new file mode 100644 index 0000000000..a3e38d754b Binary files /dev/null and b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/3_vsapipeline1.png differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/4_modelconversionandcompression.png b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/4_modelconversionandcompression.png new file mode 100644 index 0000000000..0b9cfc7eb7 Binary files /dev/null and b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/4_modelconversionandcompression.png differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/4_train-voice-sentiment-model.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/4_train-voice-sentiment-model.md new file mode 100644 index 0000000000..e6d0251c48 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/4_train-voice-sentiment-model.md @@ -0,0 +1,376 @@ +--- +title: Train the voice sentiment classification model +weight: 5 +layout: learningpathall +--- +In this section, you will train a model to classify voice sentiment directly from audio. The raw waveform is converted into audio-based features, then passed through a pre-trained HuBERT model with a classification head. + +For this Learning Path, you train the model on the RAVDESS dataset using three sentiment classes: `neutral`, `happy`, and `angry`. This keeps the demo manageable while still showing a realistic transfer-learning workflow. You can extend the same approach to more sentiment classes or other datasets. + +Before you begin, make sure you have completed the environment setup from the previous section and installed the required Python dependencies. + +This example uses a simple per-sample training loop for clarity. On CPU, training can take 10 to 30 minutes or more depending on your machine. Apple Silicon `mps` and CUDA-enabled GPUs are often faster. + +### Step 2.1 - Prepare the dataset + +This step downloads the RAVDESS speech dataset, extracts it locally, and builds a dataframe that maps each audio file to a sentiment label. RAVDESS encodes emotion labels in the filename format, which we use to select the classes for this demo. + +In practice, this turns a folder of audio files into a structured table that your training code can work with. Each row is one example, and each column stores details such as the file path, actor ID, and sentiment label. + +```python +import os +import urllib.request +import zipfile +import pandas as pd + +# Download and unpack the dataset into the local project directory. +RAVDESS_URL = "https://zenodo.org/records/1188976/files/Audio_Speech_Actors_01-24.zip?download=1" +output_dir = "data/ravdess" +os.makedirs(output_dir, exist_ok=True) + +zip_path = f"{output_dir}/Audio_Speech_Actors_01-24.zip" +extract_dir = f"{output_dir}/Audio_Speech_Actors_01-24" + +if not os.path.exists(zip_path): + print("Downloading dataset...") + urllib.request.urlretrieve(RAVDESS_URL, zip_path) + +if not os.path.exists(extract_dir): + zipfile.ZipFile(zip_path).extractall(extract_dir) + +DATASET_PATH = "data/ravdess/Audio_Speech_Actors_01-24" +emotion_map = {1: "neutral", 3: "happy", 5: "angry"} +data = [] + +for actor in os.listdir(DATASET_PATH): + actor_path = os.path.join(DATASET_PATH, actor) + if not os.path.isdir(actor_path): + continue + + for f in os.listdir(actor_path): + if f.endswith(".wav"): + # RAVDESS encodes emotion ID in the filename. + emotion = int(f.split("-")[2]) + + if emotion in emotion_map: + data.append({ + "emotion": emotion_map[emotion], + "actor": actor, + # Store absolute sample path for later loading. + "path": os.path.join(actor_path, f) + }) + +df = pd.DataFrame(data) +print(df.head()) +print(df["emotion"].value_counts()) +``` + +At the end of this step, you should have a dataframe with the selected audio samples and their sentiment labels. + +### Step 2.2 - Create train and validation splits + +Next, split the dataset by speaker so the same speaker does not appear in both training and validation sets. This helps avoid data leakage and gives you a more realistic evaluation. +This matters because a speech model can accidentally learn speaker-specific traits instead of sentiment if the same speakers appear in both splits. Separating speakers gives you a better measure of how well the model generalizes. +The training split is the data the model learns from. The validation split is held back and used only to test how well the model performs on unseen examples. + +```python +import random + +# Split by speaker so the same actor is not in both training and validation. +actors = sorted(df["actor"].unique()) +random.seed(42) +random.shuffle(actors) + +val_count = max(1, int(0.2 * len(actors))) +val_actors = set(actors[:val_count]) +train_actors = set(actors[val_count:]) + +train_df = df[df["actor"].isin(train_actors)].reset_index(drop=True) +val_df = df[df["actor"].isin(val_actors)].reset_index(drop=True) + +print(f"Training samples: {len(train_df)}") +print(f"Validation samples: {len(val_df)}") +``` + +### Step 2.3 - Load the feature extractor and model + +This step loads the HuBERT feature extractor and initializes a sequence classification model. The pre-trained HuBERT base model provides a strong starting point for transfer learning on speech sentiment classification. +The feature extractor prepares raw audio in the format expected by the model. The HuBERT model then uses those inputs to predict one of the target sentiment classes. + +```python +from transformers import AutoFeatureExtractor, HubertForSequenceClassification + +MODEL_NAME = "facebook/hubert-base-ls960" +# The label list defines the numeric class order used during training. +labels = ["angry", "happy", "neutral"] + +# Use the HuBERT feature extractor that matches the pretrained checkpoint. +feature_extractor = AutoFeatureExtractor.from_pretrained(MODEL_NAME) + +model = HubertForSequenceClassification.from_pretrained( + MODEL_NAME, + num_labels=len(labels), + id2label={i: label for i, label in enumerate(labels)}, + label2id={label: i for i, label in enumerate(labels)}, +) +``` + +### Step 2.4 - Train and evaluate the model + +This step performs the core training pass: load audio, extract features, compute loss, and update the model weights. For clarity, this example uses a simple per-sample training loop rather than batching with a `DataLoader`. +During training, the model compares its predictions with the correct labels and updates its weights to reduce the error. After each epoch, the validation loop checks how well the model performs on held-out speakers. + +For better performance in a production training pipeline, you would typically use batching and a PyTorch `DataLoader`. + +```python +import librosa +import torch + +def load_audio(path): + # Resample audio to 16 kHz, the sampling rate expected by HuBERT. + return librosa.load(path, sr=16000)[0] + +if torch.cuda.is_available(): + device = torch.device("cuda") +elif torch.backends.mps.is_available(): + device = torch.device("mps") +else: + device = torch.device("cpu") + +model.to(device) + +optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5) +EPOCHS = 4 + +for epoch in range(EPOCHS): + print(f"Epoch {epoch + 1}/{EPOCHS}") + # Switch back to training mode at the start of each epoch. + model.train() + + for i, (_, row) in enumerate(train_df.iterrows(), start=1): + audio = load_audio(row["path"]) + + # Convert raw waveform into model input tensors. + inputs = feature_extractor( + audio, + sampling_rate=16000, + return_tensors="pt" + ) + inputs = {k: v.to(device) for k, v in inputs.items()} + + # Convert the string label into a class index. + label = torch.tensor([labels.index(row["emotion"])]).to(device) + + loss = model(**inputs, labels=label).loss + + loss.backward() + optimizer.step() + optimizer.zero_grad() + + if i % 50 == 0: + print(f"Processed {i} training samples") + + model.eval() + correct = 0 + + with torch.no_grad(): + for _, row in val_df.iterrows(): + audio = load_audio(row["path"]) + inputs = feature_extractor( + audio, + sampling_rate=16000, + return_tensors="pt" + ) + inputs = {k: v.to(device) for k, v in inputs.items()} + + logits = model(**inputs).logits + pred = logits.argmax(dim=-1).item() + + if pred == labels.index(row["emotion"]): + correct += 1 + + accuracy = correct / max(1, len(val_df)) + print(f"Validation accuracy: {accuracy:.2f}") +``` + +At the end of this step, you will have a trained model and a simple validation accuracy printed after each epoch. You should typically see validation accuracy improve over time, although the exact values will vary depending on randomness and hardware. + +### Step 2.5 - Save the model + +Save both the trained model and the feature extractor so they can be reused later for inference and ONNX export. Keeping them together helps avoid preprocessing mismatches. +This is an important final step because the model weights alone are not enough. You also need the matching feature extractor so inference uses the same preprocessing as training. + +```python +import os + +SAVE_DIR = "models/hubert_vsa_ravdess" +os.makedirs(SAVE_DIR, exist_ok=True) + +# Save both the trained model and the preprocessing configuration. +model.save_pretrained(SAVE_DIR) +feature_extractor.save_pretrained(SAVE_DIR) +``` + +After this step, the trained model should be saved in `models/hubert_vsa_ravdess`. + +## Full training script + +The following script combines the previous steps into one runnable example. Save it as `train_sentiment_model.py` and run it from the root of your project. +Use this script if you want to run the full workflow without copying each block separately. + +```python +import os +import random +import urllib.request +import zipfile + +import librosa +import pandas as pd +import torch +from transformers import AutoFeatureExtractor, HubertForSequenceClassification + +# Configuration for dataset download, training, and model export. +RAVDESS_URL = "https://zenodo.org/records/1188976/files/Audio_Speech_Actors_01-24.zip?download=1" +OUTPUT_DIR = "data/ravdess" +ZIP_PATH = f"{OUTPUT_DIR}/Audio_Speech_Actors_01-24.zip" +EXTRACT_DIR = f"{OUTPUT_DIR}/Audio_Speech_Actors_01-24" +MODEL_NAME = "facebook/hubert-base-ls960" +SAVE_DIR = "models/hubert_vsa_ravdess" +EPOCHS = 4 + +emotion_map = {1: "neutral", 3: "happy", 5: "angry"} +# The label list defines the numeric class order used during training. +labels = ["angry", "happy", "neutral"] + +os.makedirs(OUTPUT_DIR, exist_ok=True) + +if not os.path.exists(ZIP_PATH): + print("Downloading dataset...") + urllib.request.urlretrieve(RAVDESS_URL, ZIP_PATH) + +if not os.path.exists(EXTRACT_DIR): + zipfile.ZipFile(ZIP_PATH).extractall(EXTRACT_DIR) + +data = [] + +for actor in os.listdir(EXTRACT_DIR): + actor_path = os.path.join(EXTRACT_DIR, actor) + if not os.path.isdir(actor_path): + continue + + for f in os.listdir(actor_path): + if f.endswith(".wav"): + # RAVDESS stores the emotion label in the filename tokens. + emotion = int(f.split("-")[2]) + if emotion in emotion_map: + data.append( + { + "emotion": emotion_map[emotion], + "actor": actor, + "path": os.path.join(actor_path, f), + } + ) + +df = pd.DataFrame(data) +print(df.head()) +print(df["emotion"].value_counts()) + +actors = sorted(df["actor"].unique()) +random.seed(42) +random.shuffle(actors) + +val_count = max(1, int(0.2 * len(actors))) +val_actors = set(actors[:val_count]) +train_actors = set(actors[val_count:]) + +train_df = df[df["actor"].isin(train_actors)].reset_index(drop=True) +val_df = df[df["actor"].isin(val_actors)].reset_index(drop=True) + +print(f"Training samples: {len(train_df)}") +print(f"Validation samples: {len(val_df)}") + +feature_extractor = AutoFeatureExtractor.from_pretrained(MODEL_NAME) +model = HubertForSequenceClassification.from_pretrained( + MODEL_NAME, + num_labels=len(labels), + id2label={i: label for i, label in enumerate(labels)}, + label2id={label: i for i, label in enumerate(labels)}, +) + +if torch.cuda.is_available(): + device = torch.device("cuda") +elif torch.backends.mps.is_available(): + device = torch.device("mps") +else: + device = torch.device("cpu") + +model.to(device) + +def load_audio(path): + # Load and resample audio to the input rate expected by HuBERT. + return librosa.load(path, sr=16000)[0] + +optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5) + +for epoch in range(EPOCHS): + print(f"Epoch {epoch + 1}/{EPOCHS}") + # Switch back to training mode at the start of each epoch. + model.train() + + for i, (_, row) in enumerate(train_df.iterrows(), start=1): + audio = load_audio(row["path"]) + # Convert raw audio into tensors the model can consume. + inputs = feature_extractor( + audio, + sampling_rate=16000, + return_tensors="pt", + ) + inputs = {k: v.to(device) for k, v in inputs.items()} + # Map the label name to its numeric class ID. + label = torch.tensor([labels.index(row["emotion"])]).to(device) + + loss = model(**inputs, labels=label).loss + loss.backward() + optimizer.step() + optimizer.zero_grad() + + if i % 50 == 0: + print(f"Processed {i} training samples") + + model.eval() + correct = 0 + + with torch.no_grad(): + for _, row in val_df.iterrows(): + audio = load_audio(row["path"]) + inputs = feature_extractor( + audio, + sampling_rate=16000, + return_tensors="pt", + ) + inputs = {k: v.to(device) for k, v in inputs.items()} + + logits = model(**inputs).logits + pred = logits.argmax(dim=-1).item() + + if pred == labels.index(row["emotion"]): + correct += 1 + + accuracy = correct / max(1, len(val_df)) + print(f"Validation accuracy: {accuracy:.2f}") + +os.makedirs(SAVE_DIR, exist_ok=True) +model.save_pretrained(SAVE_DIR) +feature_extractor.save_pretrained(SAVE_DIR) + +print("Training complete. Model saved successfully.") +print(f"Saved model to {SAVE_DIR}") +``` + +Run the training script: + +```bash +python train_sentiment_model.py +``` + +At this point, you have a trained voice sentiment classification model saved locally. In the next section, you will convert this model to ONNX format and compress it for more efficient inference. diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/5_compress-and-convert-the-model.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/5_compress-and-convert-the-model.md new file mode 100644 index 0000000000..eae611c416 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/5_compress-and-convert-the-model.md @@ -0,0 +1,176 @@ +--- +title: Convert and quantize the model +weight: 6 +layout: learningpathall +--- +In this section, you will convert the trained model to ONNX format and compress it for efficient on-device inference. First, export the trained PyTorch model to ONNX. Then apply post-training quantization to reduce model size and improve runtime efficiency. + +### Step 3.1 - Export to ONNX + +This step loads the trained HuBERT checkpoint and exports it to an ONNX graph. +The exported ONNX model becomes the base file used in the quantization step. + +ONNX export may take a few seconds. If it fails, ensure the trained model from the previous section exists in `models/hubert_vsa_ravdess`. + +Save the following workflow as `convert_and_quantize_model.py`, or reuse the relevant parts in your own conversion script: + +```python +import os, torch +from transformers import AutoFeatureExtractor, HubertForSequenceClassification + +MODEL_DIR = "models/hubert_vsa_ravdess" +ONNX_DIR = "models/hubert_vsa_ravdess_onnx" +# Create the output directory for ONNX files. +os.makedirs(ONNX_DIR, exist_ok=True) + +# Load trained model weights from the previous section. +model = HubertForSequenceClassification.from_pretrained(MODEL_DIR) +# Switch to evaluation mode for deterministic export behavior. +model.eval() + +feature_extractor = AutoFeatureExtractor.from_pretrained(MODEL_DIR) + +# Create a dummy 1-second input to define input tensor shapes. +dummy = feature_extractor( + [0.0] * 16000, + sampling_rate=16000, + return_tensors="pt", + return_attention_mask=True +) + +# Export model graph and IO signatures to ONNX. +torch.onnx.export( + model, + (dummy["input_values"], dummy["attention_mask"]), + os.path.join(ONNX_DIR, "hubert_vsa_ravdess.onnx"), + input_names=["input_values", "attention_mask"], + output_names=["logits"], + opset_version=18 +) +``` + +After this step, you should have `hubert_vsa_ravdess.onnx` in the ONNX output directory. + +For production use, consider exporting with dynamic axes to support variable-length audio inputs. + +You can verify the exported file with: + +```bash +ls models/hubert_vsa_ravdess_onnx +``` + +### Step 3.2 - Quantize model + +This step applies dynamic INT8 quantization to the ONNX model. The model weights are quantized to integer 8 bits ahead of time, while activations are quantized dynamically during inference. + +Quantization typically reduces model size by around 3x to 4x and often improves CPU inference speed, depending on the hardware and model. + +```python +from onnxruntime.quantization import quantize_dynamic, QuantType + +quantize_dynamic( + model_input="models/hubert_vsa_ravdess_onnx/hubert_vsa_ravdess.onnx", + model_output="models/hubert_vsa_ravdess_onnx/hubert_vsa_ravdess_int8.onnx", + weight_type=QuantType.QInt8 +) +``` + +After this step, you should have a smaller quantized model file for deployment. + +You can verify the quantized model file with: + +```bash +ls models/hubert_vsa_ravdess_onnx +``` + +To compare file sizes more easily, you can also run: + +```bash +ls -lh models/hubert_vsa_ravdess_onnx +``` + +### Step 3.3 - Run ONNX inference + +This step validates that the quantized model can run inference with ONNX Runtime. +It also checks label decoding so you can confirm the output end to end. + +You can add this inference check to `convert_and_quantize_model.py` after the quantization step, or run it separately in a short test script. + +```python +import onnxruntime as ort +import numpy as np +import librosa +from transformers import AutoFeatureExtractor, AutoConfig + +MODEL_DIR = "models/hubert_vsa_ravdess" +ONNX_PATH = "models/hubert_vsa_ravdess_onnx/hubert_vsa_ravdess_int8.onnx" +SAMPLE_PATH = "data/ravdess/Audio_Speech_Actors_01-24/Actor_01/03-01-01-01-01-01-01.wav" + +# Initialize ONNX Runtime session with the quantized graph. +session = ort.InferenceSession(ONNX_PATH, providers=["CPUExecutionProvider"]) + +fx = AutoFeatureExtractor.from_pretrained(MODEL_DIR) +id2label = AutoConfig.from_pretrained(MODEL_DIR).id2label + +# Load one sample at 16 kHz. +audio, _ = librosa.load(SAMPLE_PATH, sr=16000) + +# Convert waveform to ONNX input tensors. +inputs = fx( + audio, + sampling_rate=16000, + return_tensors="np", + return_attention_mask=True +) + +# Run inference and retrieve logits output. +logits = session.run(None, { + "input_values": inputs["input_values"], + "attention_mask": inputs["attention_mask"] +})[0] + +# Convert highest-scoring class index to label name. +pred = int(np.argmax(logits)) +print("Predicted:", id2label[pred]) +``` + +If this file does not exist on your system, replace `SAMPLE_PATH` with any `.wav` file from your extracted RAVDESS dataset. + +After this step, you have verified that quantized ONNX inference works with your trained model. + +You should see a predicted label such as: + +```text +Predicted: happy +``` + +This layout helps keep the training and deployment files separate and easy to manage. + +```text +models/ + hubert_vsa_ravdess/ + hubert_vsa_ravdess_onnx/ + hubert_vsa_ravdess.onnx + hubert_vsa_ravdess_int8.onnx +``` + +### Model compression results + +We can also compare the model metrics before and after quantization. + +![Model ONNX conversion and int-8 quantization results#center](4_modelconversionandcompression.png "Model ONNX conversion and int-8 quantization results") + +Typical results look like: + +```text +Original ONNX model: larger size, higher memory footprint +Quantized INT8 ONNX model: smaller size, lower memory footprint, faster CPU inference +``` + +## Troubleshooting + +- ONNX export fails: ensure the trained model exists in `models/hubert_vsa_ravdess`. +- Inference error about missing inputs: confirm that both `input_values` and `attention_mask` are passed to the ONNX session. +- Slow inference: make sure you are using `hubert_vsa_ravdess_int8.onnx` rather than the unquantized ONNX model. + +The model is now ready for integration into the voice pipeline in the next section. diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/6_integrate-voice-sentiment-classification-model.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/6_integrate-voice-sentiment-classification-model.md new file mode 100644 index 0000000000..3e56280ad9 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/6_integrate-voice-sentiment-classification-model.md @@ -0,0 +1,256 @@ +--- +title: Integrate the voice sentiment classification model +weight: 7 +layout: learningpathall +--- +In this section, you will integrate the voice sentiment classification model into the baseline voice-to-LLM pipeline and build a sentiment-aware voice assistant. You will update the prompt sent to the LLM so it includes both the transcript and the predicted sentiment, combining speech, emotion, and language understanding into a single application. + +End-to-end response time may take a few seconds because the application runs Whisper transcription, ONNX sentiment inference, and local LLM generation for each interaction. + +![Integrated voice-to-LLM pipeline#center](6_vsapipeline4.png "Integrated voice-to-LLM pipeline") + + + +### Step 4.1 - Load the voice sentiment classification model + +This step initializes the trained and quantized ONNX model, then loads the matching preprocessing components used during training. You will reuse these objects every time you run sentiment inference on new audio. + +```python +import onnxruntime as ort +import numpy as np +import librosa +from transformers import AutoFeatureExtractor, AutoConfig + +MODEL_DIR = "models/hubert_vsa_ravdess" +ONNX_PATH = "models/hubert_vsa_ravdess_onnx/hubert_vsa_ravdess_int8.onnx" + +# Run the quantized model with the CPU execution provider. +session = ort.InferenceSession( + ONNX_PATH, + providers=["CPUExecutionProvider"] +) +feature_extractor = AutoFeatureExtractor.from_pretrained(MODEL_DIR) +id2label = AutoConfig.from_pretrained(MODEL_DIR).id2label +``` + +### Step 4.2 - Create sentiment prediction function + +Next, create a helper function that takes an audio file path, runs the sentiment model, and returns the predicted label. This adds the emotion-aware part of the assistant. + +```python +def predict_sentiment(audio_path): + audio, _ = librosa.load(audio_path, sr=16000, mono=True) + + inputs = feature_extractor( + audio, + sampling_rate=16000, + return_tensors="np", + return_attention_mask=True + ) + + logits = session.run(None, { + "input_values": inputs["input_values"], + "attention_mask": inputs["attention_mask"] + })[0] + + pred = int(np.argmax(logits)) + return id2label[pred] +``` + +### Step 4.3 - Update pipeline to include sentiment + +Now that the voice sentiment classification function is ready, combine transcription and sentiment prediction before sending the request to the local LLM. Reuse the `model`, `requests`, and `LOCAL_LLM_URL` definitions from the baseline voice-to-LLM pipeline section. + +```python +def handle_audio(audio_path): + if not audio_path: + return "", "", "" + + # 1. Transcribe + text = model.transcribe(audio_path)["text"] + + # 2. Predict sentiment + sentiment = predict_sentiment(audio_path) + + # 3. Send to LLM with sentiment context + prompt = f"The user said: '{text}'. Their tone sounds {sentiment}. Respond appropriately." + + response = requests.post( + LOCAL_LLM_URL, + json={ + "model": "local-model", + "messages": [ + {"role": "system", "content": "Acknowledge the user sentiment. Respond with a joke on the sentiment."}, + {"role": "user", "content": prompt}, + ], + }, + ) + + if response.status_code != 200: + return text, sentiment, "Error: LLM request failed" + + answer = response.json()["choices"][0]["message"]["content"] + + return text, sentiment, answer +``` + +### Step 4.4 - Update User Interface (UI) + +In this step, update the interface so it displays three outputs: the transcript, the predicted sentiment, and the final LLM response. For consistency with the earlier page, connect the microphone input with `mic.change(...)`. + +```python +import gradio as gr + +with gr.Blocks() as demo: + mic = gr.Audio(sources="microphone", type="filepath") + + transcript = gr.Textbox(label="Transcription") + sentiment_box = gr.Textbox(label="Predicted sentiment") + llm_output = gr.Textbox(label="LLM response") + + mic.change( + fn=handle_audio, + inputs=mic, + outputs=[transcript, sentiment_box, llm_output] + ) + +demo.launch() +``` + +### Step 4.5 - Run the final demo + +Now that the voice classification model is integrated with the baseline pipeline, test the full end-to-end experience locally. + +```bash +python app.py +``` + +Open: + +`http://127.0.0.1:7860` + +## Full final app + +Update `app.py` so it combines the baseline voice-to-LLM pipeline with ONNX-based sentiment prediction into one complete application: + +```python +import gradio as gr +import whisper +import requests +import onnxruntime as ort +import numpy as np +import librosa +from transformers import AutoFeatureExtractor, AutoConfig + +MODEL_DIR = "models/hubert_vsa_ravdess" +ONNX_PATH = "models/hubert_vsa_ravdess_onnx/hubert_vsa_ravdess_int8.onnx" +LOCAL_LLM_URL = "http://127.0.0.1:8080/v1/chat/completions" + +# Load Whisper for speech-to-text. +model = whisper.load_model("base") + +# Load the quantized ONNX sentiment model and preprocessing config. +session = ort.InferenceSession( + ONNX_PATH, + providers=["CPUExecutionProvider"] +) +feature_extractor = AutoFeatureExtractor.from_pretrained(MODEL_DIR) +id2label = AutoConfig.from_pretrained(MODEL_DIR).id2label + +def predict_sentiment(audio_path): + audio, _ = librosa.load(audio_path, sr=16000, mono=True) + inputs = feature_extractor( + audio, + sampling_rate=16000, + return_tensors="np", + return_attention_mask=True + ) + + logits = session.run(None, { + "input_values": inputs["input_values"], + "attention_mask": inputs["attention_mask"] + })[0] + + pred = int(np.argmax(logits)) + return id2label[pred] + +def handle_audio(audio_path): + if not audio_path: + return "", "", "" + + text = model.transcribe(audio_path)["text"] + sentiment = predict_sentiment(audio_path) + + prompt = f"The user said: '{text}'. Their tone sounds {sentiment}. Respond appropriately." + + response = requests.post( + LOCAL_LLM_URL, + json={ + "model": "local-model", + "messages": [ + { + "role": "system", + "content": "Respond helpfully and acknowledge the user's emotional tone." + }, + {"role": "user", "content": prompt}, + ], + }, + ) + + if response.status_code != 200: + return text, sentiment, "Error: LLM request failed" + + answer = response.json()["choices"][0]["message"]["content"] + return text, sentiment, answer + +with gr.Blocks() as demo: + mic = gr.Audio(sources="microphone", type="filepath") + transcript = gr.Textbox(label="Transcription") + sentiment_box = gr.Textbox(label="Predicted sentiment") + llm_output = gr.Textbox(label="LLM response") + + mic.change(fn=handle_audio, inputs=mic, outputs=[transcript, sentiment_box, llm_output]) + +demo.launch() +``` + +### What to expect + +Try different tones of speech. + +Example 1: + +- Voice: frustrated +- Input: "This is not working at all" +- Sentiment: angry +- Response: empathetic and helpful + +Example 2: + +- Voice: cheerful +- Input: "This is awesome!" +- Sentiment: happy +- Response: enthusiastic + +Voice-based sentiment classification can be ambiguous and sensitive to real-world variations. Voice captures how something is said (tone, pitch, emotion), but accents, speaking styles, background noise, and limited or unrepresentative training data can affect prediction accuracy. Text captures what is said (semantic meaning), which may not reflect intent. For example, a phrase like “That’s just great” may sound sarcastic in tone but appear positive in text, while facial signals may reinforce or contradict the tone. + +A more robust approach is to combine voice with other modalities such as text and visual cues, and to train models on more diverse datasets. By fusing these signals with simple rules or weighted averaging, you can reduce ambiguity and produce more reliable sentiment before sending it to the LLM. You can further improve performance by collecting user voice data responsibly, with consent and privacy safeguards, to better capture real-world variations and refine the model over time. + +## Troubleshooting + +- No sentiment output: check that `ONNX_PATH` points to the quantized ONNX model from the previous section. +- Slow response: make sure you are using `hubert_vsa_ravdess_int8.onnx` and that `llama-server` is already running. +- LLM not responding: verify that `llama-server` is still available at `http://127.0.0.1:8080`. +- Microphone input not updating: check browser microphone permissions. + + + +## Summary + +In this section, you integrated the quantized ONNX sentiment model into the voice pipeline. The LLM now receives both the transcribed text and the predicted voice sentiment, allowing it to generate more context-aware responses. + +You now have an end-to-end system that understands: + +- what the user says (text) +- how they say it (sentiment) +- and generates context-aware responses diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/6_vsapipeline4.png b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/6_vsapipeline4.png new file mode 100644 index 0000000000..e78df2612d Binary files /dev/null and b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/6_vsapipeline4.png differ diff --git a/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/_index.md b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/_index.md new file mode 100644 index 0000000000..22cbb71d11 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/_index.md @@ -0,0 +1,69 @@ +--- +title: Build a Sentiment-Aware Voice Assistant with On-Device LLMs + +draft: true +cascade: + draft: true + +description: Build an end-to-end, on-device voice assistant that understands both speech and emotion using Whisper, HuBERT, ONNX Runtime, and a local LLM with llama.cpp on Arm. + +minutes_to_complete: 90 + +who_is_this_for: This Learning Path is for developers, ML practitioners, and game developers interested in building on-device AI applications, including voice interfaces, real-time interactions with non-player characters (NPCs), and edge AI systems powered by LLMs on Arm platforms. + + +learning_objectives: + - Build a voice-to-LLM pipeline using Whisper and llama.cpp. + - Train a voice sentiment classification model using HuBERT on the RAVDESS dataset. + - Quantize the model and convert into ONNX Runtime for on-device inference. + - Integrate sentiment classification model with voice-to-LLM pipeline to generate context-aware LLM responses. + +prerequisites: + - Python 3.9 or later for programming. + - A working microphone for voice input. + - "ffmpeg installed on your system (required by Whisper for audio decoding)." + - "Build tools to compile llama.cpp (git, make, gcc/g++)." + - Basic Python and command-line knowledge. + +author: Bhanu Arya + +### Tags +skilllevels: Advanced +subjects: ML +armips: + - Cortex-A +operatingsystems: + - Linux + - Windows + - macOS +tools_software_languages: + - Python + - Transformers + - ONNX Runtime + - llama.cpp + - Gradio + +further_reading: + - resource: + title: Whisper model docs + link: https://github.com/openai/whisper + type: documentation + - resource: + title: llama.cpp + link: https://github.com/ggml-org/llama.cpp + type: documentation + - resource: + title: ONNX Runtime quantization + link: https://onnxruntime.ai/docs/performance/model-optimizations/quantization.html + type: documentation + - resource: + title: "Multimodal Sentiment Analysis: A Survey" + link: https://arxiv.org/abs/2305.07611 + type: research paper + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 +layout: "learningpathall" +learning_path_main_page: "yes" +---