Merge pull request #3214 from madeline-underwood/voice

Voice

Author: pareenaverma

content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/1_overview.md

@@ -1,12 +1,15 @@
 ---
-title: Overview
+title: Understand voice sentiment analysis for on-device AI
 weight: 2
 layout: learningpathall
 ---
+
+## Learn voice-based sentiment classification for LLM applications
+
 Voice-based LLM applications often rely primarily on transcribed text from speech input, such as in interactions with non-player characters in games or voice assistants. This approach can overlook vocal cues—like tone, pitch, and emotion—present in a speaker’s voice. As a result, responses may feel less natural and may not fully capture the user’s underlying intent.
 
-To address this, voice-based sentiment classification analyzes audio input to determine the user’s emotional state, which is then incorporated into the LLM prompt to enable more context-aware responses. In this Learning Path, you will build a sentiment-aware voice assistant that runs entirely on-device. The application records audio, performs transcription—converting speech into written text—using Whisper, classifies sentiment directly from the voice signal, and combines the transcript and voice-based sentiment to guide responses from a local LLM running with llama.cpp.
+To address this, voice-based sentiment classification analyzes audio input to determine the user's emotional state, which is then incorporated into the LLM prompt to enable more context-aware responses. In this Learning Path, you'll build a sentiment-aware voice assistant that runs entirely on-device. The application records audio, performs transcription—converting speech into written text—using Whisper, classifies sentiment directly from the voice signal, and combines the transcript and voice-based sentiment to guide responses from a local LLM running with llama.cpp.
 
-![Voice sentiment classification pipeline#center](1_vsapipeline2.png "Voice sentiment classification pipeline")
+![Pipeline diagram showing audio input flowing through Whisper transcription and HuBERT sentiment classification, then both text and sentiment being combined into an LLM prompt that generates context-aware responses#center](1_vsapipeline2.png "Voice sentiment classification pipeline")
 
-You will start by building a baseline voice-to-LLM pipeline—capturing audio, transcribing it into text, and using it to generate responses with an LLM. You will then extend this pipeline with a voice-based sentiment classification model. This involves training the model, optimizing it for efficient on-device inference, and integrating it into a unified application.
+You'll start by building a baseline voice-to-LLM pipeline—capturing audio, transcribing it into text, and using it to generate responses with an LLM. You'll then extend this pipeline with a voice-based sentiment classification model. This involves training the model, optimizing it for efficient on-device inference, and integrating it into a unified application.

content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/2_set-up-your-environment.md

@@ -6,6 +6,8 @@ weight: 3
 layout: learningpathall
 ---
 
+## Prepare the development environment for voice sentiment classification
+
 Before building the voice assistant, create a project workspace and set up an isolated `UV` environment. This keeps project dependencies separate from your system installation and makes it easier to reproduce the steps in the rest of the Learning Path.
 
 These instructions support Ubuntu, macOS, and Windows, with Python 3.9 or later and a working microphone.
@@ -42,7 +44,7 @@ py -3 --version
 
 ## Set up the Python environment with UV
 
-Install `UV` first so the `uv` command is available in your terminal. `UV` is a fast Python package and environment manager that you will use throughout this Learning Path to create the project environment and install dependencies.
+Install `UV` first so the `uv` command is available in your terminal. `UV` is a fast Python package and environment manager that you'll use throughout this Learning Path to create the project environment and install dependencies.
 
 {{< tabpane code=true >}}
   {{< tab header="Ubuntu/macOS" language="bash">}}
@@ -152,4 +154,13 @@ Run the following command from the `llama.cpp` directory:
 
 Leave this terminal running while you test the application in later steps. The server listens on a local OpenAI-compatible endpoint that your app will call to generate responses.
 
-At this point, your development environment is ready. You have installed the required audio and build tools, created a `UV` environment, installed the Python dependencies, and started a local `llama.cpp` server. In the next section, you will use this setup to build the baseline voice-to-LLM pipeline by creating a simple Gradio interface, transcribing microphone input with Whisper, and sending the transcript to the local LLM.
+## What you've learned and what's next
+
+In this section, you:
+
+- Installed system dependencies for audio processing and building llama.cpp
+- Created a UV-managed Python environment with required packages
+- Built llama.cpp from source for local LLM inference
+- Started the llama-server with a quantized Gemma model
+
+Your development environment is now ready with all tools needed for voice transcription, model training, and local LLM inference. In the next section, you'll build the baseline voice-to-LLM pipeline using Gradio, Whisper, and llama.cpp.

content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/3_build-baseline-voice-to-llm-pipeline.md

@@ -4,16 +4,18 @@ weight: 4
 layout: learningpathall
 ---
 
-In this section, you will build an end-to-end pipeline that:
+## Create the baseline voice-to-LLM pipeline
 
-1. Records audio from your microphone
-2. Transcribes it to text using Whisper
-3. Sends the text to a locally hosted LLM
-4. Displays the model's response
+In this section, you'll build an end-to-end pipeline that:
+
+- Records audio from your microphone
+- Transcribes it to text using Whisper
+- Sends the text to a locally hosted LLM
+- Displays the model's response
 
 This forms the foundation of your voice assistant.
 
-![Baseline voice-to-LLM pipeline#center](3_vsapipeline1.png "Baseline voice-to-LLM pipeline")
+![Diagram showing the baseline pipeline with three stages: audio input from microphone flows to Whisper for transcription, then transcript text is sent to local LLM, which returns a response#center](3_vsapipeline1.png "Baseline voice-to-LLM pipeline")
 
 Before you begin, make sure you have completed the environment setup in the previous section and that your `llama-server` is still running.
 
@@ -177,7 +179,7 @@ with gr.Blocks() as demo:
 demo.launch()
 ```
 
-## What you should see
+## Expected output
 
 After recording audio in the browser:
 
@@ -191,4 +193,13 @@ After recording audio in the browser:
 - Whisper is slow on first run: this is expected due to model download and initialization.
 - Microphone not working: check browser permissions for microphone access.
 
-At this point, you have a working voice-to-LLM pipeline. In the next section, you will extend this pipeline by adding a voice sentiment classification model.
+## What you've learned and what's next
+
+In this section, you:
+
+- Created a Gradio web interface with microphone input
+- Integrated Whisper for speech-to-text transcription
+- Connected to a local LLM endpoint using llama.cpp
+- Built a working voice-to-LLM pipeline
+
+You now have a baseline voice assistant that transcribes speech and generates responses. In the next section, you'll train a voice sentiment classification model to extend this pipeline with emotion awareness.

content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/4_train-voice-sentiment-model.md

@@ -3,6 +3,7 @@ title: Train the voice sentiment classification model
 weight: 5
 layout: learningpathall
 ---
+## Train a HuBERT model for voice sentiment classification
 In this section, you will train a model to classify voice sentiment directly from audio. The raw waveform is converted into audio-based features, then passed through a pre-trained HuBERT model with a classification head.
 
 For this Learning Path, you train the model on the RAVDESS dataset using three sentiment classes: `neutral`, `happy`, and `angry`. This keeps the demo manageable while still showing a realistic transfer-learning workflow. You can extend the same approach to more sentiment classes or other datasets.
@@ -13,7 +14,7 @@ This example uses a simple per-sample training loop for clarity. On CPU, trainin
 
 All code snippets in Steps 2.1 to 2.5 are meant to be added to a single Python file: `train_sentiment_model.py` in your project root (`$HOME/voice-sentiment-assistant`).
 
-Make sure you are located in the project directory:
+Make sure you're located in the project directory:
 
 ```bash
 cd $HOME/voice-sentiment-assistant
@@ -208,7 +209,7 @@ for epoch in range(EPOCHS):
     print(f"Validation accuracy: {accuracy:.2f}")
 ```
 
-At the end of this step, you will have a trained model and a simple validation accuracy printed after each epoch. You should typically see validation accuracy improve over time, although the exact values will vary depending on randomness and hardware.
+At the end of this step, you'll have a trained model and a simple validation accuracy printed after each epoch. You should typically see validation accuracy improve over time, although the exact values will vary depending on randomness and hardware.
 
 ### Step 2.5 - Save the model
 
@@ -392,4 +393,13 @@ Run the training script:
 python train_sentiment_model.py
 ```
 
-At this point, you have a trained voice sentiment classification model saved locally. In the next section, you will convert this model to ONNX format and compress it for more efficient inference.
+## What you've learned and what's next
+
+In this section, you:
+
+- Downloaded and prepared the RAVDESS speech emotion dataset
+- Fine-tuned a HuBERT model for voice sentiment classification
+- Validated the model's accuracy on neutral, happy, and angry emotions
+- Saved the trained model and feature extractor for inference
+
+You now have a trained voice sentiment model ready for deployment. In the next section, you'll export this model to ONNX format and apply int-8 quantization to optimize it for efficient on-device inference.

content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/5_compress-and-convert-the-model.md

@@ -3,7 +3,10 @@ title: Convert and quantize the model
 weight: 6
 layout: learningpathall
 ---
-In this section, you will convert the trained model to ONNX format and compress it for efficient on-device inference. First, export the trained PyTorch model to ONNX. Then apply post-training quantization to reduce model size and improve runtime efficiency.
+
+## Export and optimize the model for on-device inference
+
+In this section, you'll convert the trained model to ONNX format and compress it for efficient on-device inference. First, export the trained PyTorch model to ONNX. Then apply post-training quantization to reduce model size and improve runtime efficiency.
 
 ### Step 3.1 - Export to ONNX
 
@@ -70,7 +73,7 @@ You can verify the exported file with:
 ls models/hubert_vsa_ravdess_onnx
 ```
 
-### Step 3.2 - Quantize model
+### Step 3.2 - Quantize the model
 
 This step applies dynamic INT8 quantization to the ONNX model. The model weights are quantized to integer 8 bits ahead of time, while activations are quantized dynamically during inference.
 
@@ -178,7 +181,7 @@ If this file does not exist on your system, replace `SAMPLE_PATH` with any `.wav
 
 After this step, you have verified that quantized ONNX inference works with your trained model.
 
-You should see a predicted label such as:
+The output is similar to:
 
 ```text
 Predicted: happy
@@ -198,7 +201,7 @@ models/
 
 You can also compare the model metrics before and after quantization.
 
-![Model ONNX conversion and int-8 quantization results#center](4_modelconversionandcompression.png "Model ONNX conversion and int-8 quantization results")
+![Diagram showing model file sizes and performance metrics comparing the original ONNX model to the int-8 quantized version, demonstrating reduced size and improved inference speed#center](4_modelconversionandcompression.png "Model ONNX conversion and int-8 quantization results")
 
 Typical results look like:
 
@@ -211,6 +214,15 @@ Quantized INT8 ONNX model: smaller size, lower memory footprint, faster CPU infe
 
 - ONNX export fails: ensure the trained model exists in `models/hubert_vsa_ravdess`.
 - Inference error about missing inputs: confirm that both `input_values` and `attention_mask` are passed to the ONNX session.
-- Slow inference: make sure you are using `hubert_vsa_ravdess_int8.onnx` rather than the unquantized ONNX model.
+- Slow inference: make sure you're using `hubert_vsa_ravdess_int8.onnx` rather than the unquantized ONNX model.
+
+## What you've learned and what's next
+
+In this section, you:
+
+- Exported the PyTorch sentiment model to ONNX format
+- Applied int-8 quantization to reduce model size and improve inference speed
+- Verified that the quantized ONNX model produces correct predictions
+- Prepared the model for efficient CPU inference on Arm devices
 
-The model is now ready for integration into the voice pipeline in the next section.
+You now have an optimized sentiment classification model ready for production use. In the next section, you'll integrate this ONNX model into the voice-to-LLM pipeline to create a complete sentiment-aware voice assistant.

content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/6_integrate-voice-sentiment-classification-model.md

@@ -3,17 +3,20 @@ title: Integrate the voice sentiment classification model
 weight: 7
 layout: learningpathall
 ---
-In this section, you will integrate the voice sentiment classification model into the baseline voice-to-LLM pipeline and build a sentiment-aware voice assistant. You will update the prompt sent to the LLM so it includes both the transcript and the predicted sentiment, combining speech, emotion, and language understanding into a single application.
+
+## Build the complete sentiment-aware voice assistant
+
+In this section, you'll integrate the voice sentiment classification model into the baseline voice-to-LLM pipeline and build a sentiment-aware voice assistant. You'll update the prompt sent to the LLM so it includes both the transcript and the predicted sentiment, combining speech, emotion, and language understanding into a single application.
 
 End-to-end response time may take a few seconds because the application runs Whisper transcription, ONNX sentiment inference, and local LLM generation for each interaction.
 
-![Integrated voice-to-LLM pipeline#center](6_vsapipeline4.png "Integrated voice-to-LLM pipeline")
+![Integrated pipeline diagram showing audio input splitting into two parallel paths: Whisper produces transcript text and HuBERT produces sentiment label, both feeding into an LLM prompt that generates context-aware responses#center](6_vsapipeline4.png "Integrated voice-to-LLM pipeline")
 
 In Steps 4.1 to 4.4, continue editing the same `app.py` file you created in the baseline pipeline section (`~/voice-sentiment-assistant/app.py`).
 
 ### Step 4.1 - Load the voice sentiment classification model
 
-This step initializes the trained and quantized ONNX model, then loads the matching preprocessing components used during training. You will reuse these objects every time you run sentiment inference on new audio.
+This step initializes the trained and quantized ONNX model, then loads the matching preprocessing components used during training. You'll reuse these objects every time you run sentiment inference on new audio.
 
 Add these imports and model-loading definitions to `app.py`:
 - add the imports with your other imports at the top of the file
@@ -37,7 +40,7 @@ feature_extractor = AutoFeatureExtractor.from_pretrained(MODEL_DIR)
 id2label = AutoConfig.from_pretrained(MODEL_DIR).id2label
 ```
 
-### Step 4.2 - Create sentiment prediction function
+### Step 4.2 - Create the sentiment prediction function
 
 Next, create a helper function that takes an audio file path, runs the sentiment model, and returns the predicted label. This adds the emotion-aware part of the assistant.
 
@@ -101,7 +104,7 @@ def handle_audio(audio_path):
     return text, sentiment, answer
 ```
 
-### Step 4.4 - Update User Interface (UI)
+### Step 4.4 - Update user interface (UI)
 
 In this step, update the interface so it displays three outputs: the transcript, the predicted sentiment, and the final LLM response. For consistency with the earlier page, connect the microphone input with `mic.change(...)`.
 
@@ -244,7 +247,7 @@ A more robust approach is to combine voice with other modalities such as text an
 ## Troubleshooting
 
 - No sentiment output: check that `ONNX_PATH` points to the quantized ONNX model from the previous section.
-- Slow response: make sure you are using `hubert_vsa_ravdess_int8.onnx` and that `llama-server` is already running.
+- Slow response: make sure you're using `hubert_vsa_ravdess_int8.onnx` and that `llama-server` is already running.
 - LLM not responding: verify that `llama-server` is still available at `http://127.0.0.1:8080`.
 - Microphone input not updating: check browser microphone permissions.
 

content/learning-paths/mobile-graphics-and-gaming/voice-sentiment-analysis-with-llm/_index.md

@@ -1,10 +1,6 @@
 ---
 title: Build a Sentiment-Aware Voice Assistant with On-Device LLMs
 
-draft: true
-cascade:
-    draft: true
-
 description: Build an end-to-end, on-device voice assistant that understands both speech and emotion using Whisper, HuBERT, ONNX Runtime, and a local LLM with llama.cpp on Arm.
 
 minutes_to_complete: 90