Merge pull request #2715 from pareenaverma/content_review

New Stability Audio LP with Executorch

Author: pareenaverma

content/learning-paths/mobile-graphics-and-gaming/run-stable-audio-with-executorch/1-prerequisites.md

@@ -0,0 +1,98 @@
+---
+title: Set up your development environment
+weight: 2
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Overview
+
+In this Learning Path, you will learn how to convert the Stable Audio Open Small model to ExecuTorch (.pte) format, then build an audio generation application to run on Android or macOS. ExecuTorch is a PyTorch framework designed for on-device inference on edge and mobile devices.
+
+## Identify requirements
+
+Before you begin, you need a development environment with the required software:
+
+- An Android device with an Arm CPU supporting FEAT_DotProd (dotprod) and optionally FEAT_I8MM (i8mm), with at least 8 GB of RAM.
+- Python 3.10 or newer.
+- CMake version 3.16.0 or newer.
+- Android NDK r27c (if building for Android).
+
+## Create workspace directory
+
+Create a workspace directory to manage the dependencies and repositories.
+
+Export the `WORKSPACE` variable to point to this directory:
+
+```bash
+mkdir my-workspace
+export WORKSPACE=$PWD/my-workspace
+cd $WORKSPACE
+```
+
+## Install Python 3.10
+
+You need Python 3.10 or newer for compatibility with the required packages.
+
+{{< tabpane code=true >}}
+  {{< tab header="Linux">}}
+sudo apt install -y python3.10 python3.10-venv
+  {{< /tab >}}
+  {{< tab header="macOS">}}
+brew install python@3.10
+brew link python@3.10 --force
+  {{< /tab >}}
+{{< /tabpane >}}
+
+Verify the installation:
+
+```console
+python3 --version
+```
+
+The output is similar to:
+
+```output
+Python 3.10.19
+```
+
+## Install CMake
+
+CMake automates the build process for the audio generation application.
+
+{{< tabpane code=true >}}
+  {{< tab header="Linux">}}
+sudo apt update
+sudo apt install cmake g++ git
+  {{< /tab >}}
+  {{< tab header="macOS">}}
+brew install cmake
+  {{< /tab >}}
+{{< /tabpane >}}
+
+Verify the installation:
+
+```console
+cmake --version
+```
+
+The output is similar to:
+
+```output
+cmake version 4.2.1
+```
+
+See the [CMake install guide](/install-guides/cmake/) for additional help.
+
+## Clone the ML examples repository
+
+Clone the Arm ML examples repository, which contains the scripts and application code:
+
+```bash
+cd $WORKSPACE
+git clone https://github.com/Arm-Examples/ML-examples.git
+cd ML-examples/kleidiai-examples/audiogen-et/
+```
+
+You now have the necessary tools and repository to proceed with model setup.

content/learning-paths/mobile-graphics-and-gaming/run-stable-audio-with-executorch/2-download-model.md

@@ -0,0 +1,62 @@
+---
+title: Download the Stable Audio Open Small model
+weight: 3
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## About the Stable Audio Open Small model
+
+Stable Audio Open Small is an open-source model optimized for generating short audio samples, sound effects, and production elements using text prompts. The model consists of three main submodules:
+
+- **Conditioners**: Include a T5-based text encoder for input prompts and a numerical duration encoder. These components encode the inputs into numerical values to be passed to the DiT model.
+- **Diffusion Transformer (DiT)**: Takes random noise and denoises it through multiple steps to produce structured latent audio, guided by conditioner embeddings.
+- **AutoEncoder**: Compresses input waveforms into a manageable sequence length for processing by the DiT model. At the end of the denoising step, it decompresses the result into a waveform.
+
+You can learn more about [stable-audio-open-small on Hugging Face](https://huggingface.co/stabilityai/stable-audio-open-small).
+
+## Download model files
+
+[Log in](https://huggingface.co/login) to HuggingFace and navigate to the model landing page:
+
+```url
+https://huggingface.co/stabilityai/stable-audio-open-small/tree/main
+```
+
+You will need to fill out a form with your contact information to access the model.
+
+Download the following files:
+- `model_config.json` (configuration file)
+- `model.ckpt` (model checkpoint)
+
+Copy both files to your workspace directory.
+
+Verify the files exist:
+
+```bash
+ls $WORKSPACE/model_config.json 
+ls $WORKSPACE/model.ckpt
+```
+
+
+## Understand prompt structure
+
+A good prompt for Stable Audio Open Small includes:
+
+- Music genre and subgenre
+- Musical elements (texture, rhythm, articulation)
+- Musical atmosphere (mood and emotion)
+- Tempo in beats per minute (BPM)
+
+The order of prompt parameters matters. For example:
+
+```text
+warm arpeggios on house beats 120BPM with drums effect
+```
+
+For more information, see the [Prompt structure user guide](https://stableaudio.com/user-guide/prompt-structure).
+
+You can explore additional training and inference code in the [Stable Audio Tools repository](https://github.com/Stability-AI/stable-audio-tools).
+
+You now have the model files ready for conversion to ExecuTorch format.

content/learning-paths/mobile-graphics-and-gaming/run-stable-audio-with-executorch/3-convert-model.md

@@ -0,0 +1,95 @@
+---
+title: Convert the model to ExecuTorch format
+weight: 4
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Create a virtual environment
+
+Create and activate a Python virtual environment to manage dependencies. Python 3.10 is recommended for compatibility with the required packages:
+
+```bash
+cd $WORKSPACE/ML-examples/kleidiai-examples/audiogen-et/
+python3.10 -m venv .venv
+source .venv/bin/activate
+```
+
+## Install ExecuTorch
+
+Install the ExecuTorch package version 1.0.0:
+
+```bash
+pip install executorch==1.0.0
+```
+
+Alternatively, you can clone the ExecuTorch repository and run the installation script:
+
+```bash
+git clone https://github.com/pytorch/executorch.git
+cd executorch
+git checkout v1.0.0
+bash ./install_executorch.sh
+cd ..
+```
+
+## Install Stable Audio tools
+
+Install the Stable Audio Open tools dependency:
+
+```bash
+pip install git+https://github.com/Stability-AI/stable-audio-tools.git@31932349d98c550c48711e7a5a40b24aa3d7c509
+```
+
+## Export the model submodules
+
+Now you can export the three model submodules to ExecuTorch format using the provided script.
+
+Run the export script from the `audiogen-et` directory:
+
+```bash
+python ./scripts/export_sao.py --ckpt_path $WORKSPACE/model.ckpt --model_config $WORKSPACE/model_config.json
+```
+
+The script converts all three submodules (Conditioners, DiT, and AutoEncoder) and exports them to `.pte` files.
+
+### Troubleshooting conversion issues
+
+If you see a `FileNotFoundError` related to missing `.fbs` files during conversion:
+
+```output
+FileNotFoundError: [Errno 2] No such file or directory: '../kleidiai-examples/audiogen-et/scripts/executorch/exir/_serialize/program.fbs'
+```
+
+Run these commands to copy the required schema files:
+
+```bash
+export EXECUTORCH_ROOT=<PATH-TO-EXECUTORCH>
+cp $EXECUTORCH_ROOT/schema/program.fbs $EXECUTORCH_ROOT/exir/_serialize/program.fbs
+cp $EXECUTORCH_ROOT/schema/scalar_type.fbs $EXECUTORCH_ROOT/exir/_serialize/scalar_type.fbs
+```
+
+Then retry the export script.
+
+## Verify exported models
+
+After successful conversion, three `.pte` model files are created in the current directory:
+- `conditioners_model.pte`
+- `dit_model.pte`
+- `autoencoder_model.pte`
+
+These files are required to run the audio generation application on Android or macOS.
+
+Verify the files were created:
+
+```bash
+ls -lh conditioners_model.pte dit_model.pte autoencoder_model.pte
+```
+
+The files should be in `$WORKSPACE/ML-examples/kleidiai-examples/audiogen-et/`.
+
+
+In this section, you installed Executorch and the Stable Audio tools dependencies. You converted the three Stable Audio Open Small submodules to ExecuTorch format.
+
+In the following sections, you will build the audio generation application for your target platform (Android or macOS).

content/learning-paths/mobile-graphics-and-gaming/run-stable-audio-with-executorch/4-build-run-macos.md

@@ -0,0 +1,130 @@
+---
+title: Build and run on macOS
+weight: 5
+
+### FIXED, DO NOT MODIFY
+layout: learningpathall
+---
+
+## Overview
+
+In this section, you will build and run the audio generation application on macOS. The application uses ExecuTorch with XNNPack and Arm KleidiAI for optimized inference on Arm CPUs.
+
+## Set up the environment
+
+Start a fresh virtual environment:
+
+```bash
+cd $WORKSPACE/ML-examples/kleidiai-examples/audiogen-et/
+python3.10 -m venv new-venv
+source new-venv/bin/activate
+```
+
+If you haven't already installed ExecuTorch, install it now:
+
+```bash
+pip install executorch==1.0.0
+```
+
+## Set model path
+
+Set the `EXECUTORCH_MODELS_PATH` environment variable to the directory containing your exported ExecuTorch models:
+
+```bash
+export EXECUTORCH_MODELS_PATH=$WORKSPACE/ML-examples/kleidiai-examples/audiogen-et
+```
+
+This should be the directory where the `.pte` model files were created during the conversion step.
+
+## Build the application
+
+Navigate to the `app` directory and create a build directory:
+
+```bash
+cd app
+mkdir build && cd build
+```
+
+Run CMake to configure the build:
+
+```bash
+cmake ..
+```
+
+Build the application:
+
+```bash
+make -j8
+```
+
+The build process creates an `audiogen` executable in the `build` directory.
+
+## Download the tokenizer model
+
+The audio generation application uses a SentencePiece-based tokenizer. Download the `spiece.model` file from HuggingFace:
+
+```bash
+curl -L https://huggingface.co/google-t5/t5-base/resolve/main/spiece.model -o $EXECUTORCH_MODELS_PATH/spiece.model
+```
+
+Verify the file was downloaded:
+
+```bash
+ls -lh $EXECUTORCH_MODELS_PATH/spiece.model
+```
+
+## Run the application
+
+Run the `audiogen` application with three required arguments:
+
+- Model Path (`-m`): Directory containing your ExecuTorch models and `spiece.model` files
+- Prompt (`-p`): Text description of the desired audio
+- CPU Threads (`-t`): Number of CPU threads to use
+
+Run the application with an example prompt:
+
+```bash
+./audiogen -m $EXECUTORCH_MODELS_PATH -p "warm arpeggios on house beats 120BPM with drums effect" -t 4
+```
+
+The application processes the prompt through the three model submodules and generates audio.
+
+The output should look like:
+
+```output
+I 00:00:00.002858 executorch:main.cpp:280] Resetting threadpool with num threads = 4
+I 00:00:00.003228 executorch:threadpool.cpp:48] Resetting threadpool to 4 threads.
+I 00:00:00.003543 executorch:main.cpp:288] Using 4 threads
+I 00:00:00.003620 executorch:main.cpp:294] Model (/Users/parver01/my-workspace/ML-examples/kleidiai-examples/audiogen-et/conditioners_model.pte) loaded
+I 00:00:00.003627 executorch:main.cpp:298] Model (/Users/parver01/my-workspace/ML-examples/kleidiai-examples/audiogen-et/dit_model.pte) loaded
+I 00:00:00.003629 executorch:main.cpp:302] Model (/Users/parver01/my-workspace/ML-examples/kleidiai-examples/audiogen-et/autoencoder_model.pte) loaded
+I 00:00:04.723761 executorch:main.cpp:478] Output saved to warm_arpeggios_on_house_beats_120bpm_with_drums_effect_99.wav
+I 00:00:04.723770 executorch:main.cpp:487] T5: 72 ms
+I 00:00:04.723771 executorch:main.cpp:488] DiT: 1474 ms
+I 00:00:04.723772 executorch:main.cpp:489] DiT Avg per step: 184.250000 ms
+I 00:00:04.723784 executorch:main.cpp:490] AutoEncoder: 3127 ms
+I 00:00:04.723785 executorch:main.cpp:491] Total execution time: 4673 ms
+```
+## Verify the output
+
+If successful, the generated audio is saved as a `.wav` file in the current directory. The filename is based on the prompt text:
+
+```bash
+ls -lh warm_arpeggios_on_house_beats_120bpm_with_drums_effect_99.wav
+```
+
+You can play the audio file using any audio player on your macOS system:
+
+```bash
+open warm_arpeggios_on_house_beats_120bpm_with_drums_effect_99.wav
+```
+
+In this section:
+- You built the audio generation application for macOS
+- You downloaded the required tokenizer model
+- You generated audio from a text prompt using ExecuTorch
+- You verified the generated audio output
+
+You can now experiment with different prompts to generate various audio samples. The application uses Arm KleidiAI optimizations to accelerate inference on Arm CPUs.
+
+If you want to deploy to Android instead, proceed to the next section.