Skip to content

Commit 7f1eb4d

Browse files
mergennachinchizkiyahu
authored andcommitted
Enable Voxtral Realtime on XNNPACK (CPU) (pytorch#17431)
Adds Mistral's Voxtral-Mini-4B-Realtime-2602 (~4B parameter streaming speech-to-text model) to ExecuTorch with XNNPACK backend support. Phase 1: Self-contained eager model (model.py) with direct Mistral checkpoint loading, multi-method export (audio_encoder, text_decoder, token_embedding) to a single .pte, and TorchAO quantization (4bit blockwise, 8bit dynamic activation for linear layer and 8bit weight per-channel for embeddings). Phase 2: C++ runner for offline transcription. Loads preprocessor.pte for mel spectrogram computation, runs audio encoding, then autoregressive decoding with element-wise audio+text embedding fusion. Phase 3: Streaming support (follow-up PR). Phase 4: Enable on CUDA and Metal (follow-up PR) Example output (8da4w quantized, 30s LibriSpeech audio): ``` $ cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner \ --model_path voxtral_realtime.pte \ --tokenizer_path tekken.json \ --preprocessor_path preprocessor.pte \ --audio_path output.wav Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel. Nor is Mr. Quilter's manner less interesting than his matter. He tells us that at this festive season of the year, with Christmas and roast beef looming before us, similes drawn from eating and its results occur most readily to the mind. He has grave doubts whether Sir Frederick Layton's work is really Greek after all, and... Generated 392 tokens in 44s (~8.8 tok/s) on M1 Macbook ``` Test Plan: https://github.com/pytorch/executorch/actions/runs/21986674876/job/63522558208?pr=17431
1 parent 817f599 commit 7f1eb4d

17 files changed

Lines changed: 2202 additions & 153 deletions

File tree

.ci/scripts/export_model_artifact.sh

Lines changed: 45 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ Arguments:
1919
hf_model HuggingFace model ID (required)
2020
Supported models:
2121
- mistralai/Voxtral-Mini-3B-2507
22+
- mistralai/Voxtral-Mini-4B-Realtime-2602
2223
- openai/whisper series (whisper-{small, medium, large, large-v2, large-v3, large-v3-turbo})
2324
- google/gemma-3-4b-it
2425
- nvidia/parakeet-tdt
@@ -119,9 +120,17 @@ case "$HF_MODEL" in
119120
PREPROCESSOR_FEATURE_SIZE=""
120121
PREPROCESSOR_OUTPUT=""
121122
;;
123+
mistralai/Voxtral-Mini-4B-Realtime-2602)
124+
MODEL_NAME="voxtral_realtime"
125+
TASK=""
126+
MAX_SEQ_LEN=""
127+
EXTRA_PIP="mistral-common librosa"
128+
PREPROCESSOR_FEATURE_SIZE=""
129+
PREPROCESSOR_OUTPUT=""
130+
;;
122131
*)
123132
echo "Error: Unsupported model '$HF_MODEL'"
124-
echo "Supported models: mistralai/Voxtral-Mini-3B-2507, openai/whisper-{small, medium, large, large-v2, large-v3, large-v3-turbo}, google/gemma-3-4b-it, nvidia/parakeet-tdt"
133+
echo "Supported models: mistralai/Voxtral-Mini-3B-2507, mistralai/Voxtral-Mini-4B-Realtime-2602, openai/whisper-{small, medium, large, large-v2, large-v3, large-v3-turbo}, google/gemma-3-4b-it, nvidia/parakeet-tdt"
125134
exit 1
126135
;;
127136
esac
@@ -201,6 +210,41 @@ if [ "$MODEL_NAME" = "parakeet" ]; then
201210
exit 0
202211
fi
203212

213+
# Voxtral Realtime uses a custom export script
214+
if [ "$MODEL_NAME" = "voxtral_realtime" ]; then
215+
pip install safetensors huggingface_hub
216+
217+
# Download model weights from HuggingFace (requires HF_TOKEN for gated model)
218+
LOCAL_MODEL_DIR="${OUTPUT_DIR}/model_weights"
219+
python -c "from huggingface_hub import snapshot_download; snapshot_download('${HF_MODEL}', local_dir='${LOCAL_MODEL_DIR}')"
220+
221+
# Voxtral Realtime has its own quantization flags (no --qlinear_encoder)
222+
VR_QUANT_ARGS=""
223+
if [ "$QUANT_NAME" = "quantized-8da4w" ]; then
224+
VR_QUANT_ARGS="--qlinear 8da4w --qlinear-group-size 32 --qembedding 8w"
225+
fi
226+
227+
python -m executorch.examples.models.voxtral_realtime.export_voxtral_rt \
228+
--model-path "$LOCAL_MODEL_DIR" \
229+
--backend xnnpack \
230+
--output-dir "${OUTPUT_DIR}" \
231+
${VR_QUANT_ARGS}
232+
233+
# Export preprocessor
234+
python -m executorch.extension.audio.mel_spectrogram \
235+
--feature_size 128 \
236+
--max_audio_len 300 \
237+
--output_file "${OUTPUT_DIR}/preprocessor.pte"
238+
239+
test -f "${OUTPUT_DIR}/model.pte"
240+
test -f "${OUTPUT_DIR}/preprocessor.pte"
241+
# Copy tokenizer from downloaded model weights
242+
cp "$LOCAL_MODEL_DIR/tekken.json" "${OUTPUT_DIR}/tekken.json"
243+
ls -al "${OUTPUT_DIR}"
244+
echo "::endgroup::"
245+
exit 0
246+
fi
247+
204248
MAX_SEQ_LEN_ARG=""
205249
if [ -n "$MAX_SEQ_LEN" ]; then
206250
MAX_SEQ_LEN_ARG="--max_seq_len $MAX_SEQ_LEN"

.ci/scripts/test_model_e2e.sh

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ Arguments:
2222
- openai/whisper series (whisper-{small, medium, large, large-v2, large-v3, large-v3-turbo})
2323
- google/gemma-3-4b-it
2424
- nvidia/parakeet-tdt
25+
- mistralai/Voxtral-Mini-4B-Realtime-2602
2526
2627
quant_name Quantization type (required)
2728
Options:
@@ -135,9 +136,21 @@ case "$HF_MODEL" in
135136
AUDIO_FILE="test_audio.wav"
136137
IMAGE_PATH=""
137138
;;
139+
mistralai/Voxtral-Mini-4B-Realtime-2602)
140+
MODEL_NAME="voxtral_realtime"
141+
RUNNER_TARGET="voxtral_realtime_runner"
142+
RUNNER_PATH="voxtral_realtime"
143+
EXPECTED_OUTPUT="Quilter"
144+
PREPROCESSOR="preprocessor.pte"
145+
TOKENIZER_URL="https://huggingface.co/mistralai/Voxtral-Mini-4B-Realtime-2602/resolve/main" # @lint-ignore
146+
TOKENIZER_FILE="tekken.json"
147+
AUDIO_URL=""
148+
AUDIO_FILE="test_audio.wav"
149+
IMAGE_PATH=""
150+
;;
138151
*)
139152
echo "Error: Unsupported model '$HF_MODEL'"
140-
echo "Supported models: mistralai/Voxtral-Mini-3B-2507, openai/whisper series (whisper-{small, medium, large, large-v2, large-v3, large-v3-turbo}), google/gemma-3-4b-it, nvidia/parakeet-tdt"
153+
echo "Supported models: mistralai/Voxtral-Mini-3B-2507, mistralai/Voxtral-Mini-4B-Realtime-2602, openai/whisper series (whisper-{small, medium, large, large-v2, large-v3, large-v3-turbo}), google/gemma-3-4b-it, nvidia/parakeet-tdt"
141154
exit 1
142155
;;
143156
esac
@@ -150,8 +163,8 @@ echo "::endgroup::"
150163
echo "::group::Prepare $MODEL_NAME Artifacts"
151164

152165

153-
# Download tokenizer files (skip for parakeet which exports tokenizer with model)
154-
if [ "$MODEL_NAME" != "parakeet" ]; then
166+
# Download tokenizer files (skip for parakeet and voxtral_realtime which bundle tokenizer in export)
167+
if [ "$MODEL_NAME" != "parakeet" ] && [ "$MODEL_NAME" != "voxtral_realtime" ]; then
155168
if [ "$TOKENIZER_FILE" != "" ]; then
156169
curl -L $TOKENIZER_URL/$TOKENIZER_FILE -o $MODEL_DIR/$TOKENIZER_FILE
157170
else
@@ -164,7 +177,7 @@ fi
164177
# Download test files
165178
if [ "$AUDIO_URL" != "" ]; then
166179
curl -L $AUDIO_URL -o ${MODEL_DIR}/$AUDIO_FILE
167-
elif [[ "$MODEL_NAME" == *whisper* ]]; then
180+
elif [[ "$MODEL_NAME" == *whisper* ]] || [ "$MODEL_NAME" = "voxtral_realtime" ]; then
168181
conda install -y -c conda-forge "ffmpeg<8"
169182
pip install datasets soundfile
170183
pip install torchcodec --extra-index-url https://download.pytorch.org/whl/nightly/cpu
@@ -222,6 +235,9 @@ case "$MODEL_NAME" in
222235
RUNNER_ARGS="$RUNNER_ARGS --data_path ${MODEL_DIR}/aoti_cuda_blob.ptd"
223236
fi
224237
;;
238+
voxtral_realtime)
239+
RUNNER_ARGS="--model_path ${MODEL_DIR}/model.pte --tokenizer_path ${MODEL_DIR}/$TOKENIZER_FILE --preprocessor_path ${MODEL_DIR}/$PREPROCESSOR --audio_path ${MODEL_DIR}/$AUDIO_FILE --temperature 0"
240+
;;
225241
esac
226242

227243
OUTPUT=$($RUNNER_BIN $RUNNER_ARGS 2>&1)

.github/workflows/pull.yml

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,42 @@ jobs:
196196
bash .ci/scripts/test_model_e2e.sh xnnpack "nvidia/parakeet-tdt" "quantized-8da4w" "./parakeet_output"
197197
echo "::endgroup::"
198198
199+
test-voxtral-realtime-xnnpack-linux:
200+
name: test-voxtral-realtime-xnnpack-linux
201+
uses: pytorch/test-infra/.github/workflows/linux_job_v2.yml@main
202+
permissions:
203+
id-token: write
204+
contents: read
205+
strategy:
206+
fail-fast: false
207+
with:
208+
runner: linux.4xlarge.memory
209+
docker-image: ci-image:executorch-ubuntu-22.04-clang12
210+
submodules: 'recursive'
211+
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
212+
timeout: 120
213+
secrets-env: VOXTRAL_REALTIME_HF_TOKEN
214+
script: |
215+
set -eux
216+
217+
# The generic Linux job chooses to use base env, not the one setup by the image
218+
CONDA_ENV=$(conda env list --json | jq -r ".envs | .[-1]")
219+
conda activate "${CONDA_ENV}"
220+
221+
export HF_TOKEN="${SECRET_VOXTRAL_REALTIME_HF_TOKEN}"
222+
223+
echo "::group::Setup ExecuTorch"
224+
./install_executorch.sh
225+
echo "::endgroup::"
226+
227+
echo "::group::Export Voxtral Realtime with XNNPACK"
228+
bash .ci/scripts/export_model_artifact.sh xnnpack "mistralai/Voxtral-Mini-4B-Realtime-2602" "quantized-8da4w" "./voxtral_realtime_output"
229+
echo "::endgroup::"
230+
231+
echo "::group::Test Voxtral Realtime with XNNPACK"
232+
bash .ci/scripts/test_model_e2e.sh xnnpack "mistralai/Voxtral-Mini-4B-Realtime-2602" "quantized-8da4w" "./voxtral_realtime_output"
233+
echo "::endgroup::"
234+
199235
test-llama-runner-linux:
200236
# Test Both linux x86 and linux aarch64
201237
name: test-llama-runner-linux

Makefile

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
# SUPPORTED MODELS:
1616
# -----------------
1717
# - voxtral: Multimodal voice + text model (CPU, CUDA, Metal)
18+
# - voxtral_realtime: Realtime speech-to-text model (CPU)
1819
# - whisper: Speech recognition model (CPU, CUDA, Metal)
1920
# - parakeet: Speech recognition model (CPU, CUDA, Metal)
2021
# - sortformer: Speaker diarization model (CPU)
@@ -90,13 +91,14 @@
9091
#
9192
# ==============================================================================
9293

93-
.PHONY: voxtral-cuda voxtral-cpu voxtral-metal whisper-cuda whisper-cuda-debug whisper-cpu whisper-metal parakeet-cuda parakeet-cuda-debug parakeet-cpu parakeet-metal sortformer-cpu silero-vad-cpu llama-cpu llava-cpu gemma3-cuda gemma3-cpu clean help
94+
.PHONY: voxtral-cuda voxtral-cpu voxtral-metal voxtral_realtime-cpu whisper-cuda whisper-cuda-debug whisper-cpu whisper-metal parakeet-cuda parakeet-cuda-debug parakeet-cpu parakeet-metal sortformer-cpu silero-vad-cpu llama-cpu llava-cpu gemma3-cuda gemma3-cpu clean help
9495

9596
help:
9697
@echo "This Makefile adds targets to build runners for various models on various backends. Run using \`make <target>\`. Available targets:"
9798
@echo " voxtral-cuda - Build Voxtral runner with CUDA backend"
9899
@echo " voxtral-cpu - Build Voxtral runner with CPU backend"
99100
@echo " voxtral-metal - Build Voxtral runner with Metal backend (macOS only)"
101+
@echo " voxtral_realtime-cpu - Build Voxtral Realtime runner with CPU backend"
100102
@echo " whisper-cuda - Build Whisper runner with CUDA backend"
101103
@echo " whisper-cuda-debug - Build Whisper runner with CUDA backend (debug mode)"
102104
@echo " whisper-cpu - Build Whisper runner with CPU backend"
@@ -221,6 +223,15 @@ sortformer-cpu:
221223
@echo "✓ Build complete!"
222224
@echo " Binary: cmake-out/examples/models/sortformer/sortformer_runner"
223225

226+
voxtral_realtime-cpu:
227+
@echo "==> Building and installing ExecuTorch..."
228+
cmake --workflow --preset llm-release
229+
@echo "==> Building Voxtral Realtime runner (CPU)..."
230+
cd examples/models/voxtral_realtime && cmake --workflow --preset voxtral-realtime-cpu
231+
@echo ""
232+
@echo "✓ Build complete!"
233+
@echo " Binary: cmake-out/examples/models/voxtral_realtime/voxtral_realtime_runner"
234+
224235
silero-vad-cpu:
225236
@echo "==> Building and installing ExecuTorch..."
226237
cmake --workflow --preset llm-release
Lines changed: 5 additions & 147 deletions
Original file line numberDiff line numberDiff line change
@@ -1,150 +1,8 @@
1-
"""Quantization utilities for Parakeet model export."""
1+
"""Quantization utilities for Parakeet model export.
22
3-
from typing import Optional
3+
Re-exports quantize_model_ from the shared ExecuTorch LLM export library.
4+
"""
45

5-
import torch
6+
from executorch.extension.llm.export.quantize import quantize_model_
67

7-
8-
def quantize_model_( # noqa: C901
9-
module: torch.nn.Module,
10-
qlinear_config: Optional[str] = None,
11-
qlinear_group_size: int = 32,
12-
qlinear_packing_format: Optional[str] = None,
13-
qembedding_config: Optional[str] = None,
14-
qembedding_group_size: int = 0,
15-
) -> None:
16-
"""Quantize linear and embedding layers in a module in-place.
17-
18-
Args:
19-
module: The PyTorch module to quantize.
20-
qlinear_config: Quantization config for linear layers ("4w", "8w", "8da4w", "8da8w", "fpa4w").
21-
qlinear_group_size: Group size for linear quantization (default: 32).
22-
qlinear_packing_format: Packing format for linear layers (e.g., "tile_packed_to_4d").
23-
qembedding_config: Quantization config for embedding layers ("4w", "8w").
24-
qembedding_group_size: Group size for embedding quantization (default: 0 = per-axis).
25-
"""
26-
if not qlinear_config and not qembedding_config:
27-
return
28-
29-
from torchao.quantization.quant_api import quantize_
30-
31-
# Metal (MPS) quantization uses different API
32-
if qlinear_config == "fpa4w":
33-
# Load MPS ops
34-
import torchao.experimental.ops.mps # noqa: F401
35-
from torchao.experimental.quant_api import UIntxWeightOnlyConfig
36-
37-
config = UIntxWeightOnlyConfig(
38-
group_size=qlinear_group_size,
39-
bitwidth=4,
40-
uintx_choose_qparams_algorithm="hqq",
41-
)
42-
43-
def linear_filter(m, fqn):
44-
if isinstance(m, torch.nn.Linear):
45-
if m.weight.shape[1] % qlinear_group_size != 0:
46-
raise ValueError(
47-
f"Metal int4 quantization requires weight dimension (K) to be multiple of group_size. "
48-
f"Layer {fqn} has weight shape {m.weight.shape} (K={m.weight.shape[1]}, group_size={qlinear_group_size})" # noqa: E501
49-
)
50-
return True
51-
return False
52-
53-
print(
54-
f" Applying {qlinear_config} linear quantization "
55-
f"(group_size={qlinear_group_size})..."
56-
)
57-
quantize_(module, config, filter_fn=linear_filter)
58-
return
59-
60-
from torchao.quantization.granularity import PerAxis, PerGroup
61-
from torchao.quantization.quant_api import (
62-
Int4WeightOnlyConfig,
63-
Int8DynamicActivationIntxWeightConfig,
64-
IntxWeightOnlyConfig,
65-
)
66-
67-
# Quantize embedding layers first
68-
if qembedding_config:
69-
if qembedding_group_size == 0:
70-
embedding_granularity = PerAxis(0)
71-
else:
72-
assert (
73-
qembedding_group_size % 2 == 0
74-
), "Embedding group size must be a multiple of 2."
75-
embedding_granularity = PerGroup(qembedding_group_size)
76-
77-
embedding_config = IntxWeightOnlyConfig(
78-
weight_dtype=torch.int4 if qembedding_config == "4w" else torch.int8,
79-
granularity=embedding_granularity,
80-
)
81-
82-
print(
83-
f" Applying {qembedding_config} embedding quantization "
84-
f"(group_size={qembedding_group_size})..."
85-
)
86-
quantize_(
87-
module,
88-
embedding_config,
89-
lambda m, fqn: isinstance(m, torch.nn.Embedding),
90-
)
91-
92-
# Quantize linear layers
93-
if qlinear_config:
94-
# Determine granularity
95-
if qlinear_group_size == 0:
96-
granularity = PerAxis(0)
97-
else:
98-
granularity = PerGroup(qlinear_group_size)
99-
100-
# Build quantization config
101-
if qlinear_config == "4w":
102-
if qlinear_packing_format:
103-
config = Int4WeightOnlyConfig(
104-
group_size=qlinear_group_size,
105-
int4_packing_format=qlinear_packing_format,
106-
int4_choose_qparams_algorithm="hqq",
107-
)
108-
else:
109-
config = IntxWeightOnlyConfig(
110-
weight_dtype=torch.int4,
111-
granularity=granularity,
112-
)
113-
elif qlinear_config == "8w":
114-
config = IntxWeightOnlyConfig(
115-
weight_dtype=torch.int8,
116-
granularity=granularity,
117-
)
118-
elif qlinear_config == "8da4w":
119-
config = Int8DynamicActivationIntxWeightConfig(
120-
weight_dtype=torch.int4,
121-
weight_granularity=granularity,
122-
intx_choose_qparams_algorithm="hqq_scale_only",
123-
)
124-
elif qlinear_config == "8da8w":
125-
config = Int8DynamicActivationIntxWeightConfig(
126-
weight_dtype=torch.int8,
127-
weight_granularity=PerAxis(0),
128-
)
129-
else:
130-
raise ValueError(f"Unsupported qlinear_config: {qlinear_config}")
131-
132-
# Filter: only quantize Linear layers with compatible dimensions
133-
def linear_filter(m, fqn):
134-
if isinstance(m, torch.nn.Linear):
135-
if qlinear_group_size == 0:
136-
return True
137-
if m.weight.shape[1] % qlinear_group_size != 0:
138-
print(
139-
f" Skipping {fqn}: weight shape {m.weight.shape} "
140-
f"incompatible with group_size={qlinear_group_size}"
141-
)
142-
return False
143-
return True
144-
return False
145-
146-
print(
147-
f" Applying {qlinear_config} linear quantization "
148-
f"(group_size={qlinear_group_size}, packing={qlinear_packing_format})..."
149-
)
150-
quantize_(module, config, filter_fn=linear_filter)
8+
__all__ = ["quantize_model_"]

0 commit comments

Comments
 (0)