speech_recognition

Automatic Speech Recognition Samples with LiteRT

This directory contains a collection of samples, tools, and applications demonstrating how to utilize the LiteRT (formerly TensorFlow Lite) runtime to execute state-of-the-art, open-weight Automatic Speech Recognition (ASR) models on device, leveraging hardware acceleration (CPU, GPU, TPU, NPU).

Supported Models

The tools and applications in this directory support several popular open-weight ASR models. The table below outlines the supported backends for each model:

Model	CPU	GPU	TPU/NPU	Description
Parakeet TDT	✅	✅	Pixel 10	Nvidia's Transducer-Duration-Transducer model
Parakeet CTC	✅	✅	Galaxy S23/24	Nvidia's Connectionist-Temporal-Classification model
Moonshine	✅	✅		A lightweight, low-latency autoregressive ASR model
Whisper	✅	✅		OpenAI's robust multilingual ASR and translation model
Qwen3-ASR	✅			Robust multimodal/speech language model capability

Directory Structure & Components

The workspace is organized into three main components:

samples/asr/
├── AndroidApp/       # Android demo application (Java/Kotlin & LiteRT SDK)
└── convert/          # Python conversion and verification pipelines

1. AndroidApp

A Gradle-based Android demo application demonstrating how to integrate the LiteRT Android SDK.

• Model Support: On-device execution of supported ASR models.
• Accelerators: Hardware accelerator delegation to execute models on CPU, GPU, TPU (Google Tensor) and NPU (Qualcomm Snapdragon, MediaTek MTK).
• Feature Processing & Decoding: Implements spectrogram audio feature extractors, custom decoders (e.g., CtcDecoder, TdtDecoder), and a JNI Hugging Face tokenizer.

Notes:

• NPU runtime libs are commented out in AndroidApp/app/build.gradle.kts. Uncomment one of them to enable TPU/NPU support.

2. convert

A suite of Python utilities to prepare, convert, and verify PyTorch and Hugging Face model weights for LiteRT inference on edge devices.

• convert_to_tflite.py: Converts PyTorch models to stateful or stateless .tflite format, supporting Dynamic Range Quantization (DRQ).
• compile_for_npu.py: Compiles converted .tflite models into NPU binaries targeting Qualcomm Snapdragon/QNN or MediaTek MTK processors.
• verify_model.py: Validates re-authored PyTorch models against reference Hugging Face models with sample wav inputs.
• verify_tflite.py: Verifies local .tflite execution against the reference PyTorch model.

Technical Details

1. Pipeline

The sample app builds a pipeline to reuse many components in processing audio, recognizing speech and decoding tokens with multiple models and different configurations.

The pipeline consists of:

1. AudioSource that reads audio data either from file or from microphone
2. AudioPreprocessor that converts audio to log MelSpectrogram or does nothing
3. SpeechRecoginizer that runs tflite model with LiteRT and decoders; Default, TDT, CTC
4. Postprocessor that decodes tokens with Huggingface tokenizer, then merges texts in consecutive sequences

2. Sliding window for audio chunks

None of the models supported currently are streaming models. Instead, models get audio data with overlapped windows.

When the audio data is read from a file, each audio chunk is 5 seconds with 2 seconds overlapped with previous and next chunks. The last chunk may be padded with zeros as silence.

When the audio data is captured from the microphone, each audio chunk is 5 seconds with 4 seconds overlapped with previous and next chunks. The first 4 chunks are pre-padded with zeros as silence which is to show texts as early as possible.

3. Audio preprocessing

Except moonshine which gets raw audio as input, all models take Mel-Spectrogram data with small different configurations like number of filter banks. The sample app defines LogMelSpectroConfig in ModelMetadataConfig to capture these configurations.

The sample app utilizes jLibrosa for MelSpectrogram processing.

4. Downloading models from litert-community in Huggingface

To make it simple, all the models are pre-converted and AOT-compiled for NPUs and uploaded to litert-community in Huggingface. For production app, it would be better to utilize AI packs to distribute models.

5. Zero copy to transfer data

Many models except Parakeet CTC are converted to multiple subgraphs to encode audio and/then decode tokens. Encoder passes data encoded as hidden states to decoder. The sample app passes data with zero-copy by passing buffer pointer instead of data content in LiteRtRunner

val decodeInputBuffers =
  buildList<TensorBuffer> {
    // Avoid copy. Build a list of TensorBuffers directly from the encoder output buffers.
    addAll(encodeOutputBuffers)
    // Add the decoder input buffers at the end of the list.
    add(tokenIdsBuffer)
    add(maskBuffer)
  }

Parakeet-TDT caches the LSTM internal states to avoid decode the same token over and over. Passing the previous internal states to the next decode step also does zero copy in TdtDecoder

private fun getInputBuffers(
  encodeOutputBuffers: List<TensorBuffer>,
  tokenIdsBuffer: TensorBuffer,
) =
  buildList<TensorBuffer> {
    // Avoid copy. Build a list of TensorBuffers directly from the encoder output buffers.
    addAll(encodeOutputBuffers)
    add(tokenIdsBuffer)
    addAll(statesBuffers[inputStatesBuffersIndex])
  }

6. Stateful vs Stateless decoding

Parakeet TDT decodes tokens statefully which means the decoder gets the internal LSTM's states of the previous token as input instead of all tokens decoded so far.

On the other hand, Moonshine, Whisper, Qwen3-ASR decode tokens statelessy which means the decoder gets all tokens decoded so far as input. Though it is for the simplicity, the number of tokens decoded from one audio chunk is likely less than 20 which would not be much overhead relatively because Moonshine and Whisper does cross-attention with encoder's states as encoder-decoder models.

Qwen3-ASR is a bit different as it is a multimodal LM which gets encoded audio embeddings as a part of tokens for self attention, and/so the number of tokens is much longer than Moonshine or Whisper. In this case, KV cache must be helpful to improve the latency. Though the sample app uses LiteRT API directly, LiteRT-LM would be better API as it already support multimodality and efficient KV cache management.

7. Merging texts in consecutive sequences

As audio chunks are overlapped, the tokens detected by SpeechRecognizer are also duplicated. To remove duplicates from consecutive token sequences, tokens in a sequence should be aligned to ones in the previous sequence.

If SpeechRecognizer outputs timestamps along with tokens, like Parakeet-TDT and Parakeet-CTC, the alignment is straightforward.

If timestamps are not output, a heuristic based on Levenshtein edit distance and the number of words/tokens are used to figure out the best alignment.

Once aligned, words are merged. Assuming that the words close to the start or the end of the sequence are less correct than others, first N words are from the previous sequence while last M words are from the next sequence.