|
1 | | -# TensorFlow Lite Micro (TFLM) Architecture |
| 1 | +# TensorFlow Lite Micro (TFLM) & Wake-on-Voice (WoV) Architecture |
2 | 2 |
|
3 | | -This directory acts as the bridge for running ML models. |
| 3 | +This directory provides the TensorFlow Lite for Microcontrollers (TFLM) classification module (`TFLMCLY`) for Sound Open Firmware (SOF), including integration with MFCC feature extraction, mtrace logging, IPC host notifications, and Key Phrase Buffer (KPB) Wake-on-Voice (WoV) trigger infrastructure. |
| 4 | + |
| 5 | +--- |
4 | 6 |
|
5 | 7 | ## Overview |
6 | 8 |
|
7 | | -Integrates TensorFlow Lite for Microcontrollers into the SOF audio pipeline. Evaluates pre-trained neural network topologies inline with the audio stream for tasks like wake-word, noise cancellation, or sound classification. |
| 9 | +The TFLM module evaluates pre-trained micro speech neural network models inline within the SOF audio processing graph. It receives pre-processed audio feature tensors (e.g. 40-bin mel spectrograms from the MFCC component), runs model inference, logs keyword detections to `mtrace`, issues IPC4 notifications to the host audio driver, and signals the KPB module to drain buffered pre-keyword audio for Wake-on-Voice. |
| 10 | + |
| 11 | +--- |
| 12 | + |
| 13 | +## Architecture & Data Flow |
8 | 14 |
|
9 | | -## Architecture Diagram |
| 15 | +### Dual-Path Wake-on-Voice (WoV) Architecture |
| 16 | + |
| 17 | +To allow continuous keyword evaluation without streaming audio to the host until a keyword is detected, the pipeline separates real-time keyword detection from host PCM draining via KPB: |
10 | 18 |
|
11 | 19 | ```mermaid |
12 | | -graph LR |
13 | | - Feat[Audio Features] --> TFLM[TFLM Inference Engine] |
14 | | - SubGraph[FlatBuffer Model] -.-> TFLM |
15 | | - TFLM --> Out[Inference Labels/Scores] |
| 20 | +graph TD |
| 21 | + DAI[HDA Mic DAI] --> Gain[Gain Component] |
| 22 | + Gain --> KPB[KPB Buffer Module] |
| 23 | +
|
| 24 | + subgraph "Real-Time Detection Path (KPB Pin 1)" |
| 25 | + KPB -- Live Audio Stream --> SRC[SRC: 48kHz -> 16kHz] |
| 26 | + SRC --> MFCC[MFCC Feature Extractor] |
| 27 | + MFCC -- 40-bin Mel Tensors --> TFLM[TFLM Classifier: tflmcly] |
| 28 | + TFLM --> VSink[Virtual Sink: virtual.tflm_sink] |
| 29 | + end |
| 30 | +
|
| 31 | + subgraph "Host Draining Path (KPB Pin 2)" |
| 32 | + KPB -- History Draining Stream --> Host[Host Copier: PCM Capture] |
| 33 | + end |
| 34 | +
|
| 35 | + TFLM -- "1. Log to mtrace (comp_info)" --> MTrace[mtrace / SOF Trace Log] |
| 36 | + TFLM -- "2. IPC4 Host Notification" --> IPC[Host Audio Driver] |
| 37 | + TFLM -- "3. KPB_EVENT_BEGIN_DRAINING (notifier_event)" --> KPB |
| 38 | +``` |
| 39 | + |
| 40 | +--- |
| 41 | + |
| 42 | +## Pipeline Execution & Event Flow |
| 43 | + |
| 44 | +1. **Continuous Real-Time Listening**: |
| 45 | + - Live microphone audio is captured by the HDA DAI and passed into `KPB` (`kpb.2.1`). |
| 46 | + - KPB Output Pin 1 continuously streams audio to `SRC` (resampling 48kHz $\to$ 16kHz), `MFCC` (generating 40-bin `int8_t` mel spectrogram features), and `TFLM` (`tflmcly.1.1`). |
| 47 | + - KPB stores the raw PCM audio continuously in its circular history buffer (e.g. 2100ms – 3000ms history). |
| 48 | + |
| 49 | +2. **Inference & Keyword Detection**: |
| 50 | + - `tflm_process()` feeds 1960-byte feature tensors ($40 \text{ features} \times 49 \text{ windows}$) into the Micro Speech TFLM interpreter (`[1, 49, 40]` `int8_t` input tensor). |
| 51 | + - Upon `TF_ProcessClassify()`, predictions are evaluated across categories: |
| 52 | + - `0`: `silence` |
| 53 | + - `1`: `unknown` |
| 54 | + - `2`: `yes` (Keyword) |
| 55 | + - `3`: `no` (Keyword) |
| 56 | + |
| 57 | +3. **Wake-on-Voice Trigger & Notification**: |
| 58 | + When a keyword (`yes` or `no`) is detected with $\ge 0.70$ confidence: |
| 59 | + - **mtrace Logging**: Logs detection with keyword label and confidence score via `comp_info()`: |
| 60 | + `"TFLM keyword detected: yes (confidence 0.852)"` |
| 61 | + - **Host IPC Notification**: Sends an IPC4 module notification (`SOF_IPC4_MODULE_NOTIFICATION`) to inform the host driver. |
| 62 | + - **KPB Draining Signal**: Fires a system notification event (`NOTIFIER_ID_KPB_CLIENT_EVT`) with `KPB_EVENT_BEGIN_DRAINING`. |
| 63 | + - **Host PCM Draining**: KPB opens Output Pin 2 to `host-copier`, draining pre-keyword history buffer audio followed by live mic audio to the host capture stream. |
| 64 | + |
| 65 | +--- |
| 66 | + |
| 67 | +## Topology v2 Integration & Usage |
| 68 | + |
| 69 | +### 1. Component Widget (`include/components/tflm.conf`) |
| 70 | + |
| 71 | +Defines `Class.Widget."tflmcly"`: |
| 72 | +- **UUID**: `42:c6:1d:c5:e1:a2:df:48:a4:90:e2:74:8c:b6:36:3e` (`c51dc642-a2e1-48df-a490e2748cb6363e`) |
| 73 | +- **Type**: `effect` |
| 74 | + |
| 75 | +### 2. Detection Pipeline Template (`include/pipelines/cavs/host-gateway-src-mfcc-tflm-capture.conf`) |
| 76 | + |
| 77 | +Instantiates the real-time detection graph: |
| 78 | +```conf |
| 79 | +Object.Widget { |
| 80 | + virtual."1" { name "virtual.tflm_sink" } |
| 81 | + src."1" { ... } |
| 82 | + mfcc."1" { ... } |
| 83 | + tflmcly."1" { ... } |
| 84 | +} |
16 | 85 | ``` |
17 | 86 |
|
18 | | -## Configuration and Scripts |
| 87 | +### 3. Top-Level Topology Configuration (`sof-hda-tflm.conf`) |
| 88 | + |
| 89 | +Instantiates the complete HDA Mic WoV topology with dual-path KPB routing: |
| 90 | +```conf |
| 91 | +Object.Base.route [ |
| 92 | + # DAI -> Gain -> KPB |
| 93 | + { source "dai-copier.HDA.Analog.capture"; sink "gain.2.1" } |
| 94 | + { source "gain.2.1"; sink "kpb.2.1" } |
| 95 | +
|
| 96 | + # KPB Pin 1 -> Real-time Detection Path |
| 97 | + { source "kpb.2.1"; sink "src.1.1" } |
| 98 | +
|
| 99 | + # KPB Pin 2 -> Host WoV Draining Path |
| 100 | + { source "kpb.2.1"; sink "host-copier.0.capture" } |
| 101 | +] |
| 102 | +``` |
| 103 | + |
| 104 | +--- |
| 105 | + |
| 106 | +## Building and Testing Topologies |
| 107 | + |
| 108 | +### Pre-processing and Compiling with `alsatplg` |
| 109 | + |
| 110 | +Build `sof-hda-tflm.tplg` using the Topology v2 pre-processor: |
| 111 | + |
| 112 | +```bash |
| 113 | +ALSA_CONFIG_DIR=tools/topology/topology2 \ |
| 114 | + tools/bin/alsatplg \ |
| 115 | + -I tools/topology/topology2/ \ |
| 116 | + -p -c tools/topology/topology2/sof-hda-tflm.conf \ |
| 117 | + -o build/sof-hda-tflm.tplg |
| 118 | +``` |
| 119 | + |
| 120 | +### Inspecting Decoded Topology Graphs |
| 121 | + |
| 122 | +Decode and verify the compiled `.tplg` binary: |
| 123 | + |
| 124 | +```bash |
| 125 | +tools/bin/alsatplg -d build/sof-hda-tflm.tplg -o decoded.txt |
| 126 | +grep -A 20 "SectionGraph" decoded.txt |
| 127 | +``` |
| 128 | + |
| 129 | +Expected graph output: |
| 130 | +```text |
| 131 | +SectionGraph { |
| 132 | + set0 { |
| 133 | + gain.2.1 <- dai-copier.HDA.Analog.capture |
| 134 | + kpb.2.1 <- gain.2.1 |
| 135 | + src.1.1 <- kpb.2.1 (Real-Time Detection Path) |
| 136 | + host-copier.0.capture <- kpb.2.1 (Host WoV Draining Path) |
| 137 | + } |
| 138 | + set1 { |
| 139 | + mfcc.1.1 <- src.1.1 |
| 140 | + tflmcly.1.1 <- mfcc.1.1 |
| 141 | + virtual.tflm_sink <- tflmcly.1.1 (Detection Sink Termination) |
| 142 | + } |
| 143 | +} |
| 144 | +``` |
| 145 | + |
| 146 | +--- |
| 147 | + |
| 148 | +## Source Files |
19 | 149 |
|
20 | | -- **Kconfig**: Enforces requirements for C++17 support and core framework staging logic (`COMP_TENSORFLOW`). |
21 | | -- **CMakeLists.txt**: An intricate build specification linking the Tensilica neural network library block computations (`nn_hifi_lib`) and the TensorFlow Lite micro core engine (`tflm_lib`). Also hooks the `tflm-classify.c` SOF adapter via compiler flags explicitly enforcing memory, precision, and XTENSA optimizations. |
22 | | -- **tflmcly.toml**: Topology definition for the specific TFLM Classifier implementation binding the engine against the UUID `UUIDREG_STR_TFLMCLY`. |
| 150 | +- **[tflm-classify.c](file:///home/lrg/work/sof-ptl/sof/src/audio/tensorflow/tflm-classify.c)**: SOF module adapter implementation for TFLM. |
| 151 | +- **[speech.cc](file:///home/lrg/work/sof-ptl/sof/src/audio/tensorflow/speech.cc)** / **[speech.h](file:///home/lrg/work/sof-ptl/sof/src/audio/tensorflow/speech.h)**: TFLM C++ API bridge & micro speech tensor wrapper. |
| 152 | +- **[tflm.conf](file:///home/lrg/work/sof-ptl/sof/tools/topology/topology2/include/components/tflm.conf)**: Topology v2 widget class definition. |
| 153 | +- **[host-gateway-src-mfcc-tflm-capture.conf](file:///home/lrg/work/sof-ptl/sof/tools/topology/topology2/include/pipelines/cavs/host-gateway-src-mfcc-tflm-capture.conf)**: Detection pipeline template. |
| 154 | +- **[sof-hda-tflm.conf](file:///home/lrg/work/sof-ptl/sof/tools/topology/topology2/sof-hda-tflm.conf)**: Top-level WoV topology configuration. |
0 commit comments