Skip to content

Commit df47967

Browse files
committed
docs: tensorflow: add comprehensive README for TFLM WoV architecture and topology usage
1 parent a6c639c commit df47967

1 file changed

Lines changed: 144 additions & 12 deletions

File tree

src/audio/tensorflow/README.md

Lines changed: 144 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,22 +1,154 @@
1-
# TensorFlow Lite Micro (TFLM) Architecture
1+
# TensorFlow Lite Micro (TFLM) & Wake-on-Voice (WoV) Architecture
22

3-
This directory acts as the bridge for running ML models.
3+
This directory provides the TensorFlow Lite for Microcontrollers (TFLM) classification module (`TFLMCLY`) for Sound Open Firmware (SOF), including integration with MFCC feature extraction, mtrace logging, IPC host notifications, and Key Phrase Buffer (KPB) Wake-on-Voice (WoV) trigger infrastructure.
4+
5+
---
46

57
## Overview
68

7-
Integrates TensorFlow Lite for Microcontrollers into the SOF audio pipeline. Evaluates pre-trained neural network topologies inline with the audio stream for tasks like wake-word, noise cancellation, or sound classification.
9+
The TFLM module evaluates pre-trained micro speech neural network models inline within the SOF audio processing graph. It receives pre-processed audio feature tensors (e.g. 40-bin mel spectrograms from the MFCC component), runs model inference, logs keyword detections to `mtrace`, issues IPC4 notifications to the host audio driver, and signals the KPB module to drain buffered pre-keyword audio for Wake-on-Voice.
10+
11+
---
12+
13+
## Architecture & Data Flow
814

9-
## Architecture Diagram
15+
### Dual-Path Wake-on-Voice (WoV) Architecture
16+
17+
To allow continuous keyword evaluation without streaming audio to the host until a keyword is detected, the pipeline separates real-time keyword detection from host PCM draining via KPB:
1018

1119
```mermaid
12-
graph LR
13-
Feat[Audio Features] --> TFLM[TFLM Inference Engine]
14-
SubGraph[FlatBuffer Model] -.-> TFLM
15-
TFLM --> Out[Inference Labels/Scores]
20+
graph TD
21+
DAI[HDA Mic DAI] --> Gain[Gain Component]
22+
Gain --> KPB[KPB Buffer Module]
23+
24+
subgraph "Real-Time Detection Path (KPB Pin 1)"
25+
KPB -- Live Audio Stream --> SRC[SRC: 48kHz -> 16kHz]
26+
SRC --> MFCC[MFCC Feature Extractor]
27+
MFCC -- 40-bin Mel Tensors --> TFLM[TFLM Classifier: tflmcly]
28+
TFLM --> VSink[Virtual Sink: virtual.tflm_sink]
29+
end
30+
31+
subgraph "Host Draining Path (KPB Pin 2)"
32+
KPB -- History Draining Stream --> Host[Host Copier: PCM Capture]
33+
end
34+
35+
TFLM -- "1. Log to mtrace (comp_info)" --> MTrace[mtrace / SOF Trace Log]
36+
TFLM -- "2. IPC4 Host Notification" --> IPC[Host Audio Driver]
37+
TFLM -- "3. KPB_EVENT_BEGIN_DRAINING (notifier_event)" --> KPB
38+
```
39+
40+
---
41+
42+
## Pipeline Execution & Event Flow
43+
44+
1. **Continuous Real-Time Listening**:
45+
- Live microphone audio is captured by the HDA DAI and passed into `KPB` (`kpb.2.1`).
46+
- KPB Output Pin 1 continuously streams audio to `SRC` (resampling 48kHz $\to$ 16kHz), `MFCC` (generating 40-bin `int8_t` mel spectrogram features), and `TFLM` (`tflmcly.1.1`).
47+
- KPB stores the raw PCM audio continuously in its circular history buffer (e.g. 2100ms – 3000ms history).
48+
49+
2. **Inference & Keyword Detection**:
50+
- `tflm_process()` feeds 1960-byte feature tensors ($40 \text{ features} \times 49 \text{ windows}$) into the Micro Speech TFLM interpreter (`[1, 49, 40]` `int8_t` input tensor).
51+
- Upon `TF_ProcessClassify()`, predictions are evaluated across categories:
52+
- `0`: `silence`
53+
- `1`: `unknown`
54+
- `2`: `yes` (Keyword)
55+
- `3`: `no` (Keyword)
56+
57+
3. **Wake-on-Voice Trigger & Notification**:
58+
When a keyword (`yes` or `no`) is detected with $\ge 0.70$ confidence:
59+
- **mtrace Logging**: Logs detection with keyword label and confidence score via `comp_info()`:
60+
`"TFLM keyword detected: yes (confidence 0.852)"`
61+
- **Host IPC Notification**: Sends an IPC4 module notification (`SOF_IPC4_MODULE_NOTIFICATION`) to inform the host driver.
62+
- **KPB Draining Signal**: Fires a system notification event (`NOTIFIER_ID_KPB_CLIENT_EVT`) with `KPB_EVENT_BEGIN_DRAINING`.
63+
- **Host PCM Draining**: KPB opens Output Pin 2 to `host-copier`, draining pre-keyword history buffer audio followed by live mic audio to the host capture stream.
64+
65+
---
66+
67+
## Topology v2 Integration & Usage
68+
69+
### 1. Component Widget (`include/components/tflm.conf`)
70+
71+
Defines `Class.Widget."tflmcly"`:
72+
- **UUID**: `42:c6:1d:c5:e1:a2:df:48:a4:90:e2:74:8c:b6:36:3e` (`c51dc642-a2e1-48df-a490e2748cb6363e`)
73+
- **Type**: `effect`
74+
75+
### 2. Detection Pipeline Template (`include/pipelines/cavs/host-gateway-src-mfcc-tflm-capture.conf`)
76+
77+
Instantiates the real-time detection graph:
78+
```conf
79+
Object.Widget {
80+
virtual."1" { name "virtual.tflm_sink" }
81+
src."1" { ... }
82+
mfcc."1" { ... }
83+
tflmcly."1" { ... }
84+
}
1685
```
1786

18-
## Configuration and Scripts
87+
### 3. Top-Level Topology Configuration (`sof-hda-tflm.conf`)
88+
89+
Instantiates the complete HDA Mic WoV topology with dual-path KPB routing:
90+
```conf
91+
Object.Base.route [
92+
# DAI -> Gain -> KPB
93+
{ source "dai-copier.HDA.Analog.capture"; sink "gain.2.1" }
94+
{ source "gain.2.1"; sink "kpb.2.1" }
95+
96+
# KPB Pin 1 -> Real-time Detection Path
97+
{ source "kpb.2.1"; sink "src.1.1" }
98+
99+
# KPB Pin 2 -> Host WoV Draining Path
100+
{ source "kpb.2.1"; sink "host-copier.0.capture" }
101+
]
102+
```
103+
104+
---
105+
106+
## Building and Testing Topologies
107+
108+
### Pre-processing and Compiling with `alsatplg`
109+
110+
Build `sof-hda-tflm.tplg` using the Topology v2 pre-processor:
111+
112+
```bash
113+
ALSA_CONFIG_DIR=tools/topology/topology2 \
114+
tools/bin/alsatplg \
115+
-I tools/topology/topology2/ \
116+
-p -c tools/topology/topology2/sof-hda-tflm.conf \
117+
-o build/sof-hda-tflm.tplg
118+
```
119+
120+
### Inspecting Decoded Topology Graphs
121+
122+
Decode and verify the compiled `.tplg` binary:
123+
124+
```bash
125+
tools/bin/alsatplg -d build/sof-hda-tflm.tplg -o decoded.txt
126+
grep -A 20 "SectionGraph" decoded.txt
127+
```
128+
129+
Expected graph output:
130+
```text
131+
SectionGraph {
132+
set0 {
133+
gain.2.1 <- dai-copier.HDA.Analog.capture
134+
kpb.2.1 <- gain.2.1
135+
src.1.1 <- kpb.2.1 (Real-Time Detection Path)
136+
host-copier.0.capture <- kpb.2.1 (Host WoV Draining Path)
137+
}
138+
set1 {
139+
mfcc.1.1 <- src.1.1
140+
tflmcly.1.1 <- mfcc.1.1
141+
virtual.tflm_sink <- tflmcly.1.1 (Detection Sink Termination)
142+
}
143+
}
144+
```
145+
146+
---
147+
148+
## Source Files
19149

20-
- **Kconfig**: Enforces requirements for C++17 support and core framework staging logic (`COMP_TENSORFLOW`).
21-
- **CMakeLists.txt**: An intricate build specification linking the Tensilica neural network library block computations (`nn_hifi_lib`) and the TensorFlow Lite micro core engine (`tflm_lib`). Also hooks the `tflm-classify.c` SOF adapter via compiler flags explicitly enforcing memory, precision, and XTENSA optimizations.
22-
- **tflmcly.toml**: Topology definition for the specific TFLM Classifier implementation binding the engine against the UUID `UUIDREG_STR_TFLMCLY`.
150+
- **[tflm-classify.c](file:///home/lrg/work/sof-ptl/sof/src/audio/tensorflow/tflm-classify.c)**: SOF module adapter implementation for TFLM.
151+
- **[speech.cc](file:///home/lrg/work/sof-ptl/sof/src/audio/tensorflow/speech.cc)** / **[speech.h](file:///home/lrg/work/sof-ptl/sof/src/audio/tensorflow/speech.h)**: TFLM C++ API bridge & micro speech tensor wrapper.
152+
- **[tflm.conf](file:///home/lrg/work/sof-ptl/sof/tools/topology/topology2/include/components/tflm.conf)**: Topology v2 widget class definition.
153+
- **[host-gateway-src-mfcc-tflm-capture.conf](file:///home/lrg/work/sof-ptl/sof/tools/topology/topology2/include/pipelines/cavs/host-gateway-src-mfcc-tflm-capture.conf)**: Detection pipeline template.
154+
- **[sof-hda-tflm.conf](file:///home/lrg/work/sof-ptl/sof/tools/topology/topology2/sof-hda-tflm.conf)**: Top-level WoV topology configuration.

0 commit comments

Comments
 (0)