Skip to content

Commit e9da0d5

Browse files
lrgirdwolrgirdwo
committed
docs: add detailed architecture, build, and usage READMEs for ffmpeg and webrtc modules
Create and update README.md documentation for the FFmpeg decoder/encoder/filter module and all four WebRTC audio modules: - ffmpeg_dec/README.md: Added Mermaid architecture diagram, detailed decoder/encoder/filter configurations, build choices, and topology guides. - webrtc_vad/README.md: Included Mermaid data flow, GMM classification details, ring-buffer accumulation behavior, and notification topology mappings. - webrtc_ns/README.md: Documented fixed-point spectral Wiener filter complexity, channel separation, and Kconfig rules. - webrtc_aec/README.md: Described dual-input AECm echo cancellation routing, pipeline-ID heuristics, and layout designs. - webrtc_ns2/README.md: Documented deep-learning/RNNoise GRU topology, 48 kHz lock rules, VAD probability thresholds, and DSP scaling metrics. Each document contains a tailored Mermaid diagram mapping data/control flows. Signed-off-by: Liam Girdwood <liam.r.girdwood@linux.intel.com>
1 parent 06e1cfc commit e9da0d5

5 files changed

Lines changed: 463 additions & 126 deletions

File tree

‎src/audio/ffmpeg_dec/README.md‎

Lines changed: 100 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -1,59 +1,100 @@
1-
# FFmpeg audio decoder module (ffmpeg_dec)
2-
3-
Wraps FFmpeg's `libavcodec` audio decoders behind the SOF `module_interface`,
4-
decoding a compressed elementary stream to PCM inside the DSP. First target
5-
codec is **FLAC**.
6-
7-
## Design
8-
9-
The SOF glue and the decoder are separated so the integration can be validated
10-
without libavcodec:
11-
12-
- `ffmpeg_dec.c` — SOF module core: `init` / `prepare` / `process_raw_data` /
13-
`set_configuration` / `reset` / `free`, plus the LLEXT manifest. Input is the
14-
compressed byte stream (raw data), output is interleaved PCM. Decoding is
15-
delegated to a `struct ffmpeg_dec_backend`.
16-
- `ffmpeg_dec-stub.c` — dependency-free passthrough backend. Lets CI build and
17-
exercise the module (pipeline, IPC config, LLEXT packaging) with **no** FFmpeg
18-
libraries. Selected by `CONFIG_COMP_FFMPEG_DEC_STUB`.
19-
- `ffmpeg_dec-ffmpeg.c` — real backend driving the standalone `libavcodec`
20-
send-packet / receive-frame API. A raw stream is framed on-DSP with an
21-
`AVCodecParser` (`av_parser_parse2`); decode is forced single-threaded
22-
(`thread_count = 1`). Requires pre-compiled static libraries and headers under
23-
`third_party/` (see below).
24-
- `ffmpeg_dec.h` — private data (`struct ffmpeg_dec_comp_data`) and the backend
25-
interface.
26-
27-
The codec setup header (e.g. FLAC STREAMINFO) is delivered as a binary control
28-
via `set_configuration` and stored as `extradata`, which the libavcodec backend
29-
installs before `avcodec_open2()`.
30-
31-
## Build
32-
33-
- **Stub (default for testing/CI):** `CONFIG_COMP_FFMPEG_DEC=y` (or `=m` for
34-
LLEXT) with `CONFIG_COMP_FFMPEG_DEC_STUB=y`. No external dependencies.
35-
- **Real decoder:** `CONFIG_COMP_FFMPEG_DEC=m`, `CONFIG_COMP_FFMPEG_DEC_STUB=n`.
36-
Requires cross-compiled decoder-only FFmpeg static libraries installed as:
37-
- `third_party/lib/libavcodec.a`, `libavutil.a`, `libswresample.a`
38-
- `third_party/include/libavcodec/…`, `libavutil/…`, `libswresample/…`
39-
Configure FFmpeg with `--disable-everything --disable-avformat
40-
--disable-pthreads --enable-decoder=flac --enable-parser=flac` (plus the
41-
target cross-compile flags).
42-
43-
## Files
44-
45-
- `Kconfig` — `COMP_FFMPEG_DEC` (tristate) and `COMP_FFMPEG_DEC_STUB`.
46-
- `CMakeLists.txt` / `llext/CMakeLists.txt` — static and LLEXT builds; the LLEXT
47-
target name is `ffmpeg_dec` and links the FFmpeg libraries in the non-stub
48-
branch.
49-
- `ffmpeg_dec.toml` / `llext/llext.toml.h` — rimage module manifest
50-
(`UUIDREG_STR_FFMPEG_DEC`); OBS is sized above IBS since PCM out ≫ compressed
51-
in.
52-
53-
## Status / TODO
54-
55-
- Codec id is currently hard-coded to FLAC in `init`; wire it from topology/IPC
56-
init config for other codecs.
57-
- Output is assumed to already match the sink sample format; add
58-
`libswresample` (or a fixed-point path) for format/rate conversion.
59-
- Topology and host test tooling for end-to-end (bit-exact) FLAC decode.
1+
# FFmpeg Audio Processing Module (`ffmpeg_dec`)
2+
3+
Wraps FFmpeg's `libavcodec` and `libavfilter` libraries behind the SOF `module_interface`. This module enables running on-DSP audio decoders, encoders, and real-time audio filter graphs.
4+
5+
---
6+
7+
## Features
8+
9+
- **Decoder Mode**: Decodes compressed audio streams (e.g., FLAC, MP3) to raw PCM.
10+
- **Encoder Mode**: Encodes raw PCM streams to compressed formats (e.g., MP3 using `libshine`).
11+
- **Filter Mode**: Runs FFmpeg's `libavfilter` graphs directly on raw PCM data (e.g., `afftdn` for noise reduction).
12+
- **Embedded-Optimized**: Supports single-threaded execution, memory shims for dynamic allocation, and LLEXT packaging.
13+
- **CI-Friendly**: Includes a pass-through stub backend to allow pipeline, IPC, and topology verification without external dependencies.
14+
15+
---
16+
17+
## Architecture & Data Flow
18+
19+
The following Mermaid diagram illustrates the module's internal architecture and the interaction between the SOF pipeline, the `ffmpeg_dec` core, and its backend translation units:
20+
21+
```mermaid
22+
graph TD
23+
%% Audio Data Flow
24+
InBuf[Input Audio Buffer] -->|Raw Bytes or PCM| Core[ffmpeg_dec.c Core]
25+
Core -->|Pass-through| BackendStub[ffmpeg_dec-stub.c Stub]
26+
Core -->|Process| BackendFFmpeg[ffmpeg_dec-ffmpeg.c FFmpeg Backend]
27+
28+
BackendFFmpeg -->|Parse/Decode/Filter| Libavcodec[libavcodec / libavfilter]
29+
Libavcodec -->|Output PCM Frame| Core
30+
BackendStub -->|Pass-through S16/S32| Core
31+
Core -->|Interleaved PCM or Encoded Bytes| OutBuf[Output Audio Buffer]
32+
33+
%% Control Flow
34+
IPC[SOF IPC set_configuration] -.->|Metadata/Extradata| Core
35+
Core -.->|avcodec_open2 / init| Libavcodec
36+
```
37+
38+
### Modular Design
39+
- **`ffmpeg_dec.c` (Core Glue)**: Implements the standard `module_interface` functions (`init`, `prepare`, `process`, `reset`, `free`).
40+
- **`ffmpeg_dec-stub.c`**: Pass-through stub backend that bypasses FFmpeg libraries.
41+
- **`ffmpeg_dec-ffmpeg.c` / `ffmpeg_dec-filter.c` / `ffmpeg_dec-encode.c`**: Real backend implementations interfacing with `libavcodec` and `libavfilter`.
42+
- **`ffmpeg_dec-shims.c`**: Overrides dynamic memory allocations (`malloc`, `realloc`, `free`, `calloc`) inside FFmpeg to use SOF's `rballoc` pools.
43+
44+
---
45+
46+
## Build Instructions
47+
48+
### 1. Stub Mode (Default / CI / Staging)
49+
No external libraries are required. The module acts as a simple pass-through.
50+
```ini
51+
CONFIG_COMP_FFMPEG_DEC=y # or =m for LLEXT
52+
CONFIG_COMP_FFMPEG_DEC_STUB=y
53+
```
54+
55+
### 2. Real FFmpeg Backend
56+
Requires pre-compiled static libraries and headers placed under the `third_party/` directory of the workspace.
57+
```ini
58+
CONFIG_COMP_FFMPEG_DEC=m
59+
CONFIG_COMP_FFMPEG_DEC_STUB=n
60+
```
61+
62+
#### Configuring the FFmpeg Build
63+
When cross-compiling FFmpeg for Xtensa DSP targets, disable unnecessary components to minimize code size:
64+
```bash
65+
./configure \
66+
--target-os=none \
67+
--arch=xtensa \
68+
--enable-cross-compile \
69+
--disable-everything \
70+
--disable-avformat \
71+
--disable-pthreads \
72+
--enable-decoder=flac,mp3 \
73+
--enable-encoder=mp3 \
74+
--enable-parser=flac,mpegaudio \
75+
--enable-filter=afftdn \
76+
--enable-static \
77+
--disable-shared
78+
```
79+
80+
#### Useful Kconfig Options
81+
- `CONFIG_FFMPEG_DEC_FILTER_MODE`: Configures the module to run as a PCM filter graph instead of a decoder.
82+
- `CONFIG_FFMPEG_DEC_FLOAT_MATH`: Automatically selected to pull in optimized floating-point shims (`fastmathf.c`).
83+
- `CONFIG_FFMPEG_DEC_COLD_SPLIT`: Relocates initialization tables and code to DRAM to conserve fast SRAM.
84+
85+
---
86+
87+
## Usage & Topology
88+
89+
### 1. Decoder Mode
90+
The topology should feed compressed bytes (e.g., from a host gateway) into the decoder input pin. The decoder outputs decoded PCM.
91+
- **Initialization**: Provide codec-specific setup data (e.g., FLAC `STREAMINFO` or MP3 headers) via IPC bytes control `set_configuration`.
92+
- **Buffer Period**: Keep period size large enough (typically 10ms or 20ms) to reduce parsing overhead.
93+
94+
### 2. Filter Mode
95+
The module is placed in the capture or playback pipeline as a normal 1-in / 1-out effect.
96+
- **Routing Example**:
97+
```
98+
DAI Copier (Mic Capture) ---> [ ffmpeg_dec (Filter Mode) ] ---> Host Copier (PCM)
99+
```
100+
- **Control**: Filter coefficients and parameters can be set dynamically via standard TLV bytes controls.

‎src/audio/webrtc_aec/README.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# WebRTC Acoustic Echo Cancellation Module (`webrtc_aec`)
2+
3+
Wraps the fixed-point mobile echo canceller (AECm) from [webrtc-audio-processing](https://gitlab.freedesktop.org/gstreamer/webrtc-audio-processing) behind the SOF `module_interface`.
4+
5+
---
6+
7+
## Features
8+
9+
- **Dual-Input Processing**: Connects both the microphone capture stream (Pin 0) and the speaker playback reference stream (Pin 1).
10+
- **AECm Algorithm**: Highly optimized Q15 fixed-point echo cancellation; runs in real-time on low-power embedded DSPs without an FPU.
11+
- **Dynamic Routing**: Automatic stream alignment using SOF pipeline-ID mapping.
12+
- **Multichannel Independent Filtering**: Allocates one `AecmCore` handle per channel, supporting independent echo cancellation across multiple microphone paths.
13+
- **Rate Options**: Natively supports 8 kHz and 16 kHz sample rates (requires matching rates for both microphone and speaker reference inputs).
14+
- **Frame Buffer**: Synchronizes inputs and processes audio in 10 ms blocks.
15+
16+
---
17+
18+
## Architecture & Data Flow
19+
20+
The following Mermaid diagram explains how the dual input pins (Microphone and Speaker Reference) are processed and aligned inside the module:
21+
22+
```mermaid
23+
graph TD
24+
%% Microphone capture path (Pin 0)
25+
MicIn[Microphone Input Pin 0] -->|S16/S32 PCM| Core[webrtc_aec.c Core]
26+
27+
%% Speaker playback reference path (Pin 1)
28+
RefIn[Speaker Reference Pin 1] -->|S16 PCM| Core
29+
30+
%% Buffering & Alignment
31+
Core -->|Accumulate Mic| MicAccum[Mic Frame Buffer 10ms]
32+
Core -->|Accumulate Ref| RefAccum[Ref Frame Buffer 10ms]
33+
34+
MicAccum -->|Plane S16| Backend[webrtc_aec-aecm.c Backend]
35+
RefAccum -->|Plane S16| Backend
36+
37+
%% AEC Processing
38+
Backend -->|WebRtcAecm_BufferFarend| AECM[AecmCore Channel Instance]
39+
Backend -->|WebRtcAecm_Process| AECM
40+
41+
AECM -->|Clean Mic Audio| Backend
42+
Backend -->|Scale & Interleave| Core
43+
Core -->|Echo Cancelled Output| OutPin[Output Pin 0]
44+
```
45+
46+
### Components
47+
- `webrtc_aec.c`: Core SOF wrapper handling dual-source mapping, preparing format layouts, and period accumulator synchronization.
48+
- `webrtc_aec-aecm.c`: Real AECm library integration translation unit.
49+
- `webrtc_aec-stub.c`: Dependency-free pass-through stub.
50+
- `webrtc_aec.cmake`: CMake compiler scripting pulling the WebRTC AECm codebase from the `webrtc-apm` dependency.
51+
52+
---
53+
54+
## Build Instructions
55+
56+
### 1. Stub Mode (CI / Staging)
57+
```ini
58+
CONFIG_COMP_WEBRTC_AEC=y # or =m for LLEXT
59+
CONFIG_COMP_WEBRTC_AEC_STUB=y
60+
```
61+
62+
### 2. Real AECm Integration
63+
Integrates the WebRTC AECm engine:
64+
```ini
65+
CONFIG_COMP_WEBRTC_AEC=m
66+
CONFIG_COMP_WEBRTC_AEC_STUB=n
67+
```
68+
*Note: Make sure to run `west update` to retrieve `modules/audio/webrtc-apm` before compiling.*
69+
70+
---
71+
72+
## Kconfig Parameters
73+
74+
| Option | Default | Range / Value | Description |
75+
|---|---|---|---|
76+
| `CONFIG_COMP_WEBRTC_AEC` | n | y / m / n | Enable WebRTC AECm module |
77+
| `CONFIG_COMP_WEBRTC_AEC_STUB` | y | y / n | Use pass-through stub |
78+
| `CONFIG_WEBRTC_AEC_CHANNELS_MAX` | 2 | 1 - 8 | Maximum channels supported |
79+
| `CONFIG_WEBRTC_AEC_ROUTING_HEURISTIC` | y | y / n | Use pipeline-ID matching |
80+
81+
---
82+
83+
## Usage & Topology
84+
85+
### Topology Connections
86+
AEC requires two input pin bindings:
87+
1. **Pin 0 (Capture)**: Fed by the Microphone SSP DAI.
88+
2. **Pin 1 (Reference)**: Fed by the Speaker SSP DAI (cross-pipeline).
89+
90+
#### Example Topology Code snippet:
91+
```
92+
Object.Widget.webrtc-aec.1 {
93+
# Pin 0: Microphone Capture
94+
Object.Base.input_pin_binding.1 {
95+
input_pin_binding_name "dai-copier.SSP.NoCodec-0.capture"
96+
}
97+
# Pin 1: Speaker Reference
98+
Object.Base.input_pin_binding.2 {
99+
input_pin_binding_name "dai-copier.SSP.NoCodec-2.capture"
100+
}
101+
}
102+
```
103+
104+
Both input streams must be active and running at the same sample rate (8 kHz or 16 kHz) for the echo canceller to align and filter the signals.

‎src/audio/webrtc_ns/README.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# WebRTC Noise Suppression Module (`webrtc_ns`)
2+
3+
Wraps the classic fixed-point WebRTC Noise Suppression (NS) algorithm (spectral Wiener filter) from [webrtc-audio-processing](https://gitlab.freedesktop.org/gstreamer/webrtc-audio-processing) behind the SOF `module_interface`.
4+
5+
---
6+
7+
## Features
8+
9+
- **Wiener Filter Suppression**: Fixed-point spectral subtraction to isolate stationary noise.
10+
- **Low Complexity**: Highly optimized for low-power embedded DSPs; requires no FPU.
11+
- **Multiple Modes**: Supports four levels of suppression (Mild, Medium, Aggressive, Very Aggressive).
12+
- **Per-Channel Processing**: Allocates one NS instance per audio channel, processing them independently.
13+
- **8 kHz and 16 kHz Support**: Natively supports 8 kHz and 16 kHz sample rates (requires resampling upstream if running at higher rates).
14+
- **Frame Size**: Operates on 10 ms audio frames (80 samples at 8 kHz, 160 samples at 16 kHz).
15+
16+
---
17+
18+
## Architecture & Data Flow
19+
20+
The following Mermaid diagram outlines the internal architecture of the `webrtc_ns` module:
21+
22+
```mermaid
23+
graph TD
24+
%% Data Flow
25+
InBuf[Input Audio Buffer] -->|S16/S32 Interleaved| Core[webrtc_ns.c Core]
26+
Core -->|1. Demux & Normalise| ScaleIn[Format Conversion]
27+
ScaleIn -->|10 ms Frame per Channel| Backend[webrtc_ns-webrtc.c Backend]
28+
29+
Backend -->|WebRTC NS Core Filter| NSInstance[NsCore Instance per Channel]
30+
NSInstance -->|Filtered Audio| Backend
31+
Backend -->|Scale & Interleave| ScaleOut[Format Conversion]
32+
33+
ScaleOut -->|S16/S32 Interleaved| Core
34+
Core -->|Denoised PCM| OutBuf[Output Audio Buffer]
35+
36+
%% Control Flow
37+
IPC[IPC Control / Set Config] -.->|Suppression Level| Core
38+
Core -.->|WebRtcNs_set_policy| NSInstance
39+
```
40+
41+
### Components
42+
- `webrtc_ns.c`: Core SOF wrapper logic, handling format parsing, period buffering, and interleaved-to-planar copying.
43+
- `webrtc_ns-webrtc.c`: Integration backend interfacing directly with the WebRTC NS codebase.
44+
- `webrtc_ns-stub.c`: Standard pass-through stub for testing.
45+
- `webrtc_ns-shims.c`: Platform-specific macros and memory mapping layers.
46+
- `webrtc_ns.cmake`: Downloads and extracts the `webrtc-audio-processing` 0.3.1 source and compiles the necessary C files.
47+
48+
---
49+
50+
## Build Instructions
51+
52+
### 1. Stub Mode (CI / Staging)
53+
```ini
54+
CONFIG_COMP_WEBRTC_NS=y # or =m for LLEXT
55+
CONFIG_COMP_WEBRTC_NS_STUB=y
56+
```
57+
58+
### 2. Real NS Integration
59+
Pulls the classic WebRTC codebase via West and builds the fixed-point Wiener filter:
60+
```ini
61+
CONFIG_COMP_WEBRTC_NS=m
62+
CONFIG_COMP_WEBRTC_NS_STUB=n
63+
```
64+
*Note: Make sure to run `west update` to retrieve `modules/audio/webrtc-apm` before compiling.*
65+
66+
---
67+
68+
## Kconfig Parameters
69+
70+
| Option | Default | Range / Value | Description |
71+
|---|---|---|---|
72+
| `CONFIG_COMP_WEBRTC_NS` | n | y / m / n | Enable WebRTC Noise Suppressor |
73+
| `CONFIG_COMP_WEBRTC_NS_STUB` | y | y / n | Use pass-through stub |
74+
| `CONFIG_WEBRTC_NS_POLICY` | 2 | 0 - 3 | Severity (0: Mild, 3: Very Aggressive) |
75+
| `CONFIG_WEBRTC_NS_CHANNELS_MAX` | 2 | 1 - 8 | Maximum channels supported |
76+
77+
---
78+
79+
## Usage & Topology
80+
81+
### Pipeline Integration
82+
The module is integrated as a normal 1-in / 1-out effect widget. It should be run in a pipeline configured for either **8000 Hz** or **16000 Hz**.
83+
84+
#### Example Topology Pipeline
85+
```
86+
DAI Copier (SSP RX) ---> [ webrtc-ns ] ---> Host Copier (PCM)
87+
```
88+
89+
For applications requiring higher rates (e.g. 48 kHz mic capture), place an ASRC component upstream to downsample to 16 kHz before running the `webrtc-ns` widget.

0 commit comments

Comments
 (0)