Skip to content

Commit 5a5a496

Browse files
cjpaiscjpais
authored
Moss transcribe diarize (#84)
* stage 1: * stage 2 * stage 3 * stage 4 * flash attn * stage 5 * fix scoring for moss * docs * m4 max * add 4750u * moss root readme * format * pr cleanup
1 parent fd8b8f8 commit 5a5a496

30 files changed

Lines changed: 6641 additions & 22 deletions

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ C/C++ speech-to-text inference library. Runs diverse STT model families via [GGU
2626
| Voxtral | `voxtral-mini-3b-2507`, `voxtral-small-24b-2507` (audio-LLM; transcription + translation) | [docs/models/voxtral.md](docs/models/voxtral.md) |
2727
| Voxtral Realtime | `voxtral-mini-4b-realtime-2602` (streaming audio-LLM) | [docs/models/voxtral-realtime.md](docs/models/voxtral-realtime.md) |
2828
| MedASR | `medasr` (Conformer + CTC, English medical-dictation, gated) | [docs/models/medasr.md](docs/models/medasr.md) |
29+
| MOSS Transcribe-Diarize | `moss-transcribe-diarize` (audio-LLM; English + Chinese ASR with inline speaker diarization) | [docs/models/moss-transcribe-diarize.md](docs/models/moss-transcribe-diarize.md) |
2930

3031
Per-variant model cards live under [`docs/models/`](docs/models/).
3132

‎docs/environment-variables.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,7 @@ of tests.
2929
| `TRANSCRIBE_CONV_DIRECT_DW` / `TRANSCRIBE_CONV_NO_DIRECT_DW` | Force the depthwise-conv dispatch to the direct `conv_2d_dw` path / the im2col path, overriding the per-family backend default. |
3030
| `TRANSCRIBE_CONV_DIRECT_PW` / `TRANSCRIBE_CONV_NO_DIRECT_PW` | Force the pointwise-conv dispatch to direct `mul_mat` / im2col, overriding the backend default. |
3131
| `TRANSCRIBE_DUMP_DIR=<dir>` | Enable the per-stage tensor dumper; writes `<name>.f32` + `<name>.json` per dumped tensor into `<dir>`. The basis for the numerical-comparison harness (`scripts/compare_tensors.py`). |
32-
| `TRANSCRIBE_PERF_DEBUG` | Print a per-stage timing breakdown to stderr (DEBUG log) on the families that profile (`cohere`, `granite`, `canary`, `canary_qwen`, `moonshine`, `moonshine_streaming`, `qwen3_asr`, `whisper`). For whisper, a value containing `cpu` or `all` additionally prints the CPU sub-section breakdown. |
32+
| `TRANSCRIBE_PERF_DEBUG` | Print a per-stage timing breakdown to stderr (DEBUG log) on the families that profile (`cohere`, `granite`, `canary`, `canary_qwen`, `moonshine`, `moonshine_streaming`, `moss`, `qwen3_asr`, `whisper`). For whisper, a value containing `cpu` or `all` additionally prints the CPU sub-section breakdown. |
3333
| `TRANSCRIBE_VOXTRAL_REALTIME_STREAM_TIMING` | Print a per-component streaming wall-time breakdown at stream finalize (voxtral_realtime). |
3434
| `TRANSCRIBE_TEST_DEV_INIT_THROW=<match>` | Fault injection: backend device init (`ggml_backend_dev_init`) throws for devices whose name contains `<match>` (`*` matches every device). Exercises throw → skip → CPU-fallback in backend probing; an explicit backend request fails with `TRANSCRIBE_ERR_BACKEND`. Used by `backend_init_throw_unit` and `scripts/ci/vulkan_degradation_check.py`. |
3535
| `TRANSCRIBE_TEST_TEARDOWN_THROW` | Fault injection: any non-empty value injects a throw after each real free inside the `transcribe::safe_*` teardown wrappers, proving containment without leaking the handle. Used by `teardown_safety_unit`. |

‎docs/models/moss-transcribe-diarize.md‎

Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
# MOSS-Transcribe-Diarize
2+
3+
OpenMOSS's [`OpenMOSS-Team/MOSS-Transcribe-Diarize`](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize)
4+
ported to transcribe.cpp. A 0.9B audio-LLM: a 24-layer Whisper-Medium audio
5+
encoder (`d_model=1024`, GELU, LayerNorm) feeds a 4x temporal merge
6+
(1024 -> 4096) and a VQAdaptor MLP bridge into a Qwen3-0.6B causal decoder
7+
(28 layers, `hidden_size=1024`, GQA 16/8 heads, `rope_theta=1e6`) via
8+
audio-token injection at `<|audio_pad|>` positions.
9+
10+
## What it's for
11+
12+
Offline English and Chinese speech-to-text with speaker diarization. Takes a
13+
16 kHz mono WAV and produces transcript text in the canonical diarized format
14+
`[start][Sxx]text[end]` (e.g. `[0.48][S01]Welcome[1.66]`) via greedy decoding.
15+
The speaker tags (`[S01]`, `[S02]`, ...) and segment start/end timestamps are
16+
emergent generated text, not special tokens. Built for long-form,
17+
multi-speaker audio. No translation; not a streaming model.
18+
19+
See OpenMOSS's [model card](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize)
20+
for training data, intended use, and upstream evaluation. All of OpenMOSS's
21+
published metrics are Chinese multi-speaker diarization CER/cpCER; LibriSpeech
22+
test-clean is used here only as an English acceptance set.
23+
24+
Licensed Apache-2.0. Ported from upstream commit
25+
[`d7231bb`](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize/commit/d7231bbae2587a4af278735eb765b318c4f64edd),
26+
pinned 2026-07-12.
27+
28+
## Memory and length
29+
30+
MOSS keeps the whole recording in memory while it transcribes, so RAM use grows
31+
with how long the audio is. Plan for roughly **85 MB of extra memory per minute
32+
of audio**, on top of the model file itself — about **2.5 GB for a 30-minute
33+
clip** and **~5 GB for an hour**. It can handle recordings up to a couple of
34+
hours if you have the memory; anything longer is rejected with a clear error
35+
instead of being silently cut off. If you run low on memory, split long audio
36+
into shorter pieces.
37+
38+
## Download
39+
40+
| Quantization | Download | Size | WER (LibriSpeech test-clean) |
41+
| --- | --- | ---: | ---: |
42+
| BF16 | [MOSS-Transcribe-Diarize-BF16.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-BF16.gguf) | 1.83 GB | 2.08% |
43+
| F16 | [MOSS-Transcribe-Diarize-F16.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-F16.gguf) | 1.83 GB | 2.07% |
44+
| Q8_0 | [MOSS-Transcribe-Diarize-Q8_0.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-Q8_0.gguf) | 987 MB | 1.93% |
45+
| Q6_K | [MOSS-Transcribe-Diarize-Q6_K.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-Q6_K.gguf) | 768 MB | 1.96% |
46+
| Q5_K_M | [MOSS-Transcribe-Diarize-Q5_K_M.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-Q5_K_M.gguf) | 700 MB | 1.99% |
47+
| Q4_K_M | [MOSS-Transcribe-Diarize-Q4_K_M.gguf](https://huggingface.co/handy-computer/MOSS-Transcribe-Diarize-gguf/resolve/main/MOSS-Transcribe-Diarize-Q4_K_M.gguf) | 617 MB | 2.59% |
48+
49+
These WER values describe this dataset only, not a general quality ranking. A
50+
quant that scores slightly better here is not necessarily better in real-world
51+
use; dataset-specific decoding near-ties can make quantization noise help or
52+
hurt individual utterances.
53+
54+
WER measured on the full LibriSpeech `test-clean` split (2620 utterances) with
55+
the Whisper-style English normalizer and jiwer 3.x. MOSS emits the diarized
56+
format `[start][Sxx]text[end]`; the bracket spans are metadata and are
57+
de-diarized to a space (for both hypothesis and reference) before scoring,
58+
matching the author-repo reference runner. The same-manifest MOSS author-repo
59+
reference (bf16, greedy) lands at **2.07%**, 95% bootstrap CI [1.82%, 2.40%];
60+
the BF16 port lands at 2.08%, within `+0.01pp` of the reference and well inside
61+
the CI. Q4_K_M's higher 2.59% is not broad degradation but a handful of 4-bit
62+
tail failures (6 empty outputs, 5 English->Chinese language-drift utterances,
63+
1 timestamp-token repetition loop); prefer Q5_K_M or higher if those matter.
64+
Dediarization is scoring-only: the runtime continues to return the raw inline
65+
timestamps and speaker labels.
66+
67+
## Quick Start
68+
69+
```bash
70+
cmake -B build
71+
cmake --build build
72+
73+
build/bin/transcribe-cli \
74+
-m models/MOSS-Transcribe-Diarize/MOSS-Transcribe-Diarize-Q8_0.gguf \
75+
samples/jfk.wav
76+
```
77+
78+
If your audio is not already 16 kHz mono WAV, convert it first:
79+
80+
```bash
81+
ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav
82+
```
83+
84+
CLI flags:
85+
86+
- `-l en` / `-l zh` (or omit for `auto`): English and Chinese are supported.
87+
- No `--task` / `--target-language`: the model is ASR-only, no translation.
88+
- Diarization tags and segment timestamps appear inline in the transcript
89+
text; there is no toggle to suppress them.
90+
91+
## Performance
92+
93+
Cells are wall-clock latency (mean over 3 iterations after 1 warmup),
94+
with speedup over realtime in parentheses. Units: `ms` below 1 s, `s`
95+
above (2 decimal places).
96+
97+
### Apple M4 Max
98+
99+
| Backend | Sample | Q8_0 | Q4_K_M |
100+
| ------- | ------------ | ------------: | ------------: |
101+
| Metal | jfk (11.0s) | 388 ms (28.3×) | 369 ms (29.8×) |
102+
| Metal | dots (35.3s) | 1.27 s (27.8×) | 1.17 s (30.1×) |
103+
| CPU | jfk (11.0s) | 2.06 s (5.3×) | 2.37 s (4.6×) |
104+
| CPU | dots (35.3s) | 5.71 s (6.2×) | 5.84 s (6.0×) |
105+
106+
macOS 26.5.1, transcribe.cpp `e745720`.
107+
108+
### AMD Ryzen 7 PRO 4750U
109+
110+
| Backend | Sample | Q8_0 | Q4_K_M |
111+
| ------- | ------------ | ------------: | ------------: |
112+
| Vulkan | jfk (11.0s) | 3.88 s (2.8×) | 3.68 s (3.0×) |
113+
| Vulkan | dots (35.3s) | 11.38 s (3.1×) | 10.68 s (3.3×) |
114+
| CPU | jfk (11.0s) | 7.49 s (1.5×) | 7.06 s (1.6×) |
115+
| CPU | dots (35.3s) | 21.20 s (1.7×) | 19.22 s (1.8×) |
116+
117+
Fedora Linux 43, transcribe.cpp `e745720`. Vulkan device: `AMD Radeon
118+
Graphics (RADV RENOIR)`.
119+
120+
Benchmark reproduction:
121+
122+
```bash
123+
uv run scripts/bench/run.py \
124+
--models moss-transcribe-diarize \
125+
--quants q8_0,q4_k_m \
126+
--samples jfk,dots \
127+
--backends metal,cpu,vulkan \
128+
--iters 3 --warmup 1 \
129+
--name moss-transcribe-diarize-publication
130+
```
131+
132+
## Numerical Validation
133+
134+
transcribe.cpp is validated tensor-by-tensor against the MOSS author repo
135+
(`scripts/dump_reference_moss_author.py`, `trust_remote_code`) on
136+
`samples/jfk.wav` with the strict CPU backend. The reference runs BF16 (torch,
137+
eager attention); the C++ path dequantizes BF16 weights to F32 and computes in
138+
F32, so C++ is the *more* precise side and the residual gap is a constant
139+
~1-3% relative bf16-vs-f32 drift, not a bug. The transcript compare is
140+
`dediarized` (bracket metadata stripped to a space). Confirmed WER-neutral: on
141+
the first 100 test-clean utterances the C++ ref-dtype WER (1.40%) is
142+
bit-identical to the Oracle reference on the same subset (1.40%). Tolerances
143+
are pinned in `tests/tolerances/moss.json` with a `_comment` block naming the
144+
precision regime, the large-pre-normalization-activation maxes, and the encoder
145+
padding-trim contract. Last validated at commit
146+
[`3f5e15c`](https://github.com/handy-computer/transcribe.cpp/tree/3f5e15c).
147+
148+
| Field | Value |
149+
| --- | --- |
150+
| Reference | MOSS author repo (`OpenMOSS-Team/MOSS-Transcribe-Diarize`) |
151+
| Dump script | `scripts/dump_reference_moss_author.py` |
152+
| Manifest | `tests/golden/moss/moss-transcribe-diarize.manifest.json` |
153+
| Tolerances | `tests/tolerances/moss.json` |
154+
| Command | `uv run scripts/validate.py all --family moss --variant moss-transcribe-diarize` |
155+
156+
Selected tensors (observed on CPU, strict backend; see the tolerance file for
157+
budgets and per-tensor notes):
158+
159+
| Tensor | Max abs diff | Mean abs diff | Notes |
160+
| --- | ---: | ---: | --- |
161+
| `enc.mel.in` | `9.872e-05` | `5.150e-06` | C++ MelFrontend vs reference feature extractor |
162+
| `enc.pos_add.out` | `5.000e-02` | `2.200e-03` | Encoder input + positional embedding |
163+
| `enc.block.0.out` | `1.200e-01` | `4.000e-03` | First Whisper encoder block |
164+
| `enc.block.23.out` | `3.100e+03` | `8.000e-02` | Pre-final-LN residual; ref \|max\| ~3.6e3, bf16 rel error dominates max_abs (renormalized by `enc.ln_post`) |
165+
| `enc.ln_post.out` | `1.300e+01` | `3.000e-03` | Encoder output LayerNorm |
166+
| `enc.merge.out` | `1.500e+00` | `3.000e-03` | 4x temporal merge |
167+
| `enc.adaptor.out` | `4.000e-01` | `1.200e-02` | VQAdaptor decoder handoff (rel_mean ~0.96%) |
168+
| `dec.audio_injected` | `4.000e-01` | `7.000e-03` | Audio tokens scattered into the prompt |
169+
| `dec.block.0.out` | `7.500e-01` | `9.500e-03` | First Qwen3 decoder block |
170+
| `dec.block.27.out` | `3.200e+02` | `2.600e-01` | Pre-final-RMSNorm residual (bf16 accumulation over 28 layers) |
171+
| `dec.out_before_head` | `8.500e+00` | `8.000e-02` | Pre-head hidden state |
172+
| `dec.logits_raw` | `4.800e-01` | `7.200e-02` | Prefill logits; argmax preserved (transcript exact) |
173+
| `dec.logits_raw.gen8` | `5.000e-01` | `6.500e-02` | Greedy step 8 logits (KV-cache decode coverage) |
174+
| `dec.token_emb` | `0.000e+00` | `0.000e+00` | Pure embedding lookup (pinned exact) |
175+
176+
For the full porting writeup, see
177+
[`docs/porting/families/moss.md`](../porting/families/moss.md).
178+
179+
## Reproduction
180+
181+
### Convert
182+
183+
```bash
184+
uv run --project scripts/envs/moss \
185+
scripts/convert-moss.py OpenMOSS-Team/MOSS-Transcribe-Diarize \
186+
--revision d7231bbae2587a4af278735eb765b318c4f64edd
187+
```
188+
189+
### Quantize
190+
191+
```bash
192+
uv run scripts/quantize-all.py models/MOSS-Transcribe-Diarize/MOSS-Transcribe-Diarize-BF16.gguf
193+
```
194+
195+
### Validate
196+
197+
```bash
198+
uv run scripts/validate.py all --family moss --variant moss-transcribe-diarize
199+
```
200+
201+
### Score WER
202+
203+
```bash
204+
PRESET=BF16
205+
uv run scripts/wer/run.py \
206+
--model models/MOSS-Transcribe-Diarize/MOSS-Transcribe-Diarize-${PRESET}.gguf \
207+
--manifest samples/wer/librispeech-test-clean.manifest.jsonl \
208+
--out reports/wer/MOSS-Transcribe-Diarize-${PRESET}.librispeech-test-clean.jsonl
209+
uv run scripts/wer/score.py \
210+
reports/wer/MOSS-Transcribe-Diarize-${PRESET}.librispeech-test-clean.jsonl \
211+
--dediarize
212+
```
213+
214+
### Score WER against the MOSS author-repo reference
215+
216+
```bash
217+
uv run --project scripts/envs/moss \
218+
scripts/wer/run_reference_moss_author.py \
219+
--model OpenMOSS-Team/MOSS-Transcribe-Diarize \
220+
--revision d7231bbae2587a4af278735eb765b318c4f64edd \
221+
--manifest samples/wer/librispeech-test-clean.manifest.jsonl \
222+
--out reports/wer/moss-transcribe-diarize-REF.librispeech-test-clean.jsonl
223+
uv run scripts/wer/score.py \
224+
reports/wer/moss-transcribe-diarize-REF.librispeech-test-clean.jsonl \
225+
--dediarize
226+
```

0 commit comments

Comments
 (0)