Parakeet multitalker

README.md

@@ -21,6 +21,7 @@ C/C++ speech-to-text inference library. Runs diverse STT model families via [GGU
 | FunASR Nano | `fun-asr-nano-2512`, `fun-asr-mlt-nano-2512` | [docs/models/fun-asr-nano.md](docs/models/fun-asr-nano.md) |
 | Nemotron Speech Streaming | `nemotron-speech-streaming-en-0.6b` | [docs/models/nemotron-speech-streaming-en-0.6b.md](docs/models/nemotron-speech-streaming-en-0.6b.md) |
 | Nemotron 3.5 ASR Streaming | `nemotron-3.5-asr-streaming-0.6b` (multilingual, 40 locales) | [docs/models/nemotron-3.5-asr-streaming-0.6b.md](docs/models/nemotron-3.5-asr-streaming-0.6b.md) |
+| Multitalker Parakeet Streaming | `multitalker-parakeet-streaming-0.6b-v1` (single-speaker ASR path only) | [docs/models/multitalker-parakeet-streaming-0.6b-v1.md](docs/models/multitalker-parakeet-streaming-0.6b-v1.md) |
 | Granite Speech 4 / 4.1 | `granite-4.0-1b-speech`, `granite-speech-4.1-2b{,-plus,-nar}` | [docs/models/granite-speech.md](docs/models/granite-speech.md) |
 | Voxtral | `voxtral-mini-3b-2507`, `voxtral-small-24b-2507` (audio-LLM; transcription + translation) | [docs/models/voxtral.md](docs/models/voxtral.md) |
 | Voxtral Realtime | `voxtral-mini-4b-realtime-2602` (streaming audio-LLM) | [docs/models/voxtral-realtime.md](docs/models/voxtral-realtime.md) |

docs/models/multitalker-parakeet-streaming-0.6b-v1.md

@@ -0,0 +1,210 @@
+# Multitalker Parakeet Streaming 0.6B v1
+
+NVIDIA's [`nvidia/multitalker-parakeet-streaming-0.6b-v1`](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1)
+ported to transcribe.cpp. A 0.6B-parameter cache-aware streaming
+FastConformer encoder with an RNN-T transducer decoder, fine-tuned from
+[`nvidia/nemotron-speech-streaming-en-0.6b`](https://huggingface.co/nvidia/nemotron-speech-streaming-en-0.6b).
+
+## What it's for
+
+Offline and cache-aware **streaming** English speech-to-text with greedy
+RNN-T decoding. Outputs cased, punctuated transcripts. Token- and
+word-level timestamps are available.
+
+Upstream this is a **multitalker (speaker-attributed)** checkpoint: it can
+transcribe several overlapping speakers into per-speaker channels. That
+path depends on machinery this port does **not** ship (see
+[Known limitations](#known-limitations)). transcribe.cpp exposes only the
+model's `single_speaker_mode` ASR path, which disables the multitalker
+machinery and runs the checkpoint as a cache-aware streaming RNN-T with
+the base model's frontend, encoder backbone, decoder, and tokenizer plus
+the checkpoint's always-on layer-0 speaker-kernel injection.
+
+This port runs the model in both **offline** and **cache-aware streaming**
+modes.
+
+See NVIDIA's [model card](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1)
+for training data, intended use, the multitalker methodology, and the full
+latency-vs-accuracy table.
+
+Licensed under the [NVIDIA Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/).
+Ported from upstream commit
+[`8749fc7`](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1/commit/8749fc71fd6e2d88ef230159bbf2aea69b524ee1),
+pinned 2026-07-12.
+
+## Download
+
+| Quantization | Download | Size | WER (LibriSpeech test-clean, offline) |
+| --- | --- | ---: | ---: |
+| F32    | [multitalker-parakeet-streaming-0.6b-v1-F32.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-F32.gguf)       | 2.49 GB | 2.19% |
+| F16    | [multitalker-parakeet-streaming-0.6b-v1-F16.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-F16.gguf)       | 1.25 GB | 2.19% |
+| Q8_0   | [multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf)     | 734 MB  | 2.18% |
+| Q6_K   | [multitalker-parakeet-streaming-0.6b-v1-Q6_K.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-Q6_K.gguf)     | 604 MB  | 2.20% |
+| Q5_K_M | [multitalker-parakeet-streaming-0.6b-v1-Q5_K_M.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-Q5_K_M.gguf) | 542 MB  | 2.18% |
+| Q4_K_M | [multitalker-parakeet-streaming-0.6b-v1-Q4_K_M.gguf](https://huggingface.co/handy-computer/multitalker-parakeet-streaming-0.6b-v1-gguf/resolve/main/multitalker-parakeet-streaming-0.6b-v1-Q4_K_M.gguf) | 478 MB  | 2.18% |
+
+WER is measured on the full LibriSpeech test-clean split (2620 utterances)
+in `single_speaker_mode` with greedy RNN-T decoding, whisper-normalizer
+scoring (PnC-stripped), and no external LM. F32 reference baseline: 2.19%.
+The measured NeMo `single_speaker_mode` reference on the same split is
+2.19%, and NVIDIA's self-reported number is 2.19% (from the
+[HF model card](https://huggingface.co/nvidia/multitalker-parakeet-streaming-0.6b-v1)).
+
+## Streaming parity
+
+Cache-aware streaming was validated tensor-by-tensor against NeMo's
+`conformer_stream_step` reference via `scripts/validate_streaming.py`. On
+`samples/jfk.wav` at `--backend cpu --threads 1`, the always-on layer-0
+speaker-kernel injection is applied per chunk on the post-drop block-0
+input (matching the offline path), and the harness reports **150/150
+streaming-tensor pairs within tolerance and the transcript byte-exact vs
+NeMo at R=13** (`att_context_size=[70, 13]`, 1.12s chunk). Streaming
+`enc_out` drift (observed 5.5e-6) is tighter than the offline C++
+`enc.final` drift (1.9e-3) on the same audio, because clean per-chunk
+caches accumulate less error than a single 24-block offline pass.
+
+Reproduce:
+
+```bash
+uv run --project scripts/envs/parakeet scripts/validate_streaming.py \
+    --hf-model nvidia/multitalker-parakeet-streaming-0.6b-v1 \
+    --gguf models/multitalker-parakeet-streaming-0.6b-v1/multitalker-parakeet-streaming-0.6b-v1-F32.gguf \
+    --audio samples/jfk.wav \
+    --out build/validate_streaming/multitalker/jfk \
+    --right 13 6 1 0 \
+    --backend cpu --threads 1 \
+    --tolerances tests/tolerances/multitalker-parakeet-streaming-0.6b-v1.streaming.json
+```
+
+## Quick Start
+
+```bash
+cmake -B build
+cmake --build build
+
+build/bin/transcribe-cli \
+  -m models/multitalker-parakeet-streaming-0.6b-v1/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf \
+  samples/jfk.wav
+```
+
+If your audio is not already 16 kHz mono WAV, convert it first:
+
+```bash
+ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav
+```
+
+## Performance
+
+Cells are wall-clock latency (mean over 3 iterations after 1 warmup),
+with speedup over realtime in parentheses. Units: `ms` below 1 s, `s`
+above (2 decimal places).
+
+### Apple M4 Max
+
+| Backend | Sample       |          Q8_0 |        Q4_K_M |
+| ------- | ------------ | ------------: | ------------: |
+| Metal   | jfk (11.0s)  |  67 ms (164×) |  69 ms (159×) |
+| Metal   | dots (35.3s) | 184 ms (192×) | 185 ms (191×) |
+| CPU     | jfk (11.0s)  | 310 ms (36×)  | 307 ms (36×)  |
+| CPU     | dots (35.3s) | 1.05 s (34×)  | 1.03 s (34×)  |
+
+macOS 26.5.1, transcribe.cpp `c55a09d`.
+
+### AMD Ryzen 7 4750U Pro
+
+| Backend | Sample       |          Q8_0 |        Q4_K_M |
+| ------- | ------------ | ------------: | ------------: |
+| Vulkan  | jfk (11.0s)  |  466 ms (24×) |  475 ms (23×) |
+| Vulkan  | dots (35.3s) |  1.36 s (26×) |  1.39 s (26×) |
+| CPU     | jfk (11.0s)  |  751 ms (15×) |  816 ms (13×) |
+| CPU     | dots (35.3s) |  2.99 s (12×) |  3.12 s (11×) |
+
+Fedora 43, transcribe.cpp `c55a09d`. Vulkan device: `AMD Radeon
+Graphics (RADV RENOIR)`.
+
+Benchmark reproduction:
+
+```bash
+uv run scripts/bench/run.py \
+  --models multitalker-parakeet-streaming-0.6b-v1 \
+  --quants q8_0,q4_k_m \
+  --samples jfk,dots \
+  --backends metal,cpu,vulkan \
+  --iters 3 --warmup 1 \
+  --name multitalker-parakeet-streaming-0.6b-v1-publication
+```
+
+## Numerical Validation
+
+transcribe.cpp is validated tensor-by-tensor against NeMo on
+`samples/jfk.wav` via `scripts/validate.py`. Per-tensor tolerances live
+in a per-variant file
+([`tests/tolerances/multitalker-parakeet-streaming-0.6b-v1.json`](../../tests/tolerances/multitalker-parakeet-streaming-0.6b-v1.json))
+rather than the family-shared one because the unnormalised log-mel
+(NeMo's `normalize="NA"` no-op) lands on a different magnitude scale than
+the per-feature-normalised siblings, and because the layer-0
+speaker-kernel injection is unique to this variant. The family-level
+forward map at
+[`reports/porting/parakeet/forward-map.md`](../../reports/porting/parakeet/forward-map.md)
+documents the per-stage divergence sources.
+
+| Field | Value |
+| --- | --- |
+| Reference | NeMo, `nvidia/multitalker-parakeet-streaming-0.6b-v1` (`single_speaker_mode`) |
+| Dump script | `scripts/dump_reference_parakeet_nemo.py` |
+| Manifest | `tests/golden/parakeet/multitalker-parakeet-streaming-0.6b-v1.manifest.json` |
+| Command | `uv run scripts/validate.py all --family parakeet --variant multitalker-parakeet-streaming-0.6b-v1` |
+
+## Batch
+
+The model ships an explicit `run_batch()` parallel fast path. Batch output
+is WER-neutral: byte-equal to the serial single-stream path at batch sizes
+2 / 4 / 8 (golden frozen at
+`tests/golden/batch/multitalker-parakeet-streaming-0.6b-v1.cpu.json`),
+CPU same-length tensor parity is bit-exact (max_abs 0.0) at batch=4 on
+`jfk.wav`, diverse-length flash tensor parity is bit-exact across arbitrary
+length mixes, and full test-clean batch-8 WER equals batch-1 (2.19%).
+
+## Known limitations
+
+- **Single-speaker only. The multitalker / speaker-attributed ASR path is
+  not ported.** Upstream, this checkpoint transcribes several overlapping
+  speakers into per-speaker channels. That path requires (1) an external
+  streaming speaker-diarization model
+  ([`nvidia/diar_streaming_sortformer_4spk-v2.1`](https://huggingface.co/nvidia/diar_streaming_sortformer_4spk-v2.1),
+  a separate Sortformer checkpoint that this repo does not ship), (2)
+  per-frame speaker-kernel injection driven by that diarizer's supervision,
+  (3) one encoder+decoder instance per speaker (up to 4), and (4) a
+  speaker-tagged SegLST output contract. None of these exist in
+  transcribe.cpp today, so the port runs `single_speaker_mode` and produces
+  a single flat transcript. It does **not** separate speakers, diarize, or
+  emit speaker turns.
+- **English only.** The model is English-only by training.
+- **No translation and no language detection.**
+
+## Reproduction
+
+### Convert
+
+```bash
+uv run --project scripts/envs/parakeet \
+  scripts/convert-parakeet.py nvidia/multitalker-parakeet-streaming-0.6b-v1
+```
+
+### Quantize
+
+Run `transcribe-quantize` once per target quant. Example for Q8_0; repeat
+with `F16`, `Q6_K`, `Q5_K_M`, `Q4_K_M`:
+
+```bash
+build/bin/transcribe-quantize \
+  models/multitalker-parakeet-streaming-0.6b-v1/multitalker-parakeet-streaming-0.6b-v1-F32.gguf \
+  models/multitalker-parakeet-streaming-0.6b-v1/multitalker-parakeet-streaming-0.6b-v1-Q8_0.gguf \
+  --quant Q8_0
+```
+
+### Validate
+
+```bash
+uv run scripts/validate.py all --family parakeet --variant multitalker-parakeet-streaming-0.6b-v1
+```

docs/porting/families/parakeet.md

@@ -31,7 +31,12 @@ CI-overlap grounds; see the family WER summary for rationale.
   - **Cache-aware streaming RNN-T (encoder-transducer)**:
     `nemotron-speech-streaming-en-0.6b` (English-only),
     `nemotron-3.5-asr-streaming-0.6b` (multilingual, 40 locales,
-    language one-hot conditioning) — intake only, Stages 2-8 TODO
+    language one-hot conditioning) — intake only, Stages 2-8 TODO,
+    `multitalker-parakeet-streaming-0.6b-v1` (English-only, fine-tuned
+    from `nemotron-speech-streaming-en-0.6b`; ships the
+    `single_speaker_mode` ASR path only — the multitalker /
+    speaker-attribution machinery is out of scope, see Capability
+    Validation rows)
 
 Per-variant intake JSON: `reports/porting/parakeet/<variant>/intake.json`.
 
@@ -146,6 +151,14 @@ Allowed statuses: `PASS` | `SKIP — not exposed by runtime` |
 | nemotron-3.5-asr-streaming-0.6b | Batch (offline) | run_batch fast path | `uv run scripts/batch_parity.py --model <gguf> --list <list.txt> --batch-sizes 2,4,8 --backend cpu --language en-US` + `uv run scripts/batch_tensor_parity.py --model <gguf> --wav samples/jfk.wav --batch 4 --backend cpu` | text byte-equal vs serial at sizes 2/4/8 (golden frozen at `tests/golden/batch/nemotron-3.5-asr-streaming-0.6b.cpu.json`); CPU tensor parity bit-exact (max_abs=0.0) at batch=4 on jfk.wav | PASS |
 | ctc-1.1b | Transcribe (CTC head) | explicit en | `build/bin/transcribe-cli -m models/parakeet-ctc-1.1b/parakeet-ctc-1.1b-F32.gguf --language en samples/jfk.wav` | English transcript | PASS |
 | ctc-0.6b | Transcribe (CTC head) | explicit en | `build/bin/transcribe-cli -m models/parakeet-ctc-0.6b/parakeet-ctc-0.6b-F32.gguf --language en samples/jfk.wav` | English transcript | PASS |
+| multitalker-parakeet-streaming-0.6b-v1 | Transcribe (single_speaker_mode, offline / cache-aware att_context_size=[70,13], 1.12s chunk) | explicit en | `build/bin/transcribe-cli -m models/multitalker-parakeet-streaming-0.6b-v1/multitalker-parakeet-streaming-0.6b-v1-F32.gguf --language en samples/jfk.wav` | English transcript with PnC | PASS |
+| multitalker-parakeet-streaming-0.6b-v1 | Transcribe (single_speaker_mode) — no-hint | auto/default | `build/bin/transcribe-cli -m <gguf> samples/jfk.wav` (no `--language`; English-only checkpoint) | English transcript with PnC | PASS |
+| multitalker-parakeet-streaming-0.6b-v1 | Punctuation/casing | output | same as the explicit-en row | output contains capital letters and `,.?!` | PASS |
+| multitalker-parakeet-streaming-0.6b-v1 | Streaming (single_speaker_mode, cache reuse across chunks) | streaming | `build/bin/transcribe-cli -m <gguf> --language en --backend cpu --threads 1 --stream-chunk-ms 1120 --stream-att-right 13 samples/jfk.wav` | byte-equal transcript vs single-speaker one-shot at the default `att_context_size=[70,13]` (1.12s chunk) | PASS |
+| multitalker-parakeet-streaming-0.6b-v1 | Other latency settings ([70,0]/[70,1]/[70,6]/[70,13]) | runtime-selectable att_context_size | `… --stream-chunk-ms 1120 --stream-att-right {0,1,6,13} …` | all four R settings stream a valid single-speaker transcript; R=6/13 byte-equal to one-shot, R=0/1 differ only in trailing punctuation (lower lookahead) | PASS |
+| multitalker-parakeet-streaming-0.6b-v1 | Batch (offline, single_speaker_mode) | run_batch fast path | `uv run scripts/batch_parity.py --model <gguf> --list <list.txt> --batch-sizes 2,4,8 --backend cpu --language en` + `uv run scripts/batch_tensor_parity.py --model <gguf> --wav samples/jfk.wav --batch 4 --backend cpu` | text byte-equal vs serial at sizes 2/4/8 (golden frozen at `tests/golden/batch/multitalker-parakeet-streaming-0.6b-v1.cpu.json`); same-length CPU tensor parity bit-exact (max_abs=0.0) at batch=4 on jfk.wav; **diverse-length** flash tensor parity bit-exact (max_abs=0.0) across arbitrary length mixes; full test-clean batch-8 WER == batch-1 (2.19%) after the causal var-len pre-encode masking fix (see forward-map Deviations) | PASS |
+| multitalker-parakeet-streaming-0.6b-v1 | Multitalker / speaker-attributed ASR (speaker-kernel injection + external Sortformer diarization + multi-instance + SegLST) | multitalker | (no runtime surface today) | per-speaker cpWER vs NeMo `speech_to_text_multitalker_streaming_infer.py` on AMI/CH109/Mixer6 | SKIP — not exposed by runtime (OUT OF SCOPE) — the single-speaker kernel path is implemented, but full speaker attribution still requires (1) a ported streaming Sortformer diarizer, (2) target-driven per-frame kernel masks, (3) N-instance-per-speaker orchestration, and (4) a SegLST speaker-tagged output contract. |
+| multitalker-parakeet-streaming-0.6b-v1 | Speaker diarization (produce speaker turns) | diarization | (not a capability of this checkpoint) | n/a | SKIP — not exposed by runtime (OUT OF SCOPE) — this checkpoint CONSUMES external diarization (Sortformer); it does not produce speaker turns. Brought back in scope only if a Sortformer diarizer is ported as a separate family. |
 | all variants | Word timestamps | only if exposed | `transcribe-cli --timestamps word -m <gguf> <wav>` (any variant) | per-word `t0_ms`/`t1_ms` in JSON output | PASS — derived host-side from emit-frame indices (TDT/RNNT) or per-frame argmax (CTC); same code path as the existing v2/v3 word-timestamp gate, no per-variant differences |
 
 ## Open decisions before Stage 3 (convert)
@@ -182,6 +195,39 @@ be resolved before the corresponding port enters porting-3-convert:
    convert time from `model.cfg.preprocessor.features` inside the
    .nemo archive. Wrong default silently degrades WER without
    changing tensor shapes elsewhere.
+5. **Layer-0 speaker-kernel injection** (`multitalker-parakeet-streaming-0.6b-v1`):
+   RESOLVED at Stage 2. The single-speaker path is NOT numerically identical
+   to `nemotron-speech-streaming-en-0.6b`. `EncDecMultiTalkerRNNTBPEModel`
+   registers a `forward_pre_hook` on `encoder.layers[0]` (from
+   `SpeakerKernelMixin`, `spk_kernel_layers=[0]`) that fires
+   **unconditionally** — even with no diarization / speaker targets. In
+   single-speaker mode the hook computes, at the INPUT of conformer layer 0:
+   `x += spk_kernels.0(x)` (speaker mask defaults to all-ones) then
+   `x += bg_spk_kernels.0(0)` (background mask defaults to zeros → a constant
+   bias vector). Each kernel is an FF block:
+   `Linear(1024,1024) → ReLU → Dropout(id at eval) → Linear(1024,1024)` with
+   bias. Stage 3 MUST export `spk_kernels.0.{0,3}.{weight,bias}` and
+   `bg_spk_kernels.0.{0,3}.{weight,bias}` (2 FF modules) and emit a GGUF KV
+   marking layer-0 injection; Stage 4 MUST apply the injection at the layer-0
+   input. Skipping it silently degrades single-speaker WER (a structural-cfg
+   distinction, not a shape change). The full multitalker path (per-frame
+   diarization mask instead of all-ones, N-instance orchestration, SegLST) is
+   OUT OF SCOPE per Stage 1 — see the multitalker integration brief.
+
+   **Stage 3 resolution (DONE).** `convert-parakeet.py` gates emission on a
+   `spk_kernel_layers` profile key. The 8 source tensors are emitted verbatim
+   (fp32 passthrough) under the loader's `enc.` prefix, keeping the source
+   layer index and the parameter-free `.1`/`.2` Sequential slots implicit:
+   - `spk_kernels.<L>.{0,3}.{weight,bias}` → `enc.spk_kernel.<L>.{0,3}.{weight,bias}`
+   - `bg_spk_kernels.<L>.{0,3}.{weight,bias}` → `enc.bg_spk_kernel.<L>.{0,3}.{weight,bias}`
+
+   Both Linear slots are `[1024,1024]` + `[1024]` bias. The layer-0 injection
+   is marked by three KVs Stage 4 reads:
+   `stt.parakeet.encoder.spk_kernel_type` (`"ff"`),
+   `stt.parakeet.encoder.spk_kernel_layers` (int array, `[0]`), and
+   `stt.parakeet.encoder.add_bg_spk_kernel` (bool, `true`). The loader binds
+   these tensors and applies the injection in both offline and streaming
+   encoder graphs.
 
 ## Tooling: NeMo-aware preflight