forward-map.md

Forward map - medasr

Reference: transformers.models.lasr.modeling_lasr @ 65dc261512cbdb1ee72b88ae5b222f2605aad8e5 (v5.0.0.dev0) Closest in-tree analog: src/arch/parakeet/ (Conformer + macaron) + src/causal_lm/ (RoPE) + src/arch/sensevoice/ (CTC head)

MedASR is the first in-tree encoder-ctc Conformer family. It does NOT fit build_conformer_block from src/conformer/conformer.h because the upstream block uses load-bearing scaled residuals on every sub-block (macaron [1.5, 0.5], conv [2.0, 1.0]) rather than the standard 0.5 macaron half-step, and the attention path is RoPE rather than relative-position. The block is therefore hand-built in src/arch/medasr/encoder.cpp using the lower-level shared helpers (layer_norm, feed_forward, conv_1d_dw_f32, fused_batch_norm, add_conv_bias) plus a per-family rope_mhsa written from src/causal_lm/'s ggml_rope_ext pattern.

Frontend

Stage	Reference location	Output shape	Gate tensor	ggml / C++ pattern	In-tree analog
log-mel	feature_extraction_lasr.LasrFeatureExtractor.__call__	[T_mel=floor((n_pcm-400)/160)+1, 128] f32	mel.in	host-side transcribe::MelFrontend; LASR knobs require center=false, pad_mode="zero", pre_emphasis=0.0, normalize="none", window=ckpt, filterbank=ckpt, log_compression="log_clamp_1e-5". Step 2 brings up the encoder with TRANSCRIBE_MEL_FROM_REF=<ref_dir> so Step 7 (frontend parity) is the first place the C++ mel actually runs.	src/transcribe-mel.{h,cpp} (extend with center=false knob and log_compression enum)

For jfk.wav (176 400 samples): T_mel = (176 400 − 400)/160 + 1 = 1 099 frames — reference dump rounds to 1 098 (one trailing frame dropped because LASR's manual unfold does len // win floor on the residual; not centered). The forward-map records the formula; bit-exact frame count is a Step 7 parity check.

Encoder

Input: mel ne [T_mel, 128, 1, B] f32 (matches MelFrontend row-major output).

Stage	Reference location	Output shape	Gate tensor	ggml / C++ pattern	In-tree analog
Subsampling: dense_0	LasrEncoderSubsampling.dense_0	[T_mel, 512]	—	mul_mat + bias add	parakeet feed_forward w/o second linear
Subsampling: relu0	F.relu	[T_mel, 512]	—	ggml_relu	n/a
Subsampling: conv_0	Conv1d(512, 512, k=5, s=2, p=0)	[T1=floor((T_mel-5)/2)+1, 512]	—	conv_1d_f32 (kernel [KW=5, IC=512, OC=512], stride 2, pad 0)	conformer conv_1d_f32
Subsampling: conv_0 bias + relu	bias add + F.relu	[T1, 512]	—	add_conv_bias + ggml_relu	conformer
Subsampling: conv_1	Conv1d(512, 256, k=5, s=2, p=0)	[T_enc=floor((T1-5)/2)+1, 256]	—	conv_1d_f32	conformer
Subsampling: conv_1 bias + relu	bias + F.relu	[T_enc, 256]	—	add_conv_bias + ggml_relu	n/a
Subsampling: dense_1	Linear(256, 512) + bias	[T_enc, 512]	enc.subsampling.out	mul_mat + bias add	n/a
Block ×17: norm_ff1	LayerNorm(512, eps=1e-6, bias=False)	[T_enc, 512]	(in post_ff1)	layer_norm(x, w, nullptr) with eps=1e-6	conformer layer_norm (eps hardcoded 1e-5 — see deviation)
Block ×17: ff1	LasrFeedForward = Linear → SiLU → Linear (no biases)	[T_enc, 512]	—	feed_forward(x, lin1_w, nullptr, lin2_w, nullptr)	conformer feed_forward
Block ×17: scaled-residual ff1	x = x*1.5 + ff1*0.5	[T_enc, 512]	enc.block.0.post_ff1 (block 0 only)	ggml_add(ggml_scale(x, 1.5), ggml_scale(ff1, 0.5))	NOT macaron_ff_residual (deviation)
Block ×17: norm_attn	LayerNorm no bias	—	—	layer_norm w/ eps 1e-6	conformer
Block ×17: self-attn Q/K/V	Linear no bias each → reshape [d_model] → [head_dim, n_head]	[head_dim=64, n_head=8, T_enc, B]	—	three mul_mat then reshape_3d+permute for head layout	causal_lm Q/K/V projection
Block ×17: RoPE on Q, K	LasrEncoderRotaryEmbedding (rope_theta=10 000, type=default) applied per-head	unchanged	—	ggml_rope_ext(Q, positions, nullptr, head_dim, GGML_ROPE_TYPE_NEOX, max_pos=10000, rope_theta=10000, 1,0,1,32,1) and same for K	causal_lm RoPE call site (deviation: no MRoPE, no extrapolation)
Block ×17: scaled-dot attn	softmax(QK/√d_k)·V, no mask	[head_dim, n_head, T_enc, B]	—	flash_attn_ext or manual mul_mat+soft_max	causal_lm
Block ×17: o_proj	Linear no bias	[T_enc, 512]	—	reshape back + mul_mat	causal_lm
Block ×17: residual attn	x = x + attn (UNSCALED)	[T_enc, 512]	enc.block.0.post_attn	ggml_add	standard
Block ×17: norm_conv	LayerNorm no bias	—	—	layer_norm	conformer
Block ×17: conv module	Conv1d pw1(512→1024,k=1) → GLU → Conv1d dw(512,k=32,p=15,g=512) → BatchNorm1d(512) → SiLU → Conv1d pw2(512→512,k=1)	[T_enc, 512]	—	mul_mat + ggml_swiglu + conv_1d_dw_f32 + fused_batch_norm + silu + mul_mat. Depthwise uses non-centered pad=15 (left=15, right=16 to keep T identical with k=32 even kernel) — verify against reference.	conformer conv_module (cannot reuse — that helper bakes in the unscaled x + conv residual; we need a custom scaled residual outside)
Block ×17: scaled-residual conv	x = x*2.0 + conv*1.0	[T_enc, 512]	enc.block.0.post_conv	ggml_add(ggml_scale(x, 2.0), conv)	deviation
Block ×17: norm_ff2 + ff2	same shape as FF1	[T_enc, 512]	—	mirrors FF1	conformer
Block ×17: scaled-residual ff2	x = x*1.5 + ff2*0.5	[T_enc, 512]	enc.block.0.post_ff2	ggml_add(ggml_scale(x, 1.5), ggml_scale(ff2, 0.5))	deviation
Block ×17: norm_out	LayerNorm no bias	[T_enc, 512]	enc.block.{0,7,8,16}.out	layer_norm w/ eps 1e-6	conformer
enc.out_norm	top-level LayerNorm no bias	[T_enc, 512]	enc.out_norm.out	layer_norm	conformer

Decoder

n/a — non-autoregressive CTC head.

Stage	Reference location	Output shape	Gate tensor	ggml / C++ pattern	In-tree analog
CTC head	LasrCtcHead.ctc_head = Conv1d(512, 512, k=1) + bias	[vocab=512, T_enc, 1, B] (note [vocab, T] order)	enc.ctc_logits	ggml_mul_mat(W_ctc, x) + b_ctc (k=1 conv = mul_mat)	sensevoice ctc_head
Greedy decode	argmax + collapse-repeats + drop-blanks (blank=0)	[N_tokens] host ints	—	host-side loop on logits	adapt parakeet decode_ctc_greedy (blank=last-id → blank=0)
Detokenize	LasrTokenizer.batch_decode (SP) — keeps </s> id 2 in stream	string	transcript.json	shared Tokenizer::decode_ctc strips blanks pre-call	shared

Generation / KV Path

n/a — encoder-only CTC; no autoregressive state. run_batch fast path batches utterances via the existing batch-axis (ne[3]) of the encoder graph, identical to parakeet's CTC variant.

Capabilities And Language Controls

Capability	Reference behavior	C++ API behavior	Family-doc Capability Validation row
English transcribe	generate greedy CTC	same	"Transcribe / explicit en" + "Transcribe / auto" — both MUST PASS
Language detection	n/a (English-only)	reject --language != en with OK fallback to en	SKIP — not advertised
Translation	n/a	not exposed	SKIP — not advertised
Timestamps	n/a (CTC has none)	not exposed	SKIP — architecture has no timestamp head
Streaming	n/a (offline only)	not exposed	SKIP — capabilities.streaming=false
Batch (offline)	HF processor pads to common T_mel	implement run_batch via ne[3] batch axis on encoder graph (mirror parakeet CTC pattern); per-utterance valid-frame masks via attn_pad_mask / conv_pad_mask on each block	MUST PASS — text byte-identical, CPU tensor bit-exact at bs 2/4/8
KenLM beam decode	shipped but not Stage-4 obligation	not exposed	SKIP — OUT OF SCOPE (first ship is greedy)

Deviations From Closest Analog

• Macaron FF residual scalars are [1.5, 0.5], NOT the standard 0.5. The shared transcribe::conformer::macaron_ff_residual helper hardcodes x + 0.5 * FF(LN(x)) and is not reusable here. medasr hand-builds the FF1/FF2 residual as ggml_add(ggml_scale(x, 1.5), ggml_scale(FF(LN(x)), 0.5)).
• Conv module residual scalars are [2.0, 1.0]. transcribe::conformer::conv_module itself is callable for the inner pw1+GLU+dw+BN+SiLU+pw2 sub-block, but the surrounding scaled residual is hand-built (ggml_add(ggml_scale(x, 2.0), conv)).
• Attention is RoPE, not relative-position. transcribe::conformer::rel_pos_mhsa and build_conformer_block are not used. The Q/K projection + ggml_rope_ext(..., GGML_ROPE_TYPE_NEOX, ...) + softmax(QK/√d)·V + o_proj is implemented per-block in src/arch/medasr/encoder.cpp using the causal_lm pattern at src/causal_lm/causal_lm.cpp:234-245. rope_theta=10 000, max_position_embeddings=10 000.
• layer_norm eps is 1e-6, not the conformer-helper-hardcoded 1e-5. The shared helper uses ggml_norm with constant kLayerNormEps = 1e-5f. medasr loads enc.layer_norm_eps from the GGUF (stt.medasr.encoder.layer_norm_eps) and passes it through a local medasr_layer_norm shim that takes eps as a runtime argument.
• All encoder LayerNorms are bias=False. Every _b slot on the medasr block view stays nullptr; layer_norm is called with beta=nullptr. Same null-bias treatment as parakeet's FFN/attn linears; just applied to LN this time.
• Subsampling is 1-D, not 2-D. LasrEncoderSubsampling = Linear(128→512) → ReLU → Conv1d(512,512,k=5,s=2) → ReLU → Conv1d(512,256,k=5,s=2) → ReLU → Linear(256→512). The leading Linear(128→512) reduces the mel dim to d_model BEFORE the convs, so the convs operate on [T, 512] channels (not [F=128, T] 2-D as in parakeet's DwStridingSubsampling). Two stride-2 convs → 4x effective downsampling. transcribe::conformer::build_pre_encode is not reusable; medasr writes its own build_subsampling in encoder.cpp using conv_1d_f32.
• CTC head is Conv1d(512→512, k=1) with bias, not Linear no-bias. Numerically equivalent to a linear projection but stored as Conv1d. The converter normalizes the tensor name to ctc.proj.weight (transposed if necessary to mul_mat-friendly layout); C++ runs ggml_add(ggml_mul_mat(W, x), b).
• CTC blank id = 0 (<epsilon>), NOT vocab_size - 1. Parakeet/sensevoice convention is blank = vocab_size - 1. medasr reads stt.medasr.ctc.blank_id from the GGUF and uses it in greedy collapse. Hard-coded last_id assumptions in the shared CTC decoder are NOT reused — medasr ships its own greedy collapse in decode_ctc.cpp (or inlined in model.cpp::run).
• Tokenizer specials match SentencePiece ids 0..3 (<epsilon>, <s>, </s>, <unk>). The reference WER scorer (scripts/wer/run_reference_medasr_transformers.py) uses processor.batch_decode(token_ids, skip_special_tokens=True). The validate.py reference dumper does the same (changed from the README's non-skipping variant, see commit), so both reference paths now agree. The C++ greedy CTC collapse mirrors this by skipping ids 1 (<s>), 2 (</s>), and 3 (<unk>) in addition to the blank id 0. Without this skip, the CTC head's occasional last-frame </s> prediction leaks the literal string into the transcript and lifts dataset WER by ~0.2pp on LibriSpeech test-clean.
• Conv-module depthwise dispatch is direct_dw_in_block=true (NOT the im2col path). LASR's conv_kernel=32 "same" padding is asymmetric (15, 16); the conformer-helper's asymmetric-pad path uses ggml_fill + ggml_concat per side, which is NOT batch-stable in ggml CPU (drift ~1.4e-6 at the encoder output between B=1 and B>1). The direct depthwise op (ggml_conv_2d_dw_direct) handles the same kernel + symmetric inner-padding shape and is bit-exact across batch sizes. The trade-off is a Metal note: direct depthwise on k=32 has not been profiled on Apple GPU and may need a fallback there; CPU + Vulkan + CUDA are fine.

Variant Notes

• medasr: single variant; no per-variant deviations. The variants list in intake.json has length 1. config.varying_across_variants is empty.