fix(grpc): forward word-level timestamps in AudioTranscription wrapper - #10402
Merged
Conversation
The gRPC server wrapper in pkg/grpc/server.go reconstructs TranscriptSegment messages when relaying AudioTranscription results from backends. The Words field was not being copied, causing all word-level timestamps to be silently dropped regardless of backend support. This was introduced when PR mudler#9621 added the TranscriptWord proto message and transcriptResultFromProto (server-side), but did not update the server-side gRPC relay to forward the new field. Fixes mudler#9306 Signed-off-by: fqscfqj <fqscfqj@outlook.com>
fqscfqj
force-pushed
the
fix/grpc-forward-words
branch
from
June 19, 2026 10:44
b4b0eba to
8d992b2
Compare
fqscfqj
added a commit
to fqscfqj/LocalAI
that referenced
this pull request
Jun 19, 2026
Add word-level timestamp extraction to the crispasr backend by calling the CrispASR C library's word accessor functions that are already exported by libgocraspasr but were not previously bound by the Go wrapper. Two families of word functions are supported: 1. Session-based (get_word_count/text/t0/t1) — works per-segment for whisper-like backends. 2. Parakeet-specific (get_parakeet_word_count/text/t0/t1) — returns a global word list for TDT/CTC/RNNT parakeet models where the session API does not expose per-segment word data. The Go code tries session-based first and falls back to parakeet-specific when the session word count is zero. Depends on mudler#10402 (grpc server Words forwarding) for the words to reach the HTTP response. Signed-off-by: fqscfqj <fqscfqj@outlook.com>
mudler
approved these changes
Jun 19, 2026
mudler
pushed a commit
that referenced
this pull request
Jun 19, 2026
* feat(crispasr): add word-level timestamp support Add word-level timestamp extraction to the crispasr backend by calling the CrispASR C library's word accessor functions that are already exported by libgocraspasr but were not previously bound by the Go wrapper. Two families of word functions are supported: 1. Session-based (get_word_count/text/t0/t1) — works per-segment for whisper-like backends. 2. Parakeet-specific (get_parakeet_word_count/text/t0/t1) — returns a global word list for TDT/CTC/RNNT parakeet models where the session API does not expose per-segment word data. The Go code tries session-based first and falls back to parakeet-specific when the session word count is zero. Depends on #10402 (grpc server Words forwarding) for the words to reach the HTTP response. Signed-off-by: fqscfqj <fqscfqj@outlook.com> * fix(crispasr): use portable sed -i.bak for macOS compatibility BSD sed requires -i '' for in-place editing while GNU sed uses -i. Replace with -i.bak which works on both platforms, then remove the backup file. Signed-off-by: fqscfqj <fqscfqj@outlook.com> --------- Signed-off-by: fqscfqj <fqscfqj@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The gRPC server wrapper in
pkg/grpc/server.goreconstructsTranscriptSegmentmessages when relayingAudioTranscriptionresults from backends to the HTTP layer. TheWordsfield was not being copied, causing all word-level timestamps to be silently dropped — regardless of which backend produced them.This explains why:
wordsis alwaysNoneinverbose_jsonoutputfaster-whisper,crispasr,qwen-asr, andnemothat populate word-level timestamps never actually surface them through the APIRoot Cause
PR #9621 added the
TranscriptWordproto message andtranscriptResultFromProto(which readss.Wordson the server/HTTP side), but did not update the gRPC server wrapper (pkg/grpc/server.go) to forward theWordsfield when relaying results from the backend process.Fix
Copy
Wordsfrom the backend result into the gRPC responseTranscriptSegment, converting eachschema.TranscriptionWordto*proto.TranscriptWord.Testing
Verified on a real deployment (LocalAI v4.4.3, Docker, RTX 3090) with the
crispasrbackend (parakeet-tdt-0.6b-v3):Before (no words):
{"segments": [{"id": 0, "start": 0.16, "end": 10.48, "text": "Been playing lots of Nordic Souls recently."}]}After (21 words):
{"segments": [{"id": 0, "start": 0.16, "end": 10.48, "text": "...", "words": [ {"start": 0.16, "end": 0.4, "text": "Been"}, {"start": 0.4, "end": 0.72, "text": "playing"}, ... ]}]}Fixes #9306