feat(crispasr): add word-level timestamp support - #10403
Merged
Merged
Conversation
Add word-level timestamp extraction to the crispasr backend by calling the CrispASR C library's word accessor functions that are already exported by libgocraspasr but were not previously bound by the Go wrapper. Two families of word functions are supported: 1. Session-based (get_word_count/text/t0/t1) — works per-segment for whisper-like backends. 2. Parakeet-specific (get_parakeet_word_count/text/t0/t1) — returns a global word list for TDT/CTC/RNNT parakeet models where the session API does not expose per-segment word data. The Go code tries session-based first and falls back to parakeet-specific when the session word count is zero. Depends on mudler#10402 (grpc server Words forwarding) for the words to reach the HTTP response. Signed-off-by: fqscfqj <fqscfqj@outlook.com>
fqscfqj
force-pushed
the
feat/crispasr-word-timestamps
branch
from
June 19, 2026 10:44
31f2d8b to
b2c8332
Compare
BSD sed requires -i '' for in-place editing while GNU sed uses -i. Replace with -i.bak which works on both platforms, then remove the backup file. Signed-off-by: fqscfqj <fqscfqj@outlook.com>
Contributor
Author
|
Hi! The darwin (macOS Metal) build failure was caused by a pre-existing incompatibility in the Makefile — Fix applied: Replaced The remaining CI jobs (linux/amd64, linux/arm64, CUDA 12/13, ROCm, Vulkan, SYCL) all passed successfully. Could you please approve the workflow run so we can verify the darwin build passes now? Thanks! |
mudler
approved these changes
Jun 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add word-level timestamp extraction to the
crispasrbackend. The CrispASR C library (libgocraspasr) already exports word accessor functions, but the Go wrapper did not bind or call them.What Changed
C shim (
backend/go/crispasr/cpp/crispasr_shim.{h,cpp})Added wrapper functions for two families of CrispASR word accessors:
get_word_count/text/t0/t1(seg_i, ...)get_parakeet_word_count/text/t0/t1(...)Go wrapper (
backend/go/crispasr/{main.go,gocrispasr.go})purego.RegisterLibFunc.AudioTranscription, after transcription, populatesTranscriptSegment.Words:Why two function families?
CrispASR uses different backends internally. Whisper-style backends expose word timestamps per-segment through the session result (
crispasr_session_result_n_words(result, seg_i)). Parakeet TDT models expose word timestamps globally throughcrispasr_parakeet_result_n_words(result)— the session-level word functions return 0 for these models.Depends on
#10402 — the gRPC server wrapper must forward the
Wordsfield for the timestamps to reach the HTTP response.Testing
Verified end-to-end on LocalAI v4.4.3 (Docker, RTX 3090) with
parakeet-tdt-0.6b-v3via thecrispasrbackend:{ "segments": [{ "id": 0, "start": 0.16, "end": 10.48, "text": "Been playing lots of Nordic Souls recently. Enjoy the list, but looking for something graphically better? Things I want to prioritize.", "words": [ {"start": 0.16, "end": 0.4, "text": "Been"}, {"start": 0.4, "end": 0.72, "text": "playing"}, {"start": 0.72, "end": 0.88, "text": "lots"}, {"start": 0.88, "end": 1.04, "text": "of"}, {"start": 1.04, "end": 1.6, "text": "Nordic"}, {"start": 1.6, "end": 2.0, "text": "Souls"}, ... ] }] }21 words returned with accurate timestamps matching the TDT duration head output.