Skip to content

Commit 181ebb6

Browse files
mudlermudler
authored
feat: voice recognition (#9500)
* feat(voice-recognition): add /v1/voice/{verify,analyze,embed} + speaker-recognition backend Audio analog to face recognition. Adds three gRPC RPCs (VoiceVerify / VoiceAnalyze / VoiceEmbed), their Go service and HTTP layers, a new FLAG_SPEAKER_RECOGNITION capability flag, and a Python backend scaffold under backend/python/speaker-recognition/ wrapping SpeechBrain ECAPA-TDNN with a parallel OnnxDirectEngine for WeSpeaker / 3D-Speaker ONNX exports. The kokoros Rust backend gets matching unimplemented trait stubs — tonic's async_trait has no defaults, so adding an RPC without Rust stubs breaks the build (same regression fixed by eb01c77 for face). Swagger, /api/instructions, and the auth RouteFeatureRegistry / APIFeatures list are updated so the endpoints surface everywhere a client or admin UI looks. Assisted-by: Claude:claude-opus-4-7 * feat(voice-recognition): add 1:N identify + register/forget endpoints Mirrors the face-recognition register/identify/forget surface. New package core/services/voicerecognition/ carries a Registry interface and a local-store-backed implementation (same in-memory vector-store plumbing facerecognition uses, separate instance so the embedding spaces stay isolated). Handlers under /v1/voice/{register,identify,forget} reuse backend.VoiceEmbed to compute the probe vector, then delegate the nearest-neighbour search to the registry. Default cosine-distance threshold is tuned for ECAPA-TDNN on VoxCeleb (0.25, EER ~1.9%). As with the face registry, the current backing is in-memory only — a pgvector implementation is a future constructor-level swap. Assisted-by: Claude:claude-opus-4-7 * feat(voice-recognition): gallery, docs, CI and e2e coverage - backend/index.yaml: speaker-recognition backend entry + CPU and CUDA-12 image variants (plus matching development variants). - gallery/index.yaml: speechbrain-ecapa-tdnn (default) and wespeaker-resnet34 model entries. The WeSpeaker SHA-256 is a deliberate placeholder — the HF URI must be curl'd and its hash filled in before the entry installs. - docs/content/features/voice-recognition.md: API reference + quickstart, mirrors the face-recognition docs. - React UI: CAP_SPEAKER_RECOGNITION flag export (consumers follow face's precedent — no dedicated tab yet). - tests/e2e-backends: voice_embed / voice_verify / voice_analyze specs. Helper resolveFaceFixture is reused as-is — the only thing face/voice share is "download a file into workDir", so no need for a new helper. - Makefile: docker-build-speaker-recognition + test-extra-backend- speaker-recognition-{ecapa,all} targets. Audio fixtures default to VCTK p225/p226 samples from HuggingFace. - CI: test-extra.yml grows a tests-speaker-recognition-grpc job mirroring insightface. backend.yml matrix gains CPU + CUDA-12 image build entries — scripts/changed-backends.js auto-picks these up. Assisted-by: Claude:claude-opus-4-7 * feat(voice-recognition): wire a working /v1/voice/analyze head Adds AnalysisHead: a lazy-loading age / gender / emotion inference wrapper that plugs into both SpeechBrainEngine and OnnxDirectEngine. Defaults to two open-licence HuggingFace checkpoints: - audeering/wav2vec2-large-robust-24-ft-age-gender (Apache 2.0) — age regression + 3-way gender (female / male / child). - superb/wav2vec2-base-superb-er (Apache 2.0) — 4-way emotion. Both are optional and degrade gracefully when transformers or the model can't be loaded — the engine raises NotImplementedError so the gRPC layer returns 501 instead of a generic 500. Emotion classes pass through from the model (neutral/happy/angry/sad on the default checkpoint); the e2e test now accepts any non-empty dominant gender so custom age_gender_model overrides don't fail it. Adds transformers to the backend's CPU and CUDA-12 requirements. Assisted-by: Claude:claude-opus-4-7 * fix(voice-recognition): pin real WeSpeaker ResNet34 ONNX SHA-256 Replaces the placeholder hash in gallery/index.yaml with the actual SHA-256 (7bb2f06e…) of the upstream Wespeaker/wespeaker-voxceleb-resnet34-LM ONNX at ~25MB. `local-ai models install wespeaker-resnet34` now succeeds. Assisted-by: Claude:claude-opus-4-7 * fix(voice-recognition): soundfile loader + honest analyze default Two issues surfaced on first end-to-end smoke with the actual backend image: 1. torchaudio.load in torchaudio 2.8+ requires the torchcodec package for audio decoding. Switch SpeechBrainEngine._load_waveform to the already-present soundfile (listed in requirements.txt) plus a numpy linear resample to 16kHz. Drops a heavy ffmpeg-linked dep and the codepath we never exercise (torchaudio's ffmpeg backend). 2. The AnalysisHead was defaulting to audeering/wav2vec2-large-robust- 24-ft-age-gender, but AutoModelForAudioClassification silently mangles that checkpoint — it reports the age head weights as UNEXPECTED and re-initialises the classifier head with random values, so the "gender" output is noise and there is no age output at all. Make age/gender opt-in instead (empty default; users wire a cleanly-loadable Wav2Vec2ForSequenceClassification checkpoint via age_gender_model: option). Emotion keeps its working Superb default. Also broaden _infer_age_gender's tensor-shape handling and catch runtime exceptions so a dodgy age/gender head never takes down the whole analyze call. Docs and README updated to match the new policy. Verified with the branch-scoped gallery on localhost: - voice/embed → 192-d ECAPA-TDNN vector - voice/verify → same-clip dist≈6e-08 verified=true; cross-speaker dist 0.76–0.99 verified=false (as expected) - voice/register/identify/forget → round-trip works, 404 on unknown id - voice/analyze → emotion populated, age/gender omitted (opt-in) Assisted-by: Claude:claude-opus-4-7 * fix(voice-recognition): real CI audio fixtures + fixture-agnostic verify spec Two issues surfaced after CI actually ran the speaker-recognition e2e target (I'd curl-tested against a running server but hadn't run the make target locally): 1. The default BACKEND_TEST_VOICE_AUDIO_* URLs pointed at huggingface.co/datasets/CSTR-Edinburgh/vctk paths that return 404 (the dataset is gated). Swap them for the speechbrain test samples served from github.com/speechbrain/speechbrain/raw/develop/ — public, no auth, correct 16kHz mono format. 2. The VoiceVerify spec required d(file1,file2) < 0.4, assuming file1/file2 were same-speaker. The speechbrain samples are three different speakers (example1/2/5), and there is no easy un-gated source of true same-speaker audio pairs (VoxCeleb/VCTK/LibriSpeech are all license- or size-gated for CI use). Replace the ceiling check with a relative-ordering assertion: d(pair) > d(same-clip) for both file2 and file3 — that's enough to prove the embeddings encode speaker info, and it works with any three non-identical clips. Actual speaker ordering d(1,2) vs d(1,3) is logged but not asserted. Local run: 4/4 voice specs pass (Health, LoadModel, VoiceEmbed, VoiceVerify) on the built backend image. 12 non-voice specs skipped as expected. Assisted-by: Claude:claude-opus-4-7 * fix(ci): checkout with submodules in the reusable backend_build workflow The kokoros Rust backend build fails with failed to read .../sources/Kokoros/kokoros/Cargo.toml: No such file because the reusable backend_build.yml workflow's actions/checkout step was missing `submodules: true`. Dockerfile.rust does `COPY . /LocalAI`, and without the submodule files the subsequent `cargo build` can't find the vendored Kokoros crate. The bug pre-dates this PR — scripts/changed-backends.js only triggers the kokoros image job when something under backend/rust/kokoros or the shared proto changes, so master had been coasting past it. The voice-recognition proto addition re-broke it. Other checkouts in backend.yml (llama-cpp-darwin) and test-extra.yml (insightface, kokoros, speaker-recognition) already pass `submodules: true`; this brings the shared backend image builder in line. Assisted-by: Claude:claude-opus-4-7
1 parent 1c59165 commit 181ebb6

53 files changed

Lines changed: 3747 additions & 6 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/backend.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -724,6 +724,19 @@ jobs:
724724
dockerfile: "./backend/Dockerfile.python"
725725
context: "./"
726726
ubuntu-version: '2404'
727+
- build-type: 'cublas'
728+
cuda-major-version: "12"
729+
cuda-minor-version: "8"
730+
platforms: 'linux/amd64'
731+
tag-latest: 'auto'
732+
tag-suffix: '-gpu-nvidia-cuda-12-speaker-recognition'
733+
runs-on: 'ubuntu-latest'
734+
base-image: "ubuntu:24.04"
735+
skip-drivers: 'false'
736+
backend: "speaker-recognition"
737+
dockerfile: "./backend/Dockerfile.python"
738+
context: "./"
739+
ubuntu-version: '2404'
727740
- build-type: 'cublas'
728741
cuda-major-version: "12"
729742
cuda-minor-version: "8"
@@ -2653,6 +2666,20 @@ jobs:
26532666
dockerfile: "./backend/Dockerfile.python"
26542667
context: "./"
26552668
ubuntu-version: '2404'
2669+
# speaker-recognition (voice/speaker biometrics)
2670+
- build-type: ''
2671+
cuda-major-version: ""
2672+
cuda-minor-version: ""
2673+
platforms: 'linux/amd64,linux/arm64'
2674+
tag-latest: 'auto'
2675+
tag-suffix: '-cpu-speaker-recognition'
2676+
runs-on: 'ubuntu-latest'
2677+
base-image: "ubuntu:24.04"
2678+
skip-drivers: 'false'
2679+
backend: "speaker-recognition"
2680+
dockerfile: "./backend/Dockerfile.python"
2681+
context: "./"
2682+
ubuntu-version: '2404'
26562683
- build-type: 'intel'
26572684
cuda-major-version: ""
26582685
cuda-minor-version: ""

‎.github/workflows/backend_build.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,8 @@ jobs:
108108
109109
- name: Checkout
110110
uses: actions/checkout@v6
111+
with:
112+
submodules: true
111113

112114
- name: Release space from worker
113115
if: inputs.runs-on == 'ubuntu-latest'

‎.github/workflows/test-extra.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@ jobs:
3939
voxtral: ${{ steps.detect.outputs.voxtral }}
4040
kokoros: ${{ steps.detect.outputs.kokoros }}
4141
insightface: ${{ steps.detect.outputs.insightface }}
42+
speaker-recognition: ${{ steps.detect.outputs.speaker-recognition }}
4243
steps:
4344
- name: Checkout repository
4445
uses: actions/checkout@v6
@@ -778,3 +779,29 @@ jobs:
778779
- name: Build insightface backend image and run both model configurations
779780
run: |
780781
make test-extra-backend-insightface-all
782+
tests-speaker-recognition-grpc:
783+
needs: detect-changes
784+
if: needs.detect-changes.outputs.speaker-recognition == 'true' || needs.detect-changes.outputs.run-all == 'true'
785+
runs-on: ubuntu-latest
786+
timeout-minutes: 90
787+
steps:
788+
- name: Clone
789+
uses: actions/checkout@v6
790+
with:
791+
submodules: true
792+
- name: Dependencies
793+
run: |
794+
sudo apt-get update
795+
sudo apt-get install -y --no-install-recommends \
796+
make build-essential curl ca-certificates git tar
797+
- name: Setup Go
798+
uses: actions/setup-go@v5
799+
with:
800+
go-version: '1.26.0'
801+
- name: Free disk space
802+
run: |
803+
sudo rm -rf /usr/share/dotnet /opt/ghc /usr/local/lib/android /opt/hostedtoolcache/CodeQL || true
804+
df -h
805+
- name: Build speaker-recognition backend image and run the ECAPA-TDNN configuration
806+
run: |
807+
make test-extra-backend-speaker-recognition-all

‎Makefile‎

Lines changed: 41 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Disable parallel execution for backend builds
2-
.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/insightface backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/tinygrad
2+
.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/tinygrad
33

44
GOCMD=go
55
GOTEST=$(GOCMD) test
@@ -435,6 +435,7 @@ prepare-test-extra: protogen-python
435435
$(MAKE) -C backend/python/trl
436436
$(MAKE) -C backend/python/tinygrad
437437
$(MAKE) -C backend/python/insightface
438+
$(MAKE) -C backend/python/speaker-recognition
438439
$(MAKE) -C backend/rust/kokoros kokoros-grpc
439440

440441
test-extra: prepare-test-extra
@@ -459,6 +460,7 @@ test-extra: prepare-test-extra
459460
$(MAKE) -C backend/python/trl test
460461
$(MAKE) -C backend/python/tinygrad test
461462
$(MAKE) -C backend/python/insightface test
463+
$(MAKE) -C backend/python/speaker-recognition test
462464
$(MAKE) -C backend/rust/kokoros test
463465

464466
##
@@ -713,6 +715,41 @@ test-extra-backend-insightface-all: \
713715
test-extra-backend-insightface-buffalo-sc \
714716
test-extra-backend-insightface-opencv
715717

718+
## speaker-recognition — voice (speaker) biometrics.
719+
##
720+
## Audio fixtures default to the speechbrain test samples served
721+
## straight from their GitHub repo — public, no auth needed, and they
722+
## ship as 16kHz mono WAV/FLAC which is exactly what the engine wants.
723+
## example{1,2,5} are three different speakers; the suite treats
724+
## example1 as the "same-image twin" probe (verify(clip, clip) must
725+
## return distance≈0) and the other two as cross-speaker ceilings.
726+
## Override with BACKEND_TEST_VOICE_AUDIO_{1,2,3}_FILE for offline runs.
727+
VOICE_AUDIO_1_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example1.wav
728+
VOICE_AUDIO_2_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example2.flac
729+
VOICE_AUDIO_3_URL ?= https://github.com/speechbrain/speechbrain/raw/develop/tests/samples/single-mic/example5.wav
730+
731+
## ECAPA-TDNN via SpeechBrain — default CI configuration. Auto-downloads
732+
## the checkpoint from HuggingFace on first LoadModel (bundled in the
733+
## backend image pip install). 192-d embeddings, cosine-distance based.
734+
## The e2e suite drives LoadModel directly so we don't rely on LocalAI's
735+
## gallery flow here.
736+
test-extra-backend-speaker-recognition-ecapa: docker-build-speaker-recognition
737+
BACKEND_IMAGE=local-ai-backend:speaker-recognition \
738+
BACKEND_TEST_MODEL_NAME=speechbrain/spkrec-ecapa-voxceleb \
739+
BACKEND_TEST_OPTIONS=engine:speechbrain,source:speechbrain/spkrec-ecapa-voxceleb \
740+
BACKEND_TEST_CAPS=health,load,voice_embed,voice_verify \
741+
BACKEND_TEST_VOICE_AUDIO_1_URL=$(VOICE_AUDIO_1_URL) \
742+
BACKEND_TEST_VOICE_AUDIO_2_URL=$(VOICE_AUDIO_2_URL) \
743+
BACKEND_TEST_VOICE_AUDIO_3_URL=$(VOICE_AUDIO_3_URL) \
744+
BACKEND_TEST_VOICE_VERIFY_DISTANCE_CEILING=0.4 \
745+
$(MAKE) test-extra-backend
746+
747+
## Aggregate — today there's only one voice config; the target exists
748+
## so the CI workflow matches the insightface-all naming convention and
749+
## can grow to include WeSpeaker / 3D-Speaker later.
750+
test-extra-backend-speaker-recognition-all: \
751+
test-extra-backend-speaker-recognition-ecapa
752+
716753
## sglang mirrors the vllm setup: HuggingFace model id, same tiny Qwen,
717754
## tool-call extraction via sglang's native qwen parser. CPU builds use
718755
## sglang's upstream pyproject_cpu.toml recipe (see backend/python/sglang/install.sh).
@@ -859,6 +896,7 @@ BACKEND_FASTER_WHISPER = faster-whisper|python|.|false|true
859896
BACKEND_COQUI = coqui|python|.|false|true
860897
BACKEND_RFDETR = rfdetr|python|.|false|true
861898
BACKEND_INSIGHTFACE = insightface|python|.|false|true
899+
BACKEND_SPEAKER_RECOGNITION = speaker-recognition|python|.|false|true
862900
BACKEND_KITTEN_TTS = kitten-tts|python|.|false|true
863901
BACKEND_NEUTTS = neutts|python|.|false|true
864902
BACKEND_KOKORO = kokoro|python|.|false|true
@@ -931,6 +969,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_FASTER_WHISPER)))
931969
$(eval $(call generate-docker-build-target,$(BACKEND_COQUI)))
932970
$(eval $(call generate-docker-build-target,$(BACKEND_RFDETR)))
933971
$(eval $(call generate-docker-build-target,$(BACKEND_INSIGHTFACE)))
972+
$(eval $(call generate-docker-build-target,$(BACKEND_SPEAKER_RECOGNITION)))
934973
$(eval $(call generate-docker-build-target,$(BACKEND_KITTEN_TTS)))
935974
$(eval $(call generate-docker-build-target,$(BACKEND_NEUTTS)))
936975
$(eval $(call generate-docker-build-target,$(BACKEND_KOKORO)))
@@ -965,7 +1004,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SAM3_CPP)))
9651004
docker-save-%: backend-images
9661005
docker save local-ai-backend:$* -o backend-images/$*.tar
9671006

968-
docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-qwen3-tts-cpp docker-build-insightface
1007+
docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-qwen3-tts-cpp docker-build-insightface docker-build-speaker-recognition
9691008

9701009
########################################################
9711010
### Mock Backend for E2E Tests

‎backend/backend.proto‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,9 @@ service Backend {
2626
rpc Detect(DetectOptions) returns (DetectResponse) {}
2727
rpc FaceVerify(FaceVerifyRequest) returns (FaceVerifyResponse) {}
2828
rpc FaceAnalyze(FaceAnalyzeRequest) returns (FaceAnalyzeResponse) {}
29+
rpc VoiceVerify(VoiceVerifyRequest) returns (VoiceVerifyResponse) {}
30+
rpc VoiceAnalyze(VoiceAnalyzeRequest) returns (VoiceAnalyzeResponse) {}
31+
rpc VoiceEmbed(VoiceEmbedRequest) returns (VoiceEmbedResponse) {}
2932

3033
rpc StoresSet(StoresSetOptions) returns (Result) {}
3134
rpc StoresDelete(StoresDeleteOptions) returns (Result) {}
@@ -528,6 +531,57 @@ message FaceAnalyzeResponse {
528531
repeated FaceAnalysis faces = 1;
529532
}
530533

534+
// --- Voice (speaker) recognition messages ---
535+
//
536+
// Analogous to the Face* messages above, but for speaker biometrics.
537+
// Audio fields accept a filesystem path (same convention as
538+
// TranscriptRequest.dst). The HTTP layer materialises base64 / URL /
539+
// data-URI inputs to a temp file before calling the gRPC backend.
540+
541+
message VoiceVerifyRequest {
542+
string audio1 = 1; // path to first audio clip
543+
string audio2 = 2; // path to second audio clip
544+
float threshold = 3; // cosine-distance threshold; 0 = use backend default
545+
bool anti_spoofing = 4; // reserved for future AASIST bolt-on
546+
}
547+
548+
message VoiceVerifyResponse {
549+
bool verified = 1;
550+
float distance = 2; // 1 - cosine_similarity
551+
float threshold = 3;
552+
float confidence = 4; // 0-100
553+
string model = 5; // e.g. "speechbrain/spkrec-ecapa-voxceleb"
554+
float processing_time_ms = 6;
555+
}
556+
557+
message VoiceAnalyzeRequest {
558+
string audio = 1; // path to audio clip
559+
repeated string actions = 2; // subset of ["age","gender","emotion"]; empty = all-supported
560+
}
561+
562+
message VoiceAnalysis {
563+
float start = 1; // segment start time in seconds (0 if single-utterance)
564+
float end = 2; // segment end time in seconds
565+
float age = 3;
566+
string dominant_gender = 4;
567+
map<string, float> gender = 5;
568+
string dominant_emotion = 6;
569+
map<string, float> emotion = 7;
570+
}
571+
572+
message VoiceAnalyzeResponse {
573+
repeated VoiceAnalysis segments = 1;
574+
}
575+
576+
message VoiceEmbedRequest {
577+
string audio = 1; // path to audio clip
578+
}
579+
580+
message VoiceEmbedResponse {
581+
repeated float embedding = 1;
582+
string model = 2;
583+
}
584+
531585
message ToolFormatMarkers {
532586
string format_type = 1; // "json_native", "tag_with_json", "tag_with_tagged"
533587

‎backend/index.yaml‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3773,3 +3773,64 @@
37733773
uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-insightface"
37743774
mirrors:
37753775
- localai/localai-backends:master-gpu-nvidia-cuda-12-insightface
3776+
3777+
# speaker-recognition (voice/speaker biometrics) — Apache-2.0 stack
3778+
- &speakerrecognition
3779+
name: "speaker-recognition"
3780+
alias: "speaker-recognition"
3781+
# SpeechBrain is Apache-2.0. WeSpeaker / 3D-Speaker ONNX exports are
3782+
# Apache-2.0. The backend itself ships only Python deps — all model
3783+
# weights flow through LocalAI's gallery download mechanism (or
3784+
# SpeechBrain's built-in HF auto-download at first LoadModel).
3785+
license: apache-2.0
3786+
description: |
3787+
Speaker (voice) recognition backend — the audio analog to
3788+
insightface. Wraps SpeechBrain ECAPA-TDNN (default engine, 192-d
3789+
embeddings, ~1.9% EER on VoxCeleb) plus an OnnxDirectEngine for
3790+
pre-exported WeSpeaker / 3D-Speaker ONNX models.
3791+
3792+
Exposes speaker verification (/v1/voice/verify), speaker embedding
3793+
(/v1/voice/embed), speaker analysis (/v1/voice/analyze), and 1:N
3794+
speaker identification (/v1/voice/{register,identify,forget}).
3795+
Registrations use LocalAI's built-in vector store — same in-memory
3796+
backing the face-recognition registry uses, separate instance.
3797+
urls:
3798+
- https://speechbrain.github.io/
3799+
- https://github.com/wenet-e2e/wespeaker
3800+
- https://github.com/modelscope/3D-Speaker
3801+
tags:
3802+
- voice-recognition
3803+
- speaker-verification
3804+
- speaker-embedding
3805+
- gpu
3806+
- cpu
3807+
capabilities:
3808+
default: "cpu-speaker-recognition"
3809+
nvidia: "cuda12-speaker-recognition"
3810+
nvidia-cuda-12: "cuda12-speaker-recognition"
3811+
- !!merge <<: *speakerrecognition
3812+
name: "speaker-recognition-development"
3813+
capabilities:
3814+
default: "cpu-speaker-recognition-development"
3815+
nvidia: "cuda12-speaker-recognition-development"
3816+
nvidia-cuda-12: "cuda12-speaker-recognition-development"
3817+
- !!merge <<: *speakerrecognition
3818+
name: "cpu-speaker-recognition"
3819+
uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-speaker-recognition"
3820+
mirrors:
3821+
- localai/localai-backends:latest-cpu-speaker-recognition
3822+
- !!merge <<: *speakerrecognition
3823+
name: "cuda12-speaker-recognition"
3824+
uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-speaker-recognition"
3825+
mirrors:
3826+
- localai/localai-backends:latest-gpu-nvidia-cuda-12-speaker-recognition
3827+
- !!merge <<: *speakerrecognition
3828+
name: "cpu-speaker-recognition-development"
3829+
uri: "quay.io/go-skynet/local-ai-backends:master-cpu-speaker-recognition"
3830+
mirrors:
3831+
- localai/localai-backends:master-cpu-speaker-recognition
3832+
- !!merge <<: *speakerrecognition
3833+
name: "cuda12-speaker-recognition-development"
3834+
uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-speaker-recognition"
3835+
mirrors:
3836+
- localai/localai-backends:master-gpu-nvidia-cuda-12-speaker-recognition

‎backend/python/speaker-recognition/Makefile‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
.DEFAULT_GOAL := install
2+
3+
.PHONY: install
4+
install:
5+
bash install.sh
6+
7+
.PHONY: protogen-clean
8+
protogen-clean:
9+
$(RM) backend_pb2_grpc.py backend_pb2.py
10+
11+
.PHONY: clean
12+
clean: protogen-clean
13+
rm -rf venv __pycache__

‎backend/python/speaker-recognition/README.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# speaker-recognition
2+
3+
Speaker (voice) recognition backend for LocalAI. The audio analog to
4+
`insightface` — produces speaker embeddings and supports 1:1 voice
5+
verification and voice demographic analysis.
6+
7+
## Engines
8+
9+
- **SpeechBrainEngine** (default): ECAPA-TDNN trained on VoxCeleb.
10+
192-d L2-normalised embeddings, cosine distance for verification.
11+
Auto-downloads from HuggingFace on first LoadModel.
12+
- **OnnxDirectEngine**: Any pre-exported ONNX speaker encoder
13+
(WeSpeaker ResNet, 3D-Speaker ERes2Net, CAM++, …). Model path comes
14+
from the gallery `files:` entry.
15+
16+
Engine selection is gallery-driven: if the model config provides
17+
`model_path:` / `onnx:` the ONNX engine is used, otherwise the
18+
SpeechBrain engine.
19+
20+
## Endpoints
21+
22+
- `POST /v1/voice/verify` — 1:1 same-speaker check.
23+
- `POST /v1/voice/embed` — extract a speaker embedding vector.
24+
- `POST /v1/voice/analyze` — voice demographics, loaded lazily on
25+
the first analyze call:
26+
- **Emotion** (default, opt-out): `superb/wav2vec2-base-superb-er`
27+
(Apache-2.0), 4-way categorical (neutral / happy / angry / sad).
28+
- **Age + gender** (opt-in): no default — wire a checkpoint with a
29+
standard `Wav2Vec2ForSequenceClassification` head via
30+
`age_gender_model:<repo>` in options. The Audeering
31+
age-gender model is *not* usable as a drop-in because its
32+
multi-task head isn't loadable via `AutoModelForAudioClassification`.
33+
34+
Both heads are optional. When nothing loads, the engine returns 501.
35+
36+
## Audio input
37+
38+
Audio is materialised by the HTTP layer to a temp wav before calling
39+
the gRPC backend. Accepted input forms on the HTTP side: URL, data-URI,
40+
or raw base64. The backend itself always receives a filesystem path.

0 commit comments

Comments
 (0)