Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion .github/bump_deps.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,19 @@ if [ -z "$FILE" ]; then
FILE="Makefile"
fi

LAST_COMMIT=$(curl -s -H "Accept: application/vnd.github.VERSION.sha" "https://api.github.com/repos/$REPO/commits/$BRANCH")
# -L so a renamed/transferred upstream repo (GitHub answers 301) still
# resolves instead of handing us the redirect body, and -f so an HTTP error
# aborts the run rather than letting an error page reach sed below.
LAST_COMMIT=$(curl -sfL -H "Accept: application/vnd.github.VERSION.sha" "https://api.github.com/repos/$REPO/commits/$BRANCH")

# Guard the sed input: anything that is not a bare 40-hex SHA (an API error
# body, an empty response) would otherwise be spliced into the Makefile pin —
# either corrupting it silently or blowing up sed with an unterminated
# expression, which is how this job failed for a renamed repo.
if ! [[ "$LAST_COMMIT" =~ ^[0-9a-f]{40}$ ]]; then
echo "Refusing to bump $VAR: expected a 40-char commit SHA for $REPO@$BRANCH, got: $LAST_COMMIT" >&2
exit 1
fi

# Read $VAR from Makefile (only first match)
set +e
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/bump_deps.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -50,15 +50,15 @@ jobs:
variable: "PARAKEET_VERSION"
branch: "master"
file: "backend/go/parakeet-cpp/Makefile"
- repository: "mudler/moss-transcribe.cpp"
- repository: "localai-org/moss-transcribe.cpp"
variable: "MOSS_VERSION"
branch: "master"
file: "backend/go/moss-transcribe-cpp/Makefile"
- repository: "mudler/ced.cpp"
- repository: "localai-org/ced.cpp"
variable: "CED_VERSION"
branch: "master"
branch: "main"
file: "backend/go/ced/Makefile"
- repository: "mudler/voice-detect.cpp"
- repository: "localai-org/voice-detect.cpp"
variable: "VOICEDETECT_VERSION"
branch: "master"
file: "backend/go/voice-detect/Makefile"
Expand Down Expand Up @@ -90,7 +90,7 @@ jobs:
variable: "SAM3_VERSION"
branch: "main"
file: "backend/go/sam3-cpp/Makefile"
- repository: "mudler/rf-detr.cpp"
- repository: "localai-org/rf-detr.cpp"
variable: "RFDETR_VERSION"
branch: "main"
file: "backend/go/rfdetr-cpp/Makefile"
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -177,7 +177,7 @@ For more details, see the [Getting Started guide](https://localai.io/basics/gett

## Latest News

- **June 2026**: New native biometric backends from the LocalAI team: [voice-detect.cpp](https://github.com/mudler/voice-detect.cpp) for speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion) and [face-detect.cpp](https://github.com/mudler/face-detect.cpp) for face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace). Both are from-scratch C++/ggml engines with no Python or onnxruntime at inference, self-contained GGUF weights, bit-exact parity with the reference, and GPU cuDNN parity, replacing the heavier Python `insightface` and `speaker-recognition` backends ([PR #10441](https://github.com/mudler/LocalAI/pull/10441)).
- **June 2026**: New native biometric backends from the LocalAI team: [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) for speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion) and [face-detect.cpp](https://github.com/mudler/face-detect.cpp) for face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace). Both are from-scratch C++/ggml engines with no Python or onnxruntime at inference, self-contained GGUF weights, bit-exact parity with the reference, and GPU cuDNN parity, replacing the heavier Python `insightface` and `speaker-recognition` backends ([PR #10441](https://github.com/mudler/LocalAI/pull/10441)).
- **June 2026**: New [realtime voice assistant demo](https://github.com/localai-org/localai-realtime-demo) (a tiny Go client for the Realtime API with a full talk-back voice loop and tool calling), plus [streaming of the realtime LLM / TTS / transcription pipeline stages](https://github.com/mudler/LocalAI/pull/10176) and [configurable WebRTC ICE candidates](https://github.com/mudler/LocalAI/pull/10231).
- **June 2026**: Big speech push: the [parakeet.cpp](https://github.com/mudler/parakeet.cpp) ASR engine gains [NeMo-faithful segment timestamps](https://github.com/mudler/LocalAI/pull/10207), a [multilingual streaming Nemotron-3.5 model](https://github.com/mudler/LocalAI/pull/10199), [dynamic batching for concurrent transcription](https://github.com/mudler/LocalAI/pull/10112) and [CUDA graphs](https://github.com/mudler/LocalAI/pull/10273); the new [CrispASR backend](https://github.com/mudler/LocalAI/pull/10099) adds multi-architecture ASR + TTS, and [60 Piper TTS voices across 42 languages](https://github.com/mudler/LocalAI/pull/10296) land in the gallery (plus [per-request TTS instructions and params](https://github.com/mudler/LocalAI/pull/10172)).
- **June 2026**: New backends and models: [locate-anything.cpp](https://github.com/mudler/LocalAI/pull/10264) for open-vocabulary object detection via ggml, [Ideogram4 image generation](https://github.com/mudler/LocalAI/pull/10201) in stablediffusion-ggml, [llama.cpp video input](https://github.com/mudler/LocalAI/pull/10216), and the [Gemma 4 QAT family with MTP speculative-decoding pairs](https://github.com/mudler/LocalAI/pull/10215). Plus an [interactive CLI chat mode](https://github.com/mudler/LocalAI/pull/10226) and [RAG source citations in agent responses](https://github.com/mudler/LocalAI/pull/10228).
Expand Down Expand Up @@ -232,13 +232,13 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
| Backend | What it does |
|---------|-------------|
| [parakeet.cpp](https://github.com/mudler/parakeet.cpp) | C++/GGML port of NVIDIA NeMo Parakeet ASR (tdt/ctc/rnnt/hybrid), with cache-aware streaming transcription |
| [moss-transcribe.cpp](https://github.com/mudler/moss-transcribe.cpp) | C++/GGML port of OpenMOSS MOSS-Transcribe-Diarize: joint long-form transcription, speaker diarization and timestamping in a single pass |
| [moss-transcribe.cpp](https://github.com/localai-org/moss-transcribe.cpp) | C++/GGML port of OpenMOSS MOSS-Transcribe-Diarize: joint long-form transcription, speaker diarization and timestamping in a single pass |
| [moss-tts.cpp](https://github.com/mudler/moss-tts.cpp) | C++/GGML port of the OpenMOSS MOSS-TTS family: text-to-speech (MOSS-TTS-Local v1.5, 48 kHz stereo) with reference-audio voice cloning, through the MOSS-Audio-Tokenizer neural codec |
| [ced.cpp](https://github.com/mudler/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
| [voice-detect.cpp](https://github.com/mudler/voice-detect.cpp) | Speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion), replacing the Python speaker-recognition backend |
| [ced.cpp](https://github.com/localai-org/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
| [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) | Speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion), replacing the Python speaker-recognition backend |
| [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Voxtral Realtime 4B speech-to-text in pure C |
| [vibevoice.cpp](https://github.com/mudler/vibevoice.cpp) | Native port of Microsoft VibeVoice for TTS (voice cloning) and long-form ASR with speaker diarization |
| [rf-detr.cpp](https://github.com/mudler/rf-detr.cpp) | Native RF-DETR object detection and instance segmentation |
| [rf-detr.cpp](https://github.com/localai-org/rf-detr.cpp) | Native RF-DETR object detection and instance segmentation |
| [locate-anything.cpp](https://github.com/mudler/locate-anything.cpp) | Open-vocabulary object detection and visual grounding (LocateAnything-3B) |
| [depth-anything.cpp](https://github.com/mudler/depth-anything.cpp) | Depth Anything 3 monocular metric depth + camera pose estimation |
| [face-detect.cpp](https://github.com/mudler/face-detect.cpp) | Face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace), replacing the Python insightface backend |
Expand Down
2 changes: 1 addition & 1 deletion backend/go/ced/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
# go build -o ced-grpc .

CED_VERSION?=c04ac14b7992d00584d9e812c9bb6268598a6ce7
CED_REPO?=https://github.com/mudler/ced.cpp
CED_REPO?=https://github.com/localai-org/ced.cpp

GOCMD?=go
GO_TAGS?=
Expand Down
2 changes: 1 addition & 1 deletion backend/go/depth-anything-cpp/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ fi
# Depth estimation needs real content; a synthetic image would be degenerate.
TEST_IMAGE_DIR="$CURDIR/test-data"
TEST_IMAGE_FILE="$TEST_IMAGE_DIR/test.jpg"
TEST_IMAGE_URL="${TEST_IMAGE_URL:-https://raw.githubusercontent.com/mudler/rf-detr.cpp/main/tests/fixtures/ci/test_image.jpg}"
TEST_IMAGE_URL="${TEST_IMAGE_URL:-https://raw.githubusercontent.com/localai-org/rf-detr.cpp/main/tests/fixtures/ci/test_image.jpg}"

mkdir -p "$TEST_IMAGE_DIR"
if [ ! -f "$TEST_IMAGE_FILE" ]; then
Expand Down
2 changes: 1 addition & 1 deletion backend/go/locate-anything-cpp/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ fi
# synthetic image would trivially yield zero detections.
TEST_IMAGE_DIR="$CURDIR/test-data"
TEST_IMAGE_FILE="$TEST_IMAGE_DIR/test.jpg"
TEST_IMAGE_URL="${TEST_IMAGE_URL:-https://raw.githubusercontent.com/mudler/rf-detr.cpp/main/tests/fixtures/ci/test_image.jpg}"
TEST_IMAGE_URL="${TEST_IMAGE_URL:-https://raw.githubusercontent.com/localai-org/rf-detr.cpp/main/tests/fixtures/ci/test_image.jpg}"

mkdir -p "$TEST_IMAGE_DIR"
if [ ! -f "$TEST_IMAGE_FILE" ]; then
Expand Down
2 changes: 1 addition & 1 deletion backend/go/moss-transcribe-cpp/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
# clone-at-pin + cmake build so CI doesn't need a side-checkout.

MOSS_VERSION?=190a569c13b4b247450f2fb3b2a431244e84833e
MOSS_REPO?=https://github.com/mudler/moss-transcribe.cpp
MOSS_REPO?=https://github.com/localai-org/moss-transcribe.cpp

GOCMD?=go
GO_TAGS?=
Expand Down
4 changes: 2 additions & 2 deletions backend/go/rfdetr-cpp/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,10 @@ GO_TAGS?=
JOBS?=$(shell nproc --ignore=1)

# rt-detr.cpp (GitHub redirects the historical mudler/rt-detr.cpp to the new
# mudler/rf-detr.cpp slug). Pin to a specific commit if you need a stable
# localai-org/rf-detr.cpp slug). Pin to a specific commit if you need a stable
# build; leaving this on `master` always picks up the latest C-API surface
# (incl. the per-detection accessor functions used by gorfdetrcpp.go).
RFDETR_REPO?=https://github.com/mudler/rf-detr.cpp.git
RFDETR_REPO?=https://github.com/localai-org/rf-detr.cpp.git
RFDETR_VERSION?=65c0ffcc9a9bc9dae38252f63d0417c9845a6cf7

ifeq ($(NATIVE),false)
Expand Down
2 changes: 1 addition & 1 deletion backend/go/rfdetr-cpp/test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ fi
# detections.
TEST_IMAGE_DIR="$CURDIR/test-data"
TEST_IMAGE_FILE="$TEST_IMAGE_DIR/test.jpg"
TEST_IMAGE_URL="${TEST_IMAGE_URL:-https://raw.githubusercontent.com/mudler/rf-detr.cpp/main/tests/fixtures/ci/test_image.jpg}"
TEST_IMAGE_URL="${TEST_IMAGE_URL:-https://raw.githubusercontent.com/localai-org/rf-detr.cpp/main/tests/fixtures/ci/test_image.jpg}"

mkdir -p "$TEST_IMAGE_DIR"
if [ ! -f "$TEST_IMAGE_FILE" ]; then
Expand Down
2 changes: 1 addition & 1 deletion backend/go/voice-detect/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
# not need a side-checkout.

VOICEDETECT_VERSION?=1db1759572c90faef6f3a78c36b5941a096a9f89
VOICEDETECT_REPO?=https://github.com/mudler/voice-detect.cpp
VOICEDETECT_REPO?=https://github.com/localai-org/voice-detect.cpp

GOCMD?=go
GO_TAGS?=
Expand Down
8 changes: 4 additions & 4 deletions backend/index.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,7 @@
It does joint long-form transcription, speaker diarization and timestamping in a single pass, emitting a speaker-labelled,
time-aligned transcript, and runs on CPU, NVIDIA CUDA, AMD ROCm/HIP, Intel SYCL, Vulkan, Apple Metal and NVIDIA Jetson (L4T) targets.
urls:
- https://github.com/mudler/moss-transcribe.cpp
- https://github.com/localai-org/moss-transcribe.cpp
tags:
- audio-transcription
- CPU
Expand Down Expand Up @@ -253,7 +253,7 @@
/v1/audio/classification REST endpoint. It runs on CPU, NVIDIA CUDA,
AMD ROCm/HIP, Intel SYCL, Vulkan and NVIDIA Jetson (L4T) targets.
urls:
- https://github.com/mudler/ced.cpp
- https://github.com/localai-org/ced.cpp
tags:
- audio-classification
- CPU
Expand Down Expand Up @@ -287,7 +287,7 @@
gRPC rpcs and the /v1/voice/* REST endpoints. It runs on CPU, NVIDIA
CUDA, AMD ROCm/HIP, Intel SYCL, Vulkan and NVIDIA Jetson (L4T) targets.
urls:
- https://github.com/mudler/voice-detect.cpp
- https://github.com/localai-org/voice-detect.cpp
tags:
- voice-recognition
- speaker-verification
Expand Down Expand Up @@ -406,7 +406,7 @@
capabilities:
nvidia: "cuda12-rfdetr"
intel: "intel-rfdetr"
#amd: "rocm-rfdetr"

Check warning on line 409 in backend/index.yaml

View workflow job for this annotation

GitHub Actions / Yamllint

409:6 [comments] missing starting space in comment
nvidia-l4t: "nvidia-l4t-arm64-rfdetr"
metal: "metal-rfdetr"
default: "cpu-rfdetr"
Expand Down Expand Up @@ -487,7 +487,7 @@
and returns bounding boxes, class labels, confidence scores, and
(for segmentation variants) PNG-encoded per-detection masks.
urls:
- https://github.com/mudler/rf-detr.cpp
- https://github.com/localai-org/rf-detr.cpp
tags:
- object-detection
- image-segmentation
Expand Down Expand Up @@ -3894,7 +3894,7 @@
capabilities:
nvidia: "cuda12-rfdetr-development"
intel: "intel-rfdetr-development"
#amd: "rocm-rfdetr-development"

Check warning on line 3897 in backend/index.yaml

View workflow job for this annotation

GitHub Actions / Yamllint

3897:6 [comments] missing starting space in comment
nvidia-l4t: "nvidia-l4t-arm64-rfdetr-development"
metal: "metal-rfdetr-development"
default: "cpu-rfdetr-development"
Expand Down
2 changes: 1 addition & 1 deletion docs/content/features/audio-classification.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ url = "/features/audio-classification/"

Sound-event classification (audio tagging) answers the question **"what am I hearing?"** - given an audio clip, it returns a list of scored [AudioSet](https://research.google.com/audioset/) labels (e.g. *Baby cry, infant cry*, *Glass breaking*, *Dog bark*, *Alarm*).

LocalAI exposes this through the `/v1/audio/classification` endpoint, modelled after `/v1/audio/transcriptions`. The reference backend is **[ced.cpp](https://github.com/mudler/ced.cpp)** (CED, a 527-class AudioSet tagger), a small ViT over a log-mel spectrogram ported to ggml with full PyTorch parity. Apache-2.0 weights are redistributable as GGUF.
LocalAI exposes this through the `/v1/audio/classification` endpoint, modelled after `/v1/audio/transcriptions`. The reference backend is **[ced.cpp](https://github.com/localai-org/ced.cpp)** (CED, a 527-class AudioSet tagger), a small ViT over a log-mel spectrogram ported to ggml with full PyTorch parity. Apache-2.0 weights are redistributable as GGUF.

Because classification is exposed as a regular OpenAI-style endpoint, any HTTP client works - there is no Python dependency on the consumer side.

Expand Down
4 changes: 2 additions & 2 deletions docs/content/features/object-detection.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ weight = 41
url = "/features/object-detection/"
+++

LocalAI supports object detection and image segmentation through various backends. This feature allows you to identify and locate objects within images with high accuracy and real-time performance. Available backends include [RF-DETR](https://github.com/roboflow/rf-detr) (Python) and [rf-detr.cpp](https://github.com/mudler/rf-detr.cpp) (native C++/ggml) for object detection and segmentation, and [sam3.cpp](https://github.com/PABannier/sam3.cpp) for image segmentation (SAM 3/2/EdgeTAM).
LocalAI supports object detection and image segmentation through various backends. This feature allows you to identify and locate objects within images with high accuracy and real-time performance. Available backends include [RF-DETR](https://github.com/roboflow/rf-detr) (Python) and [rf-detr.cpp](https://github.com/localai-org/rf-detr.cpp) (native C++/ggml) for object detection and segmentation, and [sam3.cpp](https://github.com/PABannier/sam3.cpp) for image segmentation (SAM 3/2/EdgeTAM).

For detecting **faces** specifically, see the dedicated
[Face Recognition](/features/face-recognition/) feature - its
Expand Down Expand Up @@ -138,7 +138,7 @@ You can browse and install this model through the LocalAI web interface or using
### RF-DETR Native Backend (rfdetr-cpp)

The `rfdetr-cpp` backend is a native C++/ggml implementation of RF-DETR
inference based on [rf-detr.cpp](https://github.com/mudler/rf-detr.cpp). It
inference based on [rf-detr.cpp](https://github.com/localai-org/rf-detr.cpp). It
runs as a Go gRPC service that dlopens a per-CPU-variant shared library, so
there is no Python runtime on the inference path - startup is fast and the
binary is self-contained.
Expand Down
2 changes: 1 addition & 1 deletion docs/content/features/voice-recognition.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ The audio analog to [Face Recognition](/features/face-recognition/),
served over the same `/v1/voice/*` HTTP API by two backends:

- **`voice-detect` (recommended, default).** A standalone C++/ggml
engine ([voice-detect.cpp](https://github.com/mudler/voice-detect.cpp)):
engine ([voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp)):
no Python, no onnxruntime, no torch runtime. Each gallery entry is a
single self-describing GGUF. This is the recommended option for new
deployments.
Expand Down
4 changes: 2 additions & 2 deletions docs/content/reference/compatibility-table.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,12 +94,12 @@ All backends listed here can be installed on demand from the [Backend Gallery]({
| Backend | Description | Acceleration |
|---------|-------------|-------------|
| [RF-DETR](https://github.com/roboflow/rf-detr) | Real-time transformer-based object detection (Python) | CPU, CUDA 12/13, Intel SYCL, Metal, Jetson L4T |
| [rf-detr.cpp](https://github.com/mudler/rf-detr.cpp) | Native RF-DETR object detection and instance segmentation in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T |
| [rf-detr.cpp](https://github.com/localai-org/rf-detr.cpp) | Native RF-DETR object detection and instance segmentation in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T |
| [locate-anything.cpp](https://github.com/mudler/locate-anything.cpp) | Open-vocabulary object detection and visual grounding (LocateAnything-3B) in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T |
| [depth-anything.cpp](https://github.com/mudler/depth-anything.cpp) | Depth Anything 3 monocular metric depth + camera pose in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T |
| [sam3.cpp](https://github.com/PABannier/sam3.cpp) | Segment Anything (SAM 3/2/EdgeTAM) with text/point/box prompts in C/C++ using GGML | CPU, CUDA 12/13, Intel SYCL, Vulkan, Jetson L4T |
| [face-detect.cpp](https://github.com/mudler/face-detect.cpp) | Native face detection, recognition, embedding, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace) in C/C++ using GGML | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| [voice-detect.cpp](https://github.com/mudler/voice-detect.cpp) | Native speaker (voice) recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2) in C/C++ using GGML | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) | Native speaker (voice) recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2) in C/C++ using GGML | CPU, CUDA 12/13, ROCm, Intel SYCL, Vulkan, Metal, Jetson L4T |
| [insightface](https://github.com/deepinsight/insightface) | Face verification, embedding, and anti-spoofing liveness (ONNX Runtime) | CPU, CUDA 12 |
| [speaker-recognition](https://speechbrain.github.io/) | Speaker (voice) recognition via SpeechBrain ECAPA-TDNN | CPU, CUDA 12, Metal |

Expand Down
Loading
Loading