Skip to content

Commit 46cb00a

Browse files
localai-botlocalai-bot
committed
merge: resolve conflict keeping deduplication logic and adding setInitialSettings
2 parents 1ea4ab1 + 47d9725 commit 46cb00a

141 files changed

Lines changed: 12145 additions & 3025 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/debugging-backends.md‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# Debugging and Rebuilding Backends
2+
3+
When a backend fails at runtime (e.g. a gRPC method error, a Python import error, or a dependency conflict), use this guide to diagnose, fix, and rebuild.
4+
5+
## Architecture Overview
6+
7+
- **Source directory**: `backend/python/<name>/` (or `backend/go/<name>/`, `backend/cpp/<name>/`)
8+
- **Installed directory**: `backends/<name>/` — this is what LocalAI actually runs. It is populated by `make backends/<name>` which builds a Docker image, exports it, and installs it via `local-ai backends install`.
9+
- **Virtual environment**: `backends/<name>/venv/` — the installed Python venv (for Python backends). The Python binary is at `backends/<name>/venv/bin/python`.
10+
11+
Editing files in `backend/python/<name>/` does **not** affect the running backend until you rebuild with `make backends/<name>`.
12+
13+
## Diagnosing Failures
14+
15+
### 1. Check the logs
16+
17+
Backend gRPC processes log to LocalAI's stdout/stderr. Look for lines tagged with the backend's model ID:
18+
19+
```
20+
GRPC stderr id="trl-finetune-127.0.0.1:37335" line="..."
21+
```
22+
23+
Common error patterns:
24+
- **"Method not implemented"** — the backend is missing a gRPC method that the Go side calls. The model loader (`pkg/model/initializers.go`) always calls `LoadModel` after `Health`; fine-tuning backends must implement it even as a no-op stub.
25+
- **Python import errors / `AttributeError`** — usually a dependency version mismatch (e.g. `pyarrow` removing `PyExtensionType`).
26+
- **"failed to load backend"** — the gRPC process crashed or never started. Check stderr lines for the traceback.
27+
28+
### 2. Test the Python environment directly
29+
30+
You can run the installed venv's Python to check imports without starting the full server:
31+
32+
```bash
33+
backends/<name>/venv/bin/python -c "import datasets; print(datasets.__version__)"
34+
```
35+
36+
If `pip` is missing from the venv, bootstrap it:
37+
38+
```bash
39+
backends/<name>/venv/bin/python -m ensurepip
40+
```
41+
42+
Then use `backends/<name>/venv/bin/python -m pip install ...` to test fixes in the installed venv before committing them to the source requirements.
43+
44+
### 3. Check upstream dependency constraints
45+
46+
When you hit a dependency conflict, check what the main library expects. For example, TRL's upstream `requirements.txt`:
47+
48+
```
49+
https://github.com/huggingface/trl/blob/main/requirements.txt
50+
```
51+
52+
Pin minimum versions in the backend's requirements files to match upstream.
53+
54+
## Common Fixes
55+
56+
### Missing gRPC methods
57+
58+
If the Go side calls a method the backend doesn't implement (e.g. `LoadModel`), add a no-op stub in `backend.py`:
59+
60+
```python
61+
def LoadModel(self, request, context):
62+
"""No-op — actual loading happens elsewhere."""
63+
return backend_pb2.Result(success=True, message="OK")
64+
```
65+
66+
The gRPC contract requires `LoadModel` to succeed for the model loader to return a usable client, even if the backend doesn't need upfront model loading.
67+
68+
### Dependency version conflicts
69+
70+
Python backends often break when a transitive dependency releases a breaking change (e.g. `pyarrow` removing `PyExtensionType`). Steps:
71+
72+
1. Identify the broken import in the logs
73+
2. Test in the installed venv: `backends/<name>/venv/bin/python -c "import <module>"`
74+
3. Check upstream requirements for version constraints
75+
4. Update **all** requirements files in `backend/python/<name>/`:
76+
- `requirements.txt` — base deps (grpcio, protobuf)
77+
- `requirements-cpu.txt` — CPU-specific (includes PyTorch CPU index)
78+
- `requirements-cublas12.txt` — CUDA 12
79+
- `requirements-cublas13.txt` — CUDA 13
80+
5. Rebuild: `make backends/<name>`
81+
82+
### PyTorch index conflicts (uv resolver)
83+
84+
The Docker build uses `uv` for pip installs. When `--extra-index-url` points to the PyTorch wheel index, `uv` may refuse to fetch packages like `requests` from PyPI if it finds a different version on the PyTorch index first. Fix this by adding `--index-strategy=unsafe-first-match` to `install.sh`:
85+
86+
```bash
87+
EXTRA_PIP_INSTALL_FLAGS+=" --upgrade --index-strategy=unsafe-first-match"
88+
installRequirements
89+
```
90+
91+
Most Python backends already do this — check `backend/python/transformers/install.sh` or similar for reference.
92+
93+
## Rebuilding
94+
95+
### Rebuild a single backend
96+
97+
```bash
98+
make backends/<name>
99+
```
100+
101+
This runs the Docker build (`Dockerfile.python`), exports the image to `backend-images/<name>.tar`, and installs it into `backends/<name>/`. It also rebuilds the `local-ai` Go binary (without extra tags).
102+
103+
**Important**: If you were previously running with `GO_TAGS=auth`, the `make backends/<name>` step will overwrite your binary without that tag. Rebuild the Go binary afterward:
104+
105+
```bash
106+
GO_TAGS=auth make build
107+
```
108+
109+
### Rebuild and restart
110+
111+
After rebuilding a backend, you must restart LocalAI for it to pick up the new backend files. The backend gRPC process is spawned on demand when the model is first loaded.
112+
113+
```bash
114+
# Kill existing process
115+
kill <pid>
116+
117+
# Restart
118+
./local-ai run --debug [your flags]
119+
```
120+
121+
### Quick iteration (skip Docker rebuild)
122+
123+
For fast iteration on a Python backend's `backend.py` without a full Docker rebuild, you can edit the installed copy directly:
124+
125+
```bash
126+
# Edit the installed copy
127+
vim backends/<name>/backend.py
128+
129+
# Restart LocalAI to respawn the gRPC process
130+
```
131+
132+
This is useful for testing but **does not persist** — the next `make backends/<name>` will overwrite it. Always commit fixes to the source in `backend/python/<name>/`.
133+
134+
## Verification
135+
136+
After fixing and rebuilding:
137+
138+
1. Start LocalAI and confirm the backend registers: look for `Registering backend name="<name>"` in the logs
139+
2. Trigger the operation that failed (e.g. start a fine-tuning job)
140+
3. Watch the GRPC stderr/stdout lines for the backend's model ID
141+
4. Confirm no errors in the traceback

‎.github/workflows/backend.yml‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,32 @@ jobs:
118118
dockerfile: "./backend/Dockerfile.python"
119119
context: "./"
120120
ubuntu-version: '2404'
121+
- build-type: ''
122+
cuda-major-version: ""
123+
cuda-minor-version: ""
124+
platforms: 'linux/amd64'
125+
tag-latest: 'auto'
126+
tag-suffix: '-cpu-trl'
127+
runs-on: 'ubuntu-latest'
128+
base-image: "ubuntu:24.04"
129+
skip-drivers: 'true'
130+
backend: "trl"
131+
dockerfile: "./backend/Dockerfile.python"
132+
context: "./"
133+
ubuntu-version: '2404'
134+
- build-type: ''
135+
cuda-major-version: ""
136+
cuda-minor-version: ""
137+
platforms: 'linux/amd64,linux/arm64'
138+
tag-latest: 'auto'
139+
tag-suffix: '-cpu-llama-cpp-quantization'
140+
runs-on: 'ubuntu-latest'
141+
base-image: "ubuntu:24.04"
142+
skip-drivers: 'true'
143+
backend: "llama-cpp-quantization"
144+
dockerfile: "./backend/Dockerfile.python"
145+
context: "./"
146+
ubuntu-version: '2404'
121147
- build-type: ''
122148
cuda-major-version: ""
123149
cuda-minor-version: ""
@@ -366,6 +392,19 @@ jobs:
366392
dockerfile: "./backend/Dockerfile.python"
367393
context: "./"
368394
ubuntu-version: '2404'
395+
- build-type: 'cublas'
396+
cuda-major-version: "12"
397+
cuda-minor-version: "8"
398+
platforms: 'linux/amd64'
399+
tag-latest: 'auto'
400+
tag-suffix: '-gpu-nvidia-cuda-12-trl'
401+
runs-on: 'ubuntu-latest'
402+
base-image: "ubuntu:24.04"
403+
skip-drivers: 'false'
404+
backend: "trl"
405+
dockerfile: "./backend/Dockerfile.python"
406+
context: "./"
407+
ubuntu-version: '2404'
369408
- build-type: 'cublas'
370409
cuda-major-version: "12"
371410
cuda-minor-version: "8"
@@ -757,6 +796,19 @@ jobs:
757796
dockerfile: "./backend/Dockerfile.python"
758797
context: "./"
759798
ubuntu-version: '2404'
799+
- build-type: 'cublas'
800+
cuda-major-version: "13"
801+
cuda-minor-version: "0"
802+
platforms: 'linux/amd64'
803+
tag-latest: 'auto'
804+
tag-suffix: '-gpu-nvidia-cuda-13-trl'
805+
runs-on: 'ubuntu-latest'
806+
base-image: "ubuntu:24.04"
807+
skip-drivers: 'false'
808+
backend: "trl"
809+
dockerfile: "./backend/Dockerfile.python"
810+
context: "./"
811+
ubuntu-version: '2404'
760812
- build-type: 'l4t'
761813
cuda-major-version: "13"
762814
cuda-minor-version: "0"
@@ -2373,6 +2425,9 @@ jobs:
23732425
tag-suffix: "-metal-darwin-arm64-local-store"
23742426
build-type: "metal"
23752427
lang: "go"
2428+
- backend: "llama-cpp-quantization"
2429+
tag-suffix: "-metal-darwin-arm64-llama-cpp-quantization"
2430+
build-type: "mps"
23762431
with:
23772432
backend: ${{ matrix.backend }}
23782433
build-type: ${{ matrix.build-type }}

‎.github/workflows/bump-inference-defaults.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
name: Bump inference defaults
2+
3+
on:
4+
schedule:
5+
# Run daily at 06:00 UTC
6+
- cron: '0 6 * * *'
7+
workflow_dispatch: # Allow manual trigger
8+
9+
permissions:
10+
contents: write
11+
pull-requests: write
12+
13+
jobs:
14+
bump:
15+
runs-on: ubuntu-latest
16+
steps:
17+
- uses: actions/checkout@v6
18+
19+
- uses: actions/setup-go@v5
20+
with:
21+
go-version-file: go.mod
22+
23+
- name: Re-fetch inference defaults
24+
run: make generate-force
25+
26+
- name: Check for changes
27+
id: diff
28+
run: |
29+
if git diff --quiet core/config/inference_defaults.json; then
30+
echo "changed=false" >> "$GITHUB_OUTPUT"
31+
else
32+
echo "changed=true" >> "$GITHUB_OUTPUT"
33+
fi
34+
35+
- name: Create Pull Request
36+
if: steps.diff.outputs.changed == 'true'
37+
uses: peter-evans/create-pull-request@v8
38+
with:
39+
commit-message: "chore: bump inference defaults from unsloth"
40+
title: "chore: bump inference defaults from unsloth"
41+
body: |
42+
Auto-generated update of `core/config/inference_defaults.json` from
43+
[unsloth's inference_defaults.json](https://github.com/unslothai/unsloth/blob/main/studio/backend/assets/configs/inference_defaults.json).
44+
45+
This PR was created automatically by the `bump-inference-defaults` workflow.
46+
branch: chore/bump-inference-defaults
47+
delete-branch: true
48+
labels: automated

‎.github/workflows/test-extra.yml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -383,6 +383,33 @@ jobs:
383383
run: |
384384
make --jobs=5 --output-sync=target -C backend/python/voxcpm
385385
make --jobs=5 --output-sync=target -C backend/python/voxcpm test
386+
tests-llama-cpp-quantization:
387+
runs-on: ubuntu-latest
388+
timeout-minutes: 30
389+
steps:
390+
- name: Clone
391+
uses: actions/checkout@v6
392+
with:
393+
submodules: true
394+
- name: Dependencies
395+
run: |
396+
sudo apt-get update
397+
sudo apt-get install -y build-essential cmake curl git python3-pip
398+
# Install UV
399+
curl -LsSf https://astral.sh/uv/install.sh | sh
400+
pip install --user --no-cache-dir grpcio-tools==1.64.1
401+
- name: Build llama-quantize from llama.cpp
402+
run: |
403+
git clone --depth 1 https://github.com/ggml-org/llama.cpp.git /tmp/llama.cpp
404+
cmake -B /tmp/llama.cpp/build -S /tmp/llama.cpp -DGGML_NATIVE=OFF
405+
cmake --build /tmp/llama.cpp/build --target llama-quantize -j$(nproc)
406+
sudo cp /tmp/llama.cpp/build/bin/llama-quantize /usr/local/bin/
407+
- name: Install backend
408+
run: |
409+
make --jobs=5 --output-sync=target -C backend/python/llama-cpp-quantization
410+
- name: Test llama-cpp-quantization
411+
run: |
412+
make --jobs=5 --output-sync=target -C backend/python/llama-cpp-quantization test
386413
tests-acestep-cpp:
387414
runs-on: ubuntu-latest
388415
steps:

‎AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ This file is an index to detailed topic guides in the `.agents/` directory. Read
1212
| [.agents/llama-cpp-backend.md](.agents/llama-cpp-backend.md) | Working on the llama.cpp backend — architecture, updating, tool call parsing |
1313
| [.agents/testing-mcp-apps.md](.agents/testing-mcp-apps.md) | Testing MCP Apps (interactive tool UIs) in the React UI |
1414
| [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) | Adding API endpoints, auth middleware, feature permissions, user access control |
15+
| [.agents/debugging-backends.md](.agents/debugging-backends.md) | Debugging runtime backend failures, dependency conflicts, rebuilding backends |
1516

1617
## Quick Reference
1718

‎Makefile‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Disable parallel execution for backend builds
2-
.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus
2+
.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization
33

44
GOCMD=go
55
GOTEST=$(GOCMD) test
@@ -107,7 +107,7 @@ core/http/react-ui/dist: react-ui
107107

108108
## Build:
109109

110-
build: protogen-go install-go-tools core/http/react-ui/dist ## Build the project
110+
build: protogen-go generate install-go-tools core/http/react-ui/dist ## Build the project
111111
$(info ${GREEN}I local-ai build info:${RESET})
112112
$(info ${GREEN}I BUILD_TYPE: ${YELLOW}$(BUILD_TYPE)${RESET})
113113
$(info ${GREEN}I GO_TAGS: ${YELLOW}$(GO_TAGS)${RESET})
@@ -398,6 +398,16 @@ protogen-go: protoc install-go-tools
398398
./protoc --experimental_allow_proto3_optional -Ibackend/ --go_out=pkg/grpc/proto/ --go_opt=paths=source_relative --go-grpc_out=pkg/grpc/proto/ --go-grpc_opt=paths=source_relative \
399399
backend/backend.proto
400400

401+
core/config/inference_defaults.json: ## Fetch inference defaults from unsloth (only if missing)
402+
$(GOCMD) generate ./core/config/...
403+
404+
.PHONY: generate
405+
generate: core/config/inference_defaults.json ## Ensure inference defaults exist
406+
407+
.PHONY: generate-force
408+
generate-force: ## Re-fetch inference defaults from unsloth (always)
409+
$(GOCMD) generate ./core/config/...
410+
401411
.PHONY: protogen-go-clean
402412
protogen-go-clean:
403413
$(RM) pkg/grpc/proto/backend.pb.go pkg/grpc/proto/backend_grpc.pb.go
@@ -421,6 +431,7 @@ prepare-test-extra: protogen-python
421431
$(MAKE) -C backend/python/voxcpm
422432
$(MAKE) -C backend/python/whisperx
423433
$(MAKE) -C backend/python/ace-step
434+
$(MAKE) -C backend/python/trl
424435

425436
test-extra: prepare-test-extra
426437
$(MAKE) -C backend/python/transformers test
@@ -440,6 +451,7 @@ test-extra: prepare-test-extra
440451
$(MAKE) -C backend/python/voxcpm test
441452
$(MAKE) -C backend/python/whisperx test
442453
$(MAKE) -C backend/python/ace-step test
454+
$(MAKE) -C backend/python/trl test
443455

444456
DOCKER_IMAGE?=local-ai
445457
IMAGE_TYPE?=core
@@ -572,6 +584,8 @@ BACKEND_VOXCPM = voxcpm|python|.|false|true
572584
BACKEND_WHISPERX = whisperx|python|.|false|true
573585
BACKEND_ACE_STEP = ace-step|python|.|false|true
574586
BACKEND_MLX_DISTRIBUTED = mlx-distributed|python|./|false|true
587+
BACKEND_TRL = trl|python|.|false|true
588+
BACKEND_LLAMA_CPP_QUANTIZATION = llama-cpp-quantization|python|.|false|true
575589

576590
# Helper function to build docker image for a backend
577591
# Usage: $(call docker-build-backend,BACKEND_NAME,DOCKERFILE_TYPE,BUILD_CONTEXT,PROGRESS_FLAG,NEEDS_BACKEND_ARG)
@@ -629,12 +643,14 @@ $(eval $(call generate-docker-build-target,$(BACKEND_WHISPERX)))
629643
$(eval $(call generate-docker-build-target,$(BACKEND_ACE_STEP)))
630644
$(eval $(call generate-docker-build-target,$(BACKEND_ACESTEP_CPP)))
631645
$(eval $(call generate-docker-build-target,$(BACKEND_MLX_DISTRIBUTED)))
646+
$(eval $(call generate-docker-build-target,$(BACKEND_TRL)))
647+
$(eval $(call generate-docker-build-target,$(BACKEND_LLAMA_CPP_QUANTIZATION)))
632648

633649
# Pattern rule for docker-save targets
634650
docker-save-%: backend-images
635651
docker save local-ai-backend:$* -o backend-images/$*.tar
636652

637-
docker-build-backends: docker-build-llama-cpp docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed
653+
docker-build-backends: docker-build-llama-cpp docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization
638654

639655
########################################################
640656
### Mock Backend for E2E Tests

0 commit comments

Comments
 (0)