Skip to content

Commit d9c1db2

Browse files
mudlermudler
authored
feat: add (experimental) fine-tuning support with TRL (#9088)
* feat: add fine-tuning endpoint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(experimental): add fine-tuning endpoint and TRL support This changeset defines new GRPC signatues for Fine tuning backends, and add TRL backend as initial fine-tuning engine. This implementation also supports exporting to GGUF and automatically importing it to LocalAI after fine-tuning. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * commit TRL backend, stop by killing process Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * move fine-tune to generic features Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * add evals, reorder menu Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fix tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
1 parent f7e3aab commit d9c1db2

49 files changed

Lines changed: 5652 additions & 110 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/debugging-backends.md‎

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# Debugging and Rebuilding Backends
2+
3+
When a backend fails at runtime (e.g. a gRPC method error, a Python import error, or a dependency conflict), use this guide to diagnose, fix, and rebuild.
4+
5+
## Architecture Overview
6+
7+
- **Source directory**: `backend/python/<name>/` (or `backend/go/<name>/`, `backend/cpp/<name>/`)
8+
- **Installed directory**: `backends/<name>/` — this is what LocalAI actually runs. It is populated by `make backends/<name>` which builds a Docker image, exports it, and installs it via `local-ai backends install`.
9+
- **Virtual environment**: `backends/<name>/venv/` — the installed Python venv (for Python backends). The Python binary is at `backends/<name>/venv/bin/python`.
10+
11+
Editing files in `backend/python/<name>/` does **not** affect the running backend until you rebuild with `make backends/<name>`.
12+
13+
## Diagnosing Failures
14+
15+
### 1. Check the logs
16+
17+
Backend gRPC processes log to LocalAI's stdout/stderr. Look for lines tagged with the backend's model ID:
18+
19+
```
20+
GRPC stderr id="trl-finetune-127.0.0.1:37335" line="..."
21+
```
22+
23+
Common error patterns:
24+
- **"Method not implemented"** — the backend is missing a gRPC method that the Go side calls. The model loader (`pkg/model/initializers.go`) always calls `LoadModel` after `Health`; fine-tuning backends must implement it even as a no-op stub.
25+
- **Python import errors / `AttributeError`** — usually a dependency version mismatch (e.g. `pyarrow` removing `PyExtensionType`).
26+
- **"failed to load backend"** — the gRPC process crashed or never started. Check stderr lines for the traceback.
27+
28+
### 2. Test the Python environment directly
29+
30+
You can run the installed venv's Python to check imports without starting the full server:
31+
32+
```bash
33+
backends/<name>/venv/bin/python -c "import datasets; print(datasets.__version__)"
34+
```
35+
36+
If `pip` is missing from the venv, bootstrap it:
37+
38+
```bash
39+
backends/<name>/venv/bin/python -m ensurepip
40+
```
41+
42+
Then use `backends/<name>/venv/bin/python -m pip install ...` to test fixes in the installed venv before committing them to the source requirements.
43+
44+
### 3. Check upstream dependency constraints
45+
46+
When you hit a dependency conflict, check what the main library expects. For example, TRL's upstream `requirements.txt`:
47+
48+
```
49+
https://github.com/huggingface/trl/blob/main/requirements.txt
50+
```
51+
52+
Pin minimum versions in the backend's requirements files to match upstream.
53+
54+
## Common Fixes
55+
56+
### Missing gRPC methods
57+
58+
If the Go side calls a method the backend doesn't implement (e.g. `LoadModel`), add a no-op stub in `backend.py`:
59+
60+
```python
61+
def LoadModel(self, request, context):
62+
"""No-op — actual loading happens elsewhere."""
63+
return backend_pb2.Result(success=True, message="OK")
64+
```
65+
66+
The gRPC contract requires `LoadModel` to succeed for the model loader to return a usable client, even if the backend doesn't need upfront model loading.
67+
68+
### Dependency version conflicts
69+
70+
Python backends often break when a transitive dependency releases a breaking change (e.g. `pyarrow` removing `PyExtensionType`). Steps:
71+
72+
1. Identify the broken import in the logs
73+
2. Test in the installed venv: `backends/<name>/venv/bin/python -c "import <module>"`
74+
3. Check upstream requirements for version constraints
75+
4. Update **all** requirements files in `backend/python/<name>/`:
76+
- `requirements.txt` — base deps (grpcio, protobuf)
77+
- `requirements-cpu.txt` — CPU-specific (includes PyTorch CPU index)
78+
- `requirements-cublas12.txt` — CUDA 12
79+
- `requirements-cublas13.txt` — CUDA 13
80+
5. Rebuild: `make backends/<name>`
81+
82+
### PyTorch index conflicts (uv resolver)
83+
84+
The Docker build uses `uv` for pip installs. When `--extra-index-url` points to the PyTorch wheel index, `uv` may refuse to fetch packages like `requests` from PyPI if it finds a different version on the PyTorch index first. Fix this by adding `--index-strategy=unsafe-first-match` to `install.sh`:
85+
86+
```bash
87+
EXTRA_PIP_INSTALL_FLAGS+=" --upgrade --index-strategy=unsafe-first-match"
88+
installRequirements
89+
```
90+
91+
Most Python backends already do this — check `backend/python/transformers/install.sh` or similar for reference.
92+
93+
## Rebuilding
94+
95+
### Rebuild a single backend
96+
97+
```bash
98+
make backends/<name>
99+
```
100+
101+
This runs the Docker build (`Dockerfile.python`), exports the image to `backend-images/<name>.tar`, and installs it into `backends/<name>/`. It also rebuilds the `local-ai` Go binary (without extra tags).
102+
103+
**Important**: If you were previously running with `GO_TAGS=auth`, the `make backends/<name>` step will overwrite your binary without that tag. Rebuild the Go binary afterward:
104+
105+
```bash
106+
GO_TAGS=auth make build
107+
```
108+
109+
### Rebuild and restart
110+
111+
After rebuilding a backend, you must restart LocalAI for it to pick up the new backend files. The backend gRPC process is spawned on demand when the model is first loaded.
112+
113+
```bash
114+
# Kill existing process
115+
kill <pid>
116+
117+
# Restart
118+
./local-ai run --debug [your flags]
119+
```
120+
121+
### Quick iteration (skip Docker rebuild)
122+
123+
For fast iteration on a Python backend's `backend.py` without a full Docker rebuild, you can edit the installed copy directly:
124+
125+
```bash
126+
# Edit the installed copy
127+
vim backends/<name>/backend.py
128+
129+
# Restart LocalAI to respawn the gRPC process
130+
```
131+
132+
This is useful for testing but **does not persist** — the next `make backends/<name>` will overwrite it. Always commit fixes to the source in `backend/python/<name>/`.
133+
134+
## Verification
135+
136+
After fixing and rebuilding:
137+
138+
1. Start LocalAI and confirm the backend registers: look for `Registering backend name="<name>"` in the logs
139+
2. Trigger the operation that failed (e.g. start a fine-tuning job)
140+
3. Watch the GRPC stderr/stdout lines for the backend's model ID
141+
4. Confirm no errors in the traceback

‎.github/workflows/backend.yml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,19 @@ jobs:
118118
dockerfile: "./backend/Dockerfile.python"
119119
context: "./"
120120
ubuntu-version: '2404'
121+
- build-type: ''
122+
cuda-major-version: ""
123+
cuda-minor-version: ""
124+
platforms: 'linux/amd64'
125+
tag-latest: 'auto'
126+
tag-suffix: '-cpu-trl'
127+
runs-on: 'ubuntu-latest'
128+
base-image: "ubuntu:24.04"
129+
skip-drivers: 'true'
130+
backend: "trl"
131+
dockerfile: "./backend/Dockerfile.python"
132+
context: "./"
133+
ubuntu-version: '2404'
121134
- build-type: ''
122135
cuda-major-version: ""
123136
cuda-minor-version: ""
@@ -366,6 +379,19 @@ jobs:
366379
dockerfile: "./backend/Dockerfile.python"
367380
context: "./"
368381
ubuntu-version: '2404'
382+
- build-type: 'cublas'
383+
cuda-major-version: "12"
384+
cuda-minor-version: "8"
385+
platforms: 'linux/amd64'
386+
tag-latest: 'auto'
387+
tag-suffix: '-gpu-nvidia-cuda-12-trl'
388+
runs-on: 'ubuntu-latest'
389+
base-image: "ubuntu:24.04"
390+
skip-drivers: 'false'
391+
backend: "trl"
392+
dockerfile: "./backend/Dockerfile.python"
393+
context: "./"
394+
ubuntu-version: '2404'
369395
- build-type: 'cublas'
370396
cuda-major-version: "12"
371397
cuda-minor-version: "8"
@@ -757,6 +783,19 @@ jobs:
757783
dockerfile: "./backend/Dockerfile.python"
758784
context: "./"
759785
ubuntu-version: '2404'
786+
- build-type: 'cublas'
787+
cuda-major-version: "13"
788+
cuda-minor-version: "0"
789+
platforms: 'linux/amd64'
790+
tag-latest: 'auto'
791+
tag-suffix: '-gpu-nvidia-cuda-13-trl'
792+
runs-on: 'ubuntu-latest'
793+
base-image: "ubuntu:24.04"
794+
skip-drivers: 'false'
795+
backend: "trl"
796+
dockerfile: "./backend/Dockerfile.python"
797+
context: "./"
798+
ubuntu-version: '2404'
760799
- build-type: 'l4t'
761800
cuda-major-version: "13"
762801
cuda-minor-version: "0"

‎AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ This file is an index to detailed topic guides in the `.agents/` directory. Read
1212
| [.agents/llama-cpp-backend.md](.agents/llama-cpp-backend.md) | Working on the llama.cpp backend — architecture, updating, tool call parsing |
1313
| [.agents/testing-mcp-apps.md](.agents/testing-mcp-apps.md) | Testing MCP Apps (interactive tool UIs) in the React UI |
1414
| [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) | Adding API endpoints, auth middleware, feature permissions, user access control |
15+
| [.agents/debugging-backends.md](.agents/debugging-backends.md) | Debugging runtime backend failures, dependency conflicts, rebuilding backends |
1516

1617
## Quick Reference
1718

‎Makefile‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Disable parallel execution for backend builds
2-
.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus
2+
.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl
33

44
GOCMD=go
55
GOTEST=$(GOCMD) test
@@ -421,6 +421,7 @@ prepare-test-extra: protogen-python
421421
$(MAKE) -C backend/python/voxcpm
422422
$(MAKE) -C backend/python/whisperx
423423
$(MAKE) -C backend/python/ace-step
424+
$(MAKE) -C backend/python/trl
424425

425426
test-extra: prepare-test-extra
426427
$(MAKE) -C backend/python/transformers test
@@ -440,6 +441,7 @@ test-extra: prepare-test-extra
440441
$(MAKE) -C backend/python/voxcpm test
441442
$(MAKE) -C backend/python/whisperx test
442443
$(MAKE) -C backend/python/ace-step test
444+
$(MAKE) -C backend/python/trl test
443445

444446
DOCKER_IMAGE?=local-ai
445447
IMAGE_TYPE?=core
@@ -572,6 +574,7 @@ BACKEND_VOXCPM = voxcpm|python|.|false|true
572574
BACKEND_WHISPERX = whisperx|python|.|false|true
573575
BACKEND_ACE_STEP = ace-step|python|.|false|true
574576
BACKEND_MLX_DISTRIBUTED = mlx-distributed|python|./|false|true
577+
BACKEND_TRL = trl|python|.|false|true
575578

576579
# Helper function to build docker image for a backend
577580
# Usage: $(call docker-build-backend,BACKEND_NAME,DOCKERFILE_TYPE,BUILD_CONTEXT,PROGRESS_FLAG,NEEDS_BACKEND_ARG)
@@ -629,12 +632,13 @@ $(eval $(call generate-docker-build-target,$(BACKEND_WHISPERX)))
629632
$(eval $(call generate-docker-build-target,$(BACKEND_ACE_STEP)))
630633
$(eval $(call generate-docker-build-target,$(BACKEND_ACESTEP_CPP)))
631634
$(eval $(call generate-docker-build-target,$(BACKEND_MLX_DISTRIBUTED)))
635+
$(eval $(call generate-docker-build-target,$(BACKEND_TRL)))
632636

633637
# Pattern rule for docker-save targets
634638
docker-save-%: backend-images
635639
docker save local-ai-backend:$* -o backend-images/$*.tar
636640

637-
docker-build-backends: docker-build-llama-cpp docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed
641+
docker-build-backends: docker-build-llama-cpp docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl
638642

639643
########################################################
640644
### Mock Backend for E2E Tests

‎backend/backend.proto‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,13 @@ service Backend {
3939
rpc AudioDecode(AudioDecodeRequest) returns (AudioDecodeResult) {}
4040

4141
rpc ModelMetadata(ModelOptions) returns (ModelMetadataResponse) {}
42+
43+
// Fine-tuning RPCs
44+
rpc StartFineTune(FineTuneRequest) returns (FineTuneJobResult) {}
45+
rpc FineTuneProgress(FineTuneProgressRequest) returns (stream FineTuneProgressUpdate) {}
46+
rpc StopFineTune(FineTuneStopRequest) returns (Result) {}
47+
rpc ListCheckpoints(ListCheckpointsRequest) returns (ListCheckpointsResponse) {}
48+
rpc ExportModel(ExportModelRequest) returns (Result) {}
4249
}
4350

4451
// Define the empty request
@@ -528,3 +535,105 @@ message ModelMetadataResponse {
528535
string rendered_template = 2; // The rendered chat template with enable_thinking=true (empty if not applicable)
529536
ToolFormatMarkers tool_format = 3; // Auto-detected tool format markers from differential template analysis
530537
}
538+
539+
// Fine-tuning messages
540+
541+
message FineTuneRequest {
542+
// Model identification
543+
string model = 1; // HF model name or local path
544+
string training_type = 2; // "lora", "loha", "lokr", "full" — what parameters to train
545+
string training_method = 3; // "sft", "dpo", "grpo", "rloo", "reward", "kto", "orpo", "network_training"
546+
547+
// Adapter config (universal across LoRA/LoHa/LoKr for LLM + diffusion)
548+
int32 adapter_rank = 10; // LoRA rank (r), default 16
549+
int32 adapter_alpha = 11; // scaling factor, default 16
550+
float adapter_dropout = 12; // default 0.0
551+
repeated string target_modules = 13; // layer names to adapt
552+
553+
// Universal training hyperparameters
554+
float learning_rate = 20; // default 2e-4
555+
int32 num_epochs = 21; // default 3
556+
int32 batch_size = 22; // default 2
557+
int32 gradient_accumulation_steps = 23; // default 4
558+
int32 warmup_steps = 24; // default 5
559+
int32 max_steps = 25; // 0 = use epochs
560+
int32 save_steps = 26; // 0 = only save final
561+
float weight_decay = 27; // default 0.01
562+
bool gradient_checkpointing = 28;
563+
string optimizer = 29; // adamw_8bit, adamw, sgd, adafactor, prodigy
564+
int32 seed = 30; // default 3407
565+
string mixed_precision = 31; // fp16, bf16, fp8, no
566+
567+
// Dataset
568+
string dataset_source = 40; // HF dataset ID, local file/dir path
569+
string dataset_split = 41; // train, test, etc.
570+
571+
// Output
572+
string output_dir = 50;
573+
string job_id = 51; // client-assigned or auto-generated
574+
575+
// Resume training from a checkpoint
576+
string resume_from_checkpoint = 55; // path to checkpoint dir to resume from
577+
578+
// Backend-specific AND method-specific extensibility
579+
map<string, string> extra_options = 60;
580+
}
581+
582+
message FineTuneJobResult {
583+
string job_id = 1;
584+
bool success = 2;
585+
string message = 3;
586+
}
587+
588+
message FineTuneProgressRequest {
589+
string job_id = 1;
590+
}
591+
592+
message FineTuneProgressUpdate {
593+
string job_id = 1;
594+
int32 current_step = 2;
595+
int32 total_steps = 3;
596+
float current_epoch = 4;
597+
float total_epochs = 5;
598+
float loss = 6;
599+
float learning_rate = 7;
600+
float grad_norm = 8;
601+
float eval_loss = 9;
602+
float eta_seconds = 10;
603+
float progress_percent = 11;
604+
string status = 12; // queued, caching, loading_model, loading_dataset, training, saving, completed, failed, stopped
605+
string message = 13;
606+
string checkpoint_path = 14; // set when a checkpoint is saved
607+
string sample_path = 15; // set when a sample is generated (video/image backends)
608+
map<string, float> extra_metrics = 16; // method-specific metrics
609+
}
610+
611+
message FineTuneStopRequest {
612+
string job_id = 1;
613+
bool save_checkpoint = 2;
614+
}
615+
616+
message ListCheckpointsRequest {
617+
string output_dir = 1;
618+
}
619+
620+
message ListCheckpointsResponse {
621+
repeated CheckpointInfo checkpoints = 1;
622+
}
623+
624+
message CheckpointInfo {
625+
string path = 1;
626+
int32 step = 2;
627+
float epoch = 3;
628+
float loss = 4;
629+
string created_at = 5;
630+
}
631+
632+
message ExportModelRequest {
633+
string checkpoint_path = 1;
634+
string output_path = 2;
635+
string export_format = 3; // lora, loha, lokr, merged_16bit, merged_4bit, gguf, diffusers
636+
string quantization_method = 4; // for GGUF: q4_k_m, q5_k_m, q8_0, f16, etc.
637+
string model = 5; // base model name (for merge operations)
638+
map<string, string> extra_options = 6;
639+
}

0 commit comments

Comments
 (0)