feat(stablediffusion-ggml): expose backend/params placement options

The upstream bump introduced new sd_ctx_params_t fields for device and
memory placement (backend, params_backend, rpc_servers, max_vram,
stream_layers) plus PuLID-Flux weights (pulid_weights_path). Wire them up
as backend options so models can be split across CPU/GPU/disk/RPC:

- backend: per-component compute placement (e.g. clip=cpu,vae=cuda0)
- params_backend: per-component weight storage incl. disk mmap
- max_vram / stream_layers: graph-cut segmented parameter offload budget
- rpc_servers: offload compute to remote RPC servers
- pulid_weights_path: PuLID-Flux identity injection

The legacy keep_*_on_cpu / offload_params_to_cpu booleans now seed and
compose with the explicit backend/params_backend specs, matching upstream
prepare_backend_assignments(). Option values are taken as everything after
the first ':' so colon-bearing values (rpc_servers host:port) survive
parsing. Documented the new options in the image-generation guide.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]

Author: mudler

backend/go/stablediffusion-ggml/cpp/gosd.cpp

@@ -391,9 +391,18 @@ int load_model(const char *model, char *model_path, char* options[], int threads
     const char *control_net_path = "";
     const char *embedding_dir = "";
     const char *photo_maker_path = "";
+    const char *pulid_weights_path = "";
     const char *tensor_type_rules = "";
     char *lora_dir = model_path;
 
+    // Upstream backend/parameter placement specs (see docs/.../stablediffusion).
+    // Empty means "leave at upstream default" (nullptr).
+    const char *backend_arg = "";
+    const char *params_backend_arg = "";
+    const char *rpc_servers_arg = "";
+    const char *max_vram_arg = "";
+    bool stream_layers = false;
+
     int n_threads = threads;
     enum sd_type_t wtype = SD_TYPE_COUNT;
     enum rng_type_t rng_type = CUDA_RNG;
@@ -417,7 +426,9 @@ int load_model(const char *model, char *model_path, char* options[], int threads
     // If options is not NULL, parse options
     for (int i = 0; options[i] != NULL; i++) {
         const char *optname = strtok(options[i], ":");
-        const char *optval = strtok(NULL, ":");
+        // Take everything after the first ':' as the value so values may
+        // themselves contain colons (e.g. rpc_servers host:port lists).
+        const char *optval = strtok(NULL, "");
         if (optval == NULL) {
             optval = "true";
         }
@@ -489,8 +500,18 @@ int load_model(const char *model, char *model_path, char* options[], int threads
             }
         }
         if (!strcmp(optname, "photo_maker_path")) photo_maker_path = strdup(optval);
+        if (!strcmp(optname, "pulid_weights_path")) pulid_weights_path = strdup(optval);
         if (!strcmp(optname, "tensor_type_rules")) tensor_type_rules = strdup(optval);
 
+        // Backend / parameter placement specs (see prepare_backend_assignments
+        // in the upstream CLI). These compose with the legacy keep_*_on_cpu /
+        // offload_params_to_cpu booleans below.
+        if (!strcmp(optname, "backend")) backend_arg = strdup(optval);
+        if (!strcmp(optname, "params_backend")) params_backend_arg = strdup(optval);
+        if (!strcmp(optname, "rpc_servers")) rpc_servers_arg = strdup(optval);
+        if (!strcmp(optname, "max_vram")) max_vram_arg = strdup(optval);
+        if (!strcmp(optname, "stream_layers")) stream_layers = (strcmp(optval, "true") == 0 || strcmp(optval, "1") == 0);
+
         // vae_decode_only is still accepted for backwards compatibility with
         // existing gallery configs, but upstream dropped the option (the model
         // now decides), so it is parsed and ignored.
@@ -592,20 +613,22 @@ int load_model(const char *model, char *model_path, char* options[], int threads
     ctx_params.embeddings = embedding_vec.empty() ? NULL : embedding_vec.data();
     ctx_params.embedding_count = static_cast<uint32_t>(embedding_vec.size());
     ctx_params.photo_maker_path = photo_maker_path;
+    if (strlen(pulid_weights_path) > 0) ctx_params.pulid_weights_path = pulid_weights_path;
     ctx_params.tensor_type_rules = tensor_type_rules;
     ctx_params.n_threads = n_threads;
     ctx_params.rng_type = rng_type;
     if (wtype != SD_TYPE_COUNT) ctx_params.wtype = wtype;
     if (sampler_rng_type != RNG_TYPE_COUNT) ctx_params.sampler_rng_type = sampler_rng_type;
     if (prediction != PREDICTION_COUNT) ctx_params.prediction = prediction;
     if (lora_apply_mode != LORA_APPLY_MODE_COUNT) ctx_params.lora_apply_mode = lora_apply_mode;
-    // Upstream replaced the boolean CPU-offload knobs (offload_params_to_cpu,
-    // keep_clip_on_cpu, keep_vae_on_cpu, keep_control_net_on_cpu) with backend
-    // assignment specs. Translate the legacy options we still accept from
-    // gallery configs into those specs, mirroring prepare_backend_assignments()
-    // in the upstream CLI. These strings must outlive new_sd_ctx() below.
-    std::string backend_spec;
-    std::string params_backend_spec;
+    // Backend / parameter placement specs. Upstream replaced the boolean
+    // CPU-offload knobs (offload_params_to_cpu, keep_clip_on_cpu, keep_vae_on_cpu,
+    // keep_control_net_on_cpu) with these specs. Seed from the explicit
+    // backend/params_backend options, then prepend the legacy boolean-derived
+    // assignments, mirroring prepare_backend_assignments() in the upstream CLI.
+    // These strings must outlive new_sd_ctx() below.
+    std::string backend_spec = backend_arg;
+    std::string params_backend_spec = params_backend_arg;
     auto prepend_spec = [](std::string& spec, const char* assignment) {
         spec = spec.empty() ? std::string(assignment) : std::string(assignment) + "," + spec;
     };
@@ -615,6 +638,12 @@ int load_model(const char *model, char *model_path, char* options[], int threads
     if (keep_control_net_on_cpu) prepend_spec(backend_spec, "controlnet=cpu");
     if (!backend_spec.empty()) ctx_params.backend = backend_spec.c_str();
     if (!params_backend_spec.empty()) ctx_params.params_backend = params_backend_spec.c_str();
+    if (strlen(rpc_servers_arg) > 0) ctx_params.rpc_servers = rpc_servers_arg;
+    // max_vram: GiB budget or per-backend spec for graph-cut segmented param
+    // offload ("0" = disabled, "-1" = auto). stream_layers only has effect when
+    // max_vram is set.
+    if (strlen(max_vram_arg) > 0) ctx_params.max_vram = max_vram_arg;
+    ctx_params.stream_layers = stream_layers;
     ctx_params.diffusion_flash_attn = diffusion_flash_attn;
     ctx_params.tae_preview_only = tae_preview_only;
     ctx_params.diffusion_conv_direct = diffusion_conv_direct;

docs/content/features/image-generation.md

@@ -71,6 +71,42 @@ options:
 2. Download the required assets to the `models` repository
 3. Start LocalAI
 
+#### Memory and device placement options
+
+When a model does not fit entirely in VRAM, the following `options:` control where weights and computation are placed. They map directly to the upstream stable-diffusion.cpp options.
+
+| Option | Example | Description |
+|--------|---------|-------------|
+| `backend` | `backend:clip=cpu,vae=cuda0,diffusion=vulkan0` | Runtime (compute) backend assignment per component. Use `cpu` to place a component's compute on the CPU. Component keys include `te` (text encoder / CLIP), `vae`, `diffusion`, `controlnet`. |
+| `params_backend` | `params_backend:diffusion=disk,clip=cpu` | Where parameters (weights) are stored. Supports `cpu`, `disk` (mmap weights from disk to save RAM/VRAM), or per-component specs. |
+| `max_vram` | `max_vram:8` or `max_vram:-1` | VRAM budget (in GiB) for graph-cut segmented parameter offload. `0` disables it, `-1` auto-selects (free VRAM minus ~1 GiB). Also accepts per-backend budgets. |
+| `stream_layers` | `stream_layers:true` | Enable residency + prefetch streaming on top of `max_vram` (no effect unless `max_vram` is set). |
+| `rpc_servers` | `rpc_servers:localhost:50052,192.168.1.3:50052` | Comma-separated list of `host:port` RPC servers to offload compute to. |
+| `pulid_weights_path` | `pulid_weights_path:pulid.safetensors` | Path to PuLID-Flux weights for identity injection. |
+
+The following convenience booleans are still accepted and are translated into the `backend` / `params_backend` specs above:
+
+| Option | Equivalent spec |
+|--------|-----------------|
+| `offload_params_to_cpu:true` | `params_backend` += `*=cpu` |
+| `keep_clip_on_cpu:true` | `backend` += `te=cpu` |
+| `keep_vae_on_cpu:true` | `backend` += `vae=cpu` |
+| `keep_control_net_on_cpu:true` | `backend` += `controlnet=cpu` |
+
+For example, to mmap the diffusion weights from disk while keeping the text encoder on the CPU:
+
+```yaml
+options:
+- "diffusion_model"
+- "sampler:euler"
+- "params_backend:diffusion=disk"
+- "keep_clip_on_cpu:true"
+```
+
+{{% alert note %}}
+`vae_decode_only` is still accepted for backwards compatibility but is now a no-op: upstream removed the flag and the model decides automatically.
+{{% /alert %}}
+
 
 ### Diffusers