Skip to content

Commit e3b7c7b

Browse files
author
Piotr Stachaczynski
committed
feat: update imagegen docs
1 parent 1d6d4da commit e3b7c7b

2 files changed

Lines changed: 84 additions & 37 deletions

File tree

docs-content/image-generation.md

Lines changed: 74 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ MaIN.NET supports AI image generation through the same `ChatContext` API used fo
1212
| Gemini | Imagen 4.0 Fast, Gemini 2.5 Flash Image | `GeminiKey` |
1313
| xAI (Grok) | Grok Image, Grok Imagine, Grok Imagine Pro | `XaiKey` |
1414
| Vertex AI | Imagen 4.0, Veo 2.0 | `GoogleServiceAccountAuth` |
15-
| Local (FLUX) | FLUX.1 Schnell | `ImageGenUrl` pointing to local service |
15+
| Local (Self) | Stable Diffusion 1.5, FLUX.1 Schnell, Qwen Image | None — runs in-process via GGUF |
1616

1717
**Not supported:** Anthropic, DeepSeek, GroqCloud, Ollama.
1818

@@ -55,8 +55,10 @@ Models.Xai.GrokImagineImagePro // "grok-imagine-image-pro"
5555
Models.Vertex.Imagen4_0_Generate // "google/imagen-4.0-generate-001"
5656
Models.Vertex.Veo2_0_Generate // "google/veo-2.0-generate-001"
5757
58-
// Local
59-
Models.Local.Flux1Shnell // "FLUX.1_Shnell" (runs via local ImageGen service)
58+
// Local (in-process, no external service required)
59+
Models.Local.StableDiffusion1_5 // "stable-diffusion-1.5" — 512×512, SD1, CPU-friendly
60+
Models.Local.Flux1Shnell // "FLUX.1_Shnell" — 768×768, FLUX, good quality
61+
Models.Local.QwenImage // "qwen-image" — 1024×1024, highest quality
6062
```
6163

6264
---
@@ -192,71 +194,108 @@ public class ChatWithImageGenVertexExample : IExample
192194
}
193195
```
194196

195-
### Local FLUX.1 Schnell
197+
### Local — in-process diffusion (no external service)
196198

197-
FLUX.1 runs via a separate local image generation service. Start that service first, then point `ImageGenUrl` at it.
199+
Local image generation runs entirely in-process via `StableDiffusion.NET` (stable-diffusion.cpp). No separate service, no `ImageGenUrl` — just register the model, download it once, and generate.
200+
201+
**Stable Diffusion 1.5** — fastest, single-file GGUF, great for CPU or low-VRAM machines:
198202

199203
```csharp
200-
using MaIN.Core;
201204
using MaIN.Core.Hub;
202-
using MaIN.Domain.Configuration;
203205
using MaIN.Domain.Models;
204206
using MaIN.Domain.Models.Concrete;
205207
using MaIN.Services.Registry;
206208

207-
namespace Examples.Chat;
209+
ModelRegistry.RegisterOrReplace(new StableDiffusion1_5());
210+
await AIHub.Model().EnsureDownloadedAsync(Models.Local.StableDiffusion1_5);
208211

209-
public class ChatWithImageGenExample : IExample
210-
{
211-
public async Task Start()
212-
{
213-
// Only needed if not already set in appsettings.json:
214-
MaINBootstrapper.Initialize(options =>
215-
{
216-
options.ImageGenUrl = "http://localhost:5003";
217-
});
212+
var result = await AIHub.Chat()
213+
.WithModel(Models.Local.StableDiffusion1_5)
214+
.WithMessage("Fluffy cat with a book - anime style")
215+
.CompleteAsync();
218216

219-
// Register the local model (required once at startup)
220-
ModelRegistry.RegisterOrReplace(new GenericLocalModel(Models.Local.Flux1Shnell));
217+
await File.WriteAllBytesAsync("output.png", result.Message.Image!);
218+
```
221219

222-
var result = await AIHub.Chat()
223-
.WithModel(Models.Local.Flux1Shnell)
224-
.WithMessage("Cyberpunk godzilla cat warrior")
225-
.CompleteAsync();
220+
**FLUX.1 Schnell** — better quality, downloads additional VAE + text-encoder assets automatically:
226221

227-
await File.WriteAllBytesAsync("output.png", result.Message.Image!);
228-
}
229-
}
222+
```csharp
223+
ModelRegistry.RegisterOrReplace(new Flux1Shnell());
224+
await AIHub.Model().EnsureDownloadedAsync(Models.Local.Flux1Shnell);
225+
226+
var result = await AIHub.Chat()
227+
.WithModel(Models.Local.Flux1Shnell)
228+
.WithMessage("Cyberpunk godzilla cat warrior")
229+
.CompleteAsync();
230+
231+
await File.WriteAllBytesAsync("output.png", result.Message.Image!);
230232
```
231233

232-
`ImageGenUrl` can also be set in `appsettings.json`:
233-
```json
234-
{
235-
"MaIN": {
236-
"ImageGenUrl": "http://localhost:5003"
237-
}
238-
}
234+
**Qwen Image** — highest quality (1024×1024), requires significant RAM/VRAM, downloads a large Qwen2.5-VL text encoder:
235+
236+
```csharp
237+
ModelRegistry.RegisterOrReplace(new QwenImage());
238+
await AIHub.Model().EnsureDownloadedAsync(Models.Local.QwenImage);
239+
240+
var result = await AIHub.Chat()
241+
.WithModel(Models.Local.QwenImage)
242+
.WithMessage("A photorealistic mountain lake at dawn")
243+
.CompleteAsync();
244+
245+
await File.WriteAllBytesAsync("output.png", result.Message.Image!);
239246
```
240247

248+
**Key facts about local diffusion:**
249+
- `EnsureDownloadedAsync` downloads all required assets (VAE, CLIP, T5-XXL, etc.) automatically — you don't manage them separately
250+
- Models are loaded once and cached in memory for the lifetime of the process
251+
- `ModelsPath` config (or `MaIN_ModelsPath` env var) controls where GGUF files are stored
252+
- No `ImageGenUrl` or external service needed
253+
241254
---
242255

243256
## Custom Model Registration
244257

245258
If you need to use a model ID not in the `Models.*` constants, register it at runtime:
246259

247260
```csharp
261+
using MaIN.Domain.Models.Abstract;
248262
using MaIN.Domain.Models.Concrete;
249263
using MaIN.Services.Registry;
250264

251265
// Cloud image generation model (picks up the current backend's service):
252266
var model = new GenericImageGenerationCloudModel("my-model-id", BackendType.Gemini);
253267
ModelRegistry.RegisterOrReplace(model);
254268

255-
// Local image generation model:
256-
var localModel = new GenericLocalModel("my-local-flux-variant");
269+
// Local in-process diffusion model (GGUF via stable-diffusion.cpp):
270+
var localModel = new GenericLocalImageGenerationModel(
271+
FileName: "my-sd-model-Q8_0.gguf",
272+
Architecture: DiffusionArchitecture.SD1,
273+
Name: "My Custom SD Model",
274+
Width: 512,
275+
Height: 512,
276+
Steps: 25,
277+
CfgScale: 7.0f
278+
);
257279
ModelRegistry.RegisterOrReplace(localModel);
258280
```
259281

282+
For FLUX-based custom models that need separate encoder assets:
283+
```csharp
284+
var fluxModel = new GenericLocalImageGenerationModel(
285+
FileName: "my-flux-model-Q4_0.gguf",
286+
Architecture: DiffusionArchitecture.Flux,
287+
Name: "My FLUX Model",
288+
Width: 768,
289+
Height: 768,
290+
Steps: 4,
291+
CfgScale: 1.0f,
292+
Vae: new ModelAsset("ae.safetensors", new Uri("https://...")),
293+
ClipL: new ModelAsset("clip_l.safetensors", new Uri("https://...")),
294+
T5Xxl: new ModelAsset("t5xxl_fp8_e4m3fn.safetensors", new Uri("https://..."))
295+
);
296+
ModelRegistry.RegisterOrReplace(fluxModel);
297+
```
298+
260299
---
261300

262301
## Result Structure

docs-content/models.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,9 @@ All supported models are available as compile-time constants in the `Models` nam
1515
| `Models.Local.DeepSeekR1_1_5b` | DeepSeek-R1 1.5B — reasoning/CoT model |
1616
| `Models.Local.Qwen2_5_0_5b` | Qwen 2.5 0.5B — ultra-small, edge deployment |
1717
| `Models.Local.Llava16Mistral_7b` | LLaVA 1.6 Mistral 7B — vision model (image input) |
18-
| `Models.Local.Flux1Shnell` | FLUX.1 Schnell — local image generation |
18+
| `Models.Local.StableDiffusion1_5` | Stable Diffusion 1.5 — local image generation (512×512, CPU-friendly) |
19+
| `Models.Local.Flux1Shnell` | FLUX.1 Schnell — local image generation (768×768, balanced quality) |
20+
| `Models.Local.QwenImage` | Qwen Image — local image generation (1024×1024, highest quality) |
1921
| `Models.Local.Kokoro82m` | Kokoro 82M — local text-to-speech |
2022

2123
**Adding a custom GGUF model** (not in the built-in catalogue):
@@ -39,9 +41,15 @@ await AIHub.Chat()
3941
- LLamaSharp is already a dependency of `MaIN.NET` — do NOT add it separately
4042
- Models are downloaded to `ModelsPath` (config key) or the `MaIN_ModelsPath` env variable
4143
- `EnsureModelDownloaded()` auto-downloads before first use; `EnsureDownloadedAsync` supports progress reporting
42-
- `LocalInferenceParams` controls temperature, grammar, top-p, max tokens for local models
44+
- `LocalInferenceParams` controls temperature, grammar, top-p, max tokens for local text models
4345
- Context size is set via `WithMemoryParams(new MemoryParams { ContextSize = 4096 })`
4446

47+
**Local image generation (diffusion models):**
48+
- `StableDiffusion1_5`, `Flux1Shnell`, and `QwenImage` run in-process via `StableDiffusion.NET` — no external service or `ImageGenUrl` needed
49+
- Register the model class first: `ModelRegistry.RegisterOrReplace(new Flux1Shnell())`
50+
- `EnsureDownloadedAsync` handles all required assets (VAE, CLIP, T5-XXL) automatically
51+
- See [image-generation.md](image-generation.md) for full examples
52+
4553
---
4654

4755
## OpenAI

0 commit comments

Comments
 (0)