Skip to content

AGENTS.md

Latest commit

 

History

History
138 lines (109 loc) · 7.17 KB

AGENTS.md

File metadata and controls

138 lines (109 loc) · 7.17 KB
Raw

AGENTS.md — orientation for AI agents working in ImageForge

ImageForge is a macOS desktop app (pywebview) for generating AI images locally (MPS) or on rented/cloud GPUs, plus training LoRAs. Python package lives in imageforge/; the desktop UI is imageforge/console/ (Python bridge) + imageforge/console/web/ (HTML/CSS/JS, no build step). FastAPI HTTP surface is imageforge/api/. There is also an MCP server (imageforge/mcp_server.py).

Run the desktop app: ./run_console.sh (uses the arm64 SD venv at ~/.floor-voice-studio/venv-sd so MPS + torch work). Launch it from a foreground Terminal/Finder — a detached/background launch runs the bridge but the Cocoa window won't front-face.

Tests: python3 -m unittest tests.test_<name> (pure-Python suites run without torch). Engine-importing suites (test_mcp_assist_fallback, anything pulling imageforge.engine) need the SD venv because they import torch. JS: just node --check imageforge/console/web/app.js (no bundler).

The AI-assist system (multi-backend "prompt assistant")

ImageForge has two assist surfaces, both routed through ONE backend layer:

  1. Prompt expansion — turn a short idea into a rich SD prompt. UI: the ✨ Enhance button by the prompt box (enhance_prompt bridge → ↩ Undo restores the original). API: POST /prompt/assist and assist_prompt=True on generate. Code: imageforge/api/prompt_bridge.py::assist().
  2. Help chat — "how do I…?" Q&A about the app. UI: the Ask panel (openAssistant in app.js) with a backend picker. Code: imageforge/console/assistant.py::ask().

The one layer everything goes through: imageforge/services/ai_backends.py

caller → ai_backends.generate(system, user, *, prefer, allow_paid, complexity, timeout)
           → order_backends()  # prefer-fast-and-cheap, availability-filtered
           → for each backend: backend.generate(...)  # fall through on None
           → GenResult(text, backend, ok, error)

Backends (registered via agents/settings ASSIST_BACKENDS). Local-first default: ollama only — cloud tiers (ollama_cloud/gemini/claude) are opt-in, not shipped by default, matching the project's local-generation / local-training positioning. Add them explicitly (e.g. ASSIST_BACKENDS=ollama,gemini) plus the matching key to enable one.

Backend tier cost notes
OllamaBackend (local) local free bring-your-own-model — discovers /api/tags, never hardcodes a model
OllamaBackend (cloud) cloud_free free same class + api_key → Authorization: Bearer to ollama.com
GeminiBackend cloud_free free Generative Language REST API
ClaudeBackend paid paid Anthropic /v1/messages; only used when allow_paid

Selection policy (prefer-fast-and-cheap): local → cloud_free → paid. An explicit prefer="<backend name>" (from the picker) wins. Paid backends are excluded unless allow_paid. If a chosen backend returns None at call time, generate() falls through to the next — a transient Ollama hiccup degrades to Gemini, not a failure.

Complexity-based model tiering (BYO-model, no bundled heavy model). The Ollama backend lists the models the user actually pulled and picks by complexity: smallest/fastest for "simple", largest/most-capable for "complex". Callers estimate complexity by input length (assistant chat:

200 chars = complex; prompt expansion: >120 chars = complex). A non-empty OLLAMA_MODEL pins one model explicitly. This is why a simple question uses a fast 4B model instead of timing out on a 9GB model.

Hard design rule: every backend path is graceful — never raises to the caller (returns None, selector moves on). The pipeline must never block image generation. Ollama calls send keep_alive: "30s" so the assistant frees the shared MPS GPU promptly.

Configuration (keys + settings)

All in imageforge/settings.py (env / .env). Users can also enter the cloud keys in-app at System → API Keys & Secrets (set_secret → ~/.imageforge/ secrets.env); secrets are injected into os.environ and read live per call, so a newly-saved key works on the next assist call (no restart needed).

  • OLLAMA_URL, OLLAMA_MODEL (empty = auto-tier)
  • OLLAMA_CLOUD_URL, OLLAMA_CLOUD_API_KEY, OLLAMA_CLOUD_MODEL
  • GEMINI_API_KEY, GEMINI_MODEL; ANTHROPIC_API_KEY, CLAUDE_MODEL
  • ASSIST_BACKENDS, ASSIST_ALLOW_PAID
  • See .env.example for the full annotated list.

Bridge / surface map (keep these 1:1)

console/commands.py (CommandLayer) holds the logic; console/bridge.py (Api) is a thin 1:1 mirror exposed to JS as window.pywebview.api.* — test_bridge_ parity enforces the mirror. Assist methods: assistant_ask(question, backend), assistant_backends(), warmup_assist(), enhance_prompt(prompt, backend), plus power: restart_app(), shutdown_app(). JS calls them via api.call('<method>', ...args); offline preview returns mocks from the api object in app.js.

To add a new AI backend

  1. Add a class to ai_backends.py with name/cost/tier, async available(), async generate(system, user, *, timeout, complexity=None) (must never raise).
  2. Register it in _BACKEND_CLASSES and KNOWN_BACKENDS; add its key to settings.py and (if user-entered) state.py::SECRET_KEYS + SECRET_LABELS.
  3. Add tests in tests/test_ai_backends.py (mock httpx.AsyncClient).

Legacy note

imageforge/services/prompt_assist.py::PromptAssist is deprecated — it's a single-backend Ollama client kept only for its shared helpers (_EXPAND_SYSTEM_PROMPT, _clean_response, imported by the new path) and old tests. Do NOT add new callers; use ai_backends.generate so the paths can't drift.

Handoff — multi-backend assist feature (branch feat/multi-backend-assist)

Built and pushed (origin Image-Generation-MCP-and-Tool). 15 commits from base adb. Status: all pure-Python test suites green (test_ai_backends, test_prompt_bridge, test_prompt_assist, test_console, test_console_restart, test_settings_config), node --check clean, bridge parity passing.

What shipped: the multi-backend assist layer above; the ✨ Enhance prompt button; the Ask-panel backend picker + cold-start warmup + clearer loading state; in-app key entry for the cloud backends (no-restart); header Restart + Shut down buttons (arm64-preserving relaunch). Hardened by an adversarial find-fix pass (4 real bugs) and two gap-closing passes.

Remaining manual check: a live click-test on the target Mac (Enhance button, warmup, arm64 restart) — these are unit-tested + node --check'd but were not clicked live (the dev session's desktop window was flaky). Launch via run_console.sh and smoke-test once.

Known gotchas:

  • The 9GB gemma4 model cold-starts ~90s (too slow for interactive chat); the 4B model loads ~30s. Complexity tiering picks the small one for simple tasks — do not "default to gemma4" again.
  • keep_alive: "30s" is intentional (free the shared MPS GPU); the tradeoff is a cold reload after 30s idle. Don't remove it without weighing GPU contention.
  • Launch the pywebview window from a foreground session or it won't appear.