|
| 1 | +# Local LLM API — Setup Guide |
| 2 | + |
| 3 | +Machine: HP ZBook Firefly 16 G11 | Intel Core Ultra 7 155H | 32 GB RAM | NVIDIA RTX A500 4 GB |
| 4 | + |
| 5 | +--- |
| 6 | + |
| 7 | +## Quick start (Ollama API on port 9000) |
| 8 | + |
| 9 | +```powershell |
| 10 | +cd C:\PersonalRepo\gemini-nano\api-server |
| 11 | +.\start.ps1 |
| 12 | +``` |
| 13 | + |
| 14 | +Open http://localhost:9000 in any browser to use the web UI. |
| 15 | + |
| 16 | +### Calling from your other projects |
| 17 | + |
| 18 | +```python |
| 19 | +import httpx |
| 20 | + |
| 21 | +resp = httpx.post("http://localhost:9000/v1/chat/completions", json={ |
| 22 | + "model": "gemma3:12b", # see model table below |
| 23 | + "messages": [{"role": "user", "content": "Hello!"}] |
| 24 | +}) |
| 25 | +print(resp.json()["choices"][0]["message"]["content"]) |
| 26 | +``` |
| 27 | + |
| 28 | +OpenAI SDK compatible — just set `base_url="http://localhost:9000/v1"` and `api_key="local"`. |
| 29 | + |
| 30 | +--- |
| 31 | + |
| 32 | +## Available models |
| 33 | + |
| 34 | +| Model | Size | Speed | Best for | |
| 35 | +|-------|------|-------|---------| |
| 36 | +| `gemma3:12b` | 8.1 GB | Medium | **Best quality — use this by default** | |
| 37 | +| `qwen3:8b` | 5.2 GB | Fast | Multilingual, thinking/reasoning mode | |
| 38 | +| `phi3:mini` | 2.2 GB | Fast | Reasoning, structured output | |
| 39 | +| `gemma3:1b` | 815 MB | Fastest | Latency-critical, embedded use | |
| 40 | + |
| 41 | +Aliases work too: `"best"` → gemma3:12b, `"nano"` → gemma3:1b, `"qwen"` → qwen3:8b |
| 42 | + |
| 43 | +--- |
| 44 | + |
| 45 | +## IMPORTANT — Fix before first use (non-ASCII username path bug) |
| 46 | + |
| 47 | +The `ä` in your Windows username `KimHarjamäki` causes llama.cpp to fail loading |
| 48 | +any model. Fix it once with these four commands (run PowerShell **as Administrator**): |
| 49 | + |
| 50 | +```powershell |
| 51 | +# 1. Junction: ASCII alias for the existing model folder (no data copied) |
| 52 | +New-Item -ItemType Junction -Path "C:\ollama-models" -Target "C:\Users\KimHarjamäki\.ollama\models" |
| 53 | +
|
| 54 | +# 2. Persist the path for your user account |
| 55 | +[System.Environment]::SetEnvironmentVariable("OLLAMA_MODELS", "C:\ollama-models", "User") |
| 56 | +
|
| 57 | +# 3. Stop Ollama |
| 58 | +Stop-Process -Name "ollama" -Force -ErrorAction SilentlyContinue |
| 59 | +
|
| 60 | +# 4. Restart Ollama tray app |
| 61 | +Start-Process "$env:LOCALAPPDATA\Programs\Ollama\ollama app.exe" |
| 62 | +``` |
| 63 | + |
| 64 | +--- |
| 65 | + |
| 66 | +## Chrome Built-in Gemini Nano (optional — requires Chrome Dev/Canary) |
| 67 | + |
| 68 | +This lets you run actual Gemini Nano inside Chrome. Different from the Ollama models above. |
| 69 | + |
| 70 | +1. Install Chrome Dev: https://www.google.com/chrome/dev/ |
| 71 | +2. Enable flags in Chrome: |
| 72 | + - `chrome://flags/#optimization-guide-on-device-model` → **Enabled BypassPerfRequirement** |
| 73 | + - `chrome://flags/#prompt-api-for-gemini-nano` → **Enabled** |
| 74 | +3. Restart Chrome, go to `chrome://components/`, update **Optimization Guide On Device Model** |
| 75 | +4. Open `chrome-demo/index.html` in Chrome — banner turns green when ready |
| 76 | + |
| 77 | +Chrome bridge API (port 8081) — exposes window.ai as HTTP: |
| 78 | +```powershell |
| 79 | +cd C:\PersonalRepo\gemini-nano\chrome-bridge |
| 80 | +.\start.ps1 |
| 81 | +# Then POST to http://localhost:8081/v1/chat/completions with model "gemini-nano" |
| 82 | +``` |
| 83 | + |
| 84 | +--- |
| 85 | + |
| 86 | +## Folder structure |
| 87 | + |
| 88 | +``` |
| 89 | +gemini-nano/ |
| 90 | + SETUP.md <- you are here |
| 91 | + api-server/ |
| 92 | + server.py <- FastAPI server (port 9000) |
| 93 | + start.ps1 <- run this |
| 94 | + test_api.py <- smoke test all models |
| 95 | + requirements.txt |
| 96 | + chrome-bridge/ |
| 97 | + server.js <- Chrome Gemini Nano HTTP bridge (port 8081) |
| 98 | + start.ps1 |
| 99 | + package.json |
| 100 | + chrome-demo/ |
| 101 | + index.html <- browser demo for Chrome built-in AI |
| 102 | + python-mediapipe/ |
| 103 | + run_nano.py <- MediaPipe fallback (for when Google releases desktop weights) |
| 104 | + requirements.txt |
| 105 | +``` |
0 commit comments