Skip to content

Commit c0b52ac

Browse files
committed
docs: clarify model agnosticism and dynamic auto-scaling tiers in README.md
1 parent 25748a4 commit c0b52ac

1 file changed

Lines changed: 26 additions & 6 deletions

File tree

README.md

Lines changed: 26 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -50,19 +50,39 @@ GhostCoder is the first coding assistant that validates its own output.
5050

5151
---
5252

53-
## Hardware Compatibility & local LLMs
53+
## Model Agnostic & GPU Auto-Scaling
5454

55-
GhostCoder runs entirely on your local machine. It is designed to work efficiently even on lightweight GPUs (e.g., **NVIDIA GTX 1650 4GB VRAM**):
56-
1. **Qwen2.5-0.5B (1GB VRAM)**: Kept loaded in memory for fast situation classification.
57-
2. **Qwen2.5-Coder-1.5B (2GB VRAM)**: Loaded on-demand for code generation, and automatically unloaded after 30 seconds of idle time.
58-
3. **CPU Fallback**: Gracefully falls back to CPU execution if GPU memory headroom (default 500MB) is exceeded.
55+
GhostCoder is **completely model-agnostic** and works with any Large Language Model supported by your local Ollama instance (including Llama 3, DeepSeek, Mistral, Qwen, Codellama, etc.).
56+
57+
By default, GhostCoder automatically detects your local GPU resources and dynamically routes requests to the largest model that fits comfortably in your VRAM budget (up to 85% utilization).
58+
59+
### GPU Tiers & Recommendations
60+
61+
| VRAM Capacity | Tier Profile | Classifier Model | Coder Model | Reasoner Model | Skeptic Model |
62+
| :--- | :--- | :--- | :--- | :--- | :--- |
63+
| **< 6 GB** | `entry` | `qwen2.5:0.5b` | `qwen2.5-coder:1.5b` | `qwen2.5-coder:1.5b` | `qwen2.5:0.5b` |
64+
| **6 - 12 GB** | `mid` | `qwen2.5:0.5b` | `qwen2.5-coder:7b` | `qwen2.5-coder:7b` | `qwen2.5:0.5b` |
65+
| **12 - 20 GB** | `high` | `qwen2.5:0.5b` | `qwen2.5-coder:14b` | `deepseek-coder:6.7b` | `qwen2.5:3b` |
66+
| **20 - 48 GB** | `workstation` | `qwen2.5:3b` | `qwen2.5-coder:32b` | `deepseek-coder:33b` | `qwen2.5:7b` |
67+
| **>= 48 GB** | `datacenter` | `qwen2.5:7b` | `qwen2.5-coder:72b` | `deepseek-coder:33b` | `qwen2.5:14b` |
68+
69+
### Customizing Your Model Configuration
70+
You can hot-swap any role to run any custom model pulled locally:
71+
72+
```bash
73+
# Configure coder model to use Llama 3
74+
ghostcoder config --model-coder llama3:8b
75+
76+
# Configure skeptic model to use Mistral
77+
ghostcoder config --model-skeptic mistral:7b
78+
```
5979

6080
---
6181

6282
## Quick Start
6383

6484
### 1. Prerequisites
65-
Install [Ollama](https://ollama.com/) and pull the models:
85+
Install [Ollama](https://ollama.com/) and pull the models you want to use. For the default Entry configuration:
6686
```bash
6787
ollama pull qwen2.5:0.5b
6888
ollama pull qwen2.5-coder:1.5b

0 commit comments

Comments
 (0)