Skip to content

Commit 4c729f8

Browse files
ForgetMeAIForgetMeAI
committed
feat: add local inference optimizer skill
0 parents  commit 4c729f8

10 files changed

Lines changed: 728 additions & 0 deletions

File tree

‎.github/workflows/validate.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
name: validate
2+
3+
on:
4+
push:
5+
pull_request:
6+
7+
jobs:
8+
validate-skill:
9+
runs-on: ubuntu-latest
10+
steps:
11+
- uses: actions/checkout@v4
12+
- uses: actions/setup-python@v5
13+
with:
14+
python-version: '3.11'
15+
- run: python3 scripts/validate-skill.py

‎.gitignore‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# Generated integration files at repository root
2+
/AGENTS.md
3+
/CLAUDE.md
4+
5+
# Local env / caches
6+
.env
7+
.venv/
8+
__pycache__/
9+
*.pyc
10+
.DS_Store
11+
12+
# Archives
13+
*.zip

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2026 ForgetMeAI
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
<p align="center">
2+
<img src="https://img.shields.io/badge/Hermes%20Skill-local--inference--optimizer-7c3aed?style=for-the-badge" alt="Hermes Skill" />
3+
<img src="https://img.shields.io/badge/LLM-Inference-0ea5e9?style=for-the-badge" alt="LLM Inference" />
4+
<img src="https://img.shields.io/badge/License-MIT-10b981?style=for-the-badge" alt="MIT" />
5+
</p>
6+
7+
# Local Inference Optimizer Skill
8+
9+
**Agent skill for choosing, installing, and tuning the right LLM inference stack for real hardware.**
10+
11+
It teaches Hermes / Claude Code / Codex-style agents to inspect your machine, choose an inference engine, create a `uv` + `.venv` project, tune kernels/quantization/batching/KV cache, and verify the server with a real smoke test.
12+
13+
> Inspired by Ahmad Osman's LLM inference engine decision guide: https://x.com/TheAhmadOsman/status/2057183854444843202
14+
15+
## What it does
16+
17+
- **Detects hardware:** Mac/Apple Silicon, NVIDIA CUDA, AMD ROCm, Intel, CPU-only.
18+
- **Chooses an engine:** MLX, llama.cpp, ExLlama, vLLM, SGLang, TensorRT-LLM, OpenVINO/ONNX Runtime.
19+
- **Sets up the project:** `uv`, `.venv`, configs, launch scripts, smoke tests, benchmark stubs.
20+
- **Tunes inference:** context, batching, prefix cache, KV cache, tensor parallel, memory utilization, quantization.
21+
- **Forces verification:** health check + OpenAI-compatible chat completion before claiming success.
22+
23+
## Decision map
24+
25+
| Hardware / workload | Primary engine |
26+
|---|---|
27+
| MacBook / Mac Studio | MLX first, llama.cpp fallback |
28+
| Laptop / edge / GGUF / odd hardware | llama.cpp |
29+
| Single consumer RTX | ExLlamaV2 or llama.cpp/vLLM |
30+
| 2–4 NVIDIA GPUs | ExLlamaV3, vLLM, or SGLang |
31+
| Production OpenAI-compatible API | vLLM |
32+
| Long context / MoE / structured output | SGLang |
33+
| NVIDIA H100/B200 max perf | TensorRT-LLM benchmark path |
34+
| AMD ROCm | vLLM or SGLang |
35+
| Intel CPU/GPU | OpenVINO or ONNX Runtime GenAI |
36+
37+
## Install in Hermes
38+
39+
### One-liner
40+
41+
```bash
42+
curl -fsSL https://raw.githubusercontent.com/ForgetMeAI/local-inference-optimizer-skill/main/install.sh | bash
43+
```
44+
45+
### Manual
46+
47+
```bash
48+
git clone https://github.com/ForgetMeAI/local-inference-optimizer-skill.git
49+
cd local-inference-optimizer-skill
50+
./install.sh
51+
```
52+
53+
Then start a fresh Hermes session:
54+
55+
```bash
56+
hermes -s local-inference-optimizer
57+
```
58+
59+
or load it in chat:
60+
61+
```text
62+
/skill local-inference-optimizer
63+
```
64+
65+
## Use with Claude Code / Codex
66+
67+
Generate agent instruction files from the skill:
68+
69+
```bash
70+
./scripts/export-agent-instructions.sh
71+
```
72+
73+
This creates:
74+
75+
- `AGENTS.md` for Codex-style agents
76+
- `CLAUDE.md` for Claude Code
77+
78+
You can also copy `SKILL.md` directly into any project instruction system.
79+
80+
## Prompt to use
81+
82+
```text
83+
Use the local-inference-optimizer skill.
84+
Determine the best inference engine for my hardware, set up the project with uv + venv, choose kernels/quantization, tune flags, batching, KV cache, and optimize launch for my model and machine.
85+
86+
Model: <HF repo or local path>
87+
Goal: <local chat / OpenAI API / production / long context / throughput / latency>
88+
```
89+
90+
## Repo contents
91+
92+
```text
93+
.
94+
├── SKILL.md # Main Hermes skill
95+
├── install.sh # Installs the skill into ~/.hermes/skills/mlops/
96+
├── scripts/
97+
│ ├── validate-skill.py # Frontmatter/content sanity checks
98+
│ └── export-agent-instructions.sh # Generates AGENTS.md + CLAUDE.md
99+
├── templates/
100+
│ ├── AGENTS.md # Codex/agent integration template
101+
│ └── CLAUDE.md # Claude Code integration template
102+
└── README.md
103+
```
104+
105+
## Validate
106+
107+
```bash
108+
python3 scripts/validate-skill.py
109+
```
110+
111+
Expected result:
112+
113+
```text
114+
OK local-inference-optimizer skill is valid
115+
```
116+
117+
## License
118+
119+
MIT. Use it, fork it, improve it.

0 commit comments

Comments
 (0)