Skip to content

Commit 06bc24b

Browse files
ForgetMeAIForgetMeAI
committed
docs: add multilingual README
1 parent 4c729f8 commit 06bc24b

1 file changed

Lines changed: 137 additions & 53 deletions

File tree

‎README.md‎

Lines changed: 137 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -6,78 +6,136 @@
66

77
# Local Inference Optimizer Skill
88

9-
**Agent skill for choosing, installing, and tuning the right LLM inference stack for real hardware.**
9+
**Источник идеи:** [Ahmad Osman's LLM inference engine decision guide](https://x.com/TheAhmadOsman/status/2057183854444843202)
1010

11-
It teaches Hermes / Claude Code / Codex-style agents to inspect your machine, choose an inference engine, create a `uv` + `.venv` project, tune kernels/quantization/batching/KV cache, and verify the server with a real smoke test.
11+
**Языки:** [🇷🇺 Русский — основная документация](#-русский) · [🇬🇧 English](#-english) · [🇨🇳 中文](#-中文)
1212

13-
> Inspired by Ahmad Osman's LLM inference engine decision guide: https://x.com/TheAhmadOsman/status/2057183854444843202
13+
---
1414

15-
## What it does
15+
## 🇷🇺 Русский
1616

17-
- **Detects hardware:** Mac/Apple Silicon, NVIDIA CUDA, AMD ROCm, Intel, CPU-only.
18-
- **Chooses an engine:** MLX, llama.cpp, ExLlama, vLLM, SGLang, TensorRT-LLM, OpenVINO/ONNX Runtime.
19-
- **Sets up the project:** `uv`, `.venv`, configs, launch scripts, smoke tests, benchmark stubs.
20-
- **Tunes inference:** context, batching, prefix cache, KV cache, tensor parallel, memory utilization, quantization.
21-
- **Forces verification:** health check + OpenAI-compatible chat completion before claiming success.
17+
**Local Inference Optimizer** — skill для Hermes / Claude Code / Codex-агентов, который помогает агенту не гадать, а реально подобрать и настроить LLM inference stack под железо, модель и сценарий.
2218

23-
## Decision map
19+
Агент должен:
2420

25-
| Hardware / workload | Primary engine |
26-
|---|---|
27-
| MacBook / Mac Studio | MLX first, llama.cpp fallback |
28-
| Laptop / edge / GGUF / odd hardware | llama.cpp |
29-
| Single consumer RTX | ExLlamaV2 or llama.cpp/vLLM |
30-
| 2–4 NVIDIA GPUs | ExLlamaV3, vLLM, or SGLang |
31-
| Production OpenAI-compatible API | vLLM |
32-
| Long context / MoE / structured output | SGLang |
33-
| NVIDIA H100/B200 max perf | TensorRT-LLM benchmark path |
34-
| AMD ROCm | vLLM or SGLang |
35-
| Intel CPU/GPU | OpenVINO or ONNX Runtime GenAI |
21+
- определить железо: Apple Silicon, NVIDIA CUDA, AMD ROCm, Intel, CPU-only;
22+
- выбрать inference engine: MLX, llama.cpp, ExLlama, vLLM, SGLang, TensorRT-LLM, OpenVINO/ONNX Runtime;
23+
- создать проект через `uv` + `.venv`;
24+
- подобрать kernels, quantization, context, batching, KV cache, prefix cache, tensor parallel;
25+
- запустить сервер и проверить его через health check + OpenAI-compatible smoke test.
3626

37-
## Install in Hermes
38-
39-
### One-liner
27+
### Быстрая установка в Hermes
4028

4129
```bash
4230
curl -fsSL https://raw.githubusercontent.com/ForgetMeAI/local-inference-optimizer-skill/main/install.sh | bash
4331
```
4432

45-
### Manual
33+
Затем в новой сессии:
34+
35+
```bash
36+
hermes -s local-inference-optimizer
37+
```
38+
39+
или внутри Hermes-чата:
40+
41+
```text
42+
/skill local-inference-optimizer
43+
```
44+
45+
### Ручная установка
4646

4747
```bash
4848
git clone https://github.com/ForgetMeAI/local-inference-optimizer-skill.git
4949
cd local-inference-optimizer-skill
5050
./install.sh
5151
```
5252

53-
Then start a fresh Hermes session:
53+
### Использование с Claude Code / Codex
5454

5555
```bash
56-
hermes -s local-inference-optimizer
56+
./scripts/export-agent-instructions.sh
5757
```
5858

59-
or load it in chat:
59+
Скрипт создаст:
60+
61+
- `AGENTS.md` — для Codex-style агентов;
62+
- `CLAUDE.md` — для Claude Code.
63+
64+
Или просто скопируйте `SKILL.md` в project instructions.
65+
66+
### Готовый prompt
6067

6168
```text
62-
/skill local-inference-optimizer
69+
Используй local-inference-optimizer skill.
70+
Определи лучший inference engine под моё железо, настрой проект через uv + venv, подбери kernels/quantization, подкрути flags, batching, KV cache и оптимизируй запуск под мою модель и машину.
71+
72+
Модель: <HF repo или локальный путь>
73+
Цель: <локальный чат / OpenAI API / production / long context / throughput / latency>
6374
```
6475

65-
## Use with Claude Code / Codex
76+
### Карта выбора engine
77+
78+
| Железо / сценарий | Стартовый engine |
79+
|---|---|
80+
| MacBook / Mac Studio | MLX, fallback llama.cpp |
81+
| Laptop / edge / GGUF / странное железо | llama.cpp |
82+
| Один consumer RTX | ExLlamaV2 или llama.cpp/vLLM |
83+
| 2–4 NVIDIA GPU | ExLlamaV3, vLLM или SGLang |
84+
| Production OpenAI-compatible API | vLLM |
85+
| Long context / MoE / structured output | SGLang |
86+
| NVIDIA H100/B200 max performance | TensorRT-LLM benchmark path |
87+
| AMD ROCm | vLLM или SGLang |
88+
| Intel CPU/GPU | OpenVINO или ONNX Runtime GenAI |
6689

67-
Generate agent instruction files from the skill:
90+
### Структура репозитория
91+
92+
```text
93+
.
94+
├── SKILL.md # основной Hermes skill
95+
├── install.sh # установка в ~/.hermes/skills/mlops/
96+
├── scripts/
97+
│ ├── validate-skill.py # проверка skill-файла
98+
│ └── export-agent-instructions.sh # генерация AGENTS.md + CLAUDE.md
99+
├── templates/
100+
│ ├── AGENTS.md
101+
│ └── CLAUDE.md
102+
└── README.md
103+
```
104+
105+
### Проверка
68106

69107
```bash
70-
./scripts/export-agent-instructions.sh
108+
python3 scripts/validate-skill.py
71109
```
72110

73-
This creates:
111+
Ожидаемый результат:
112+
113+
```text
114+
OK local-inference-optimizer skill is valid
115+
```
74116

75-
- `AGENTS.md` for Codex-style agents
76-
- `CLAUDE.md` for Claude Code
117+
---
77118

78-
You can also copy `SKILL.md` directly into any project instruction system.
119+
## 🇬🇧 English
79120

80-
## Prompt to use
121+
**Local Inference Optimizer** is a skill for Hermes / Claude Code / Codex-style agents. It helps the agent inspect real hardware, choose the right LLM inference engine, create a `uv` + `.venv` project, tune runtime flags, and verify the server with a real smoke test.
122+
123+
### Install
124+
125+
```bash
126+
curl -fsSL https://raw.githubusercontent.com/ForgetMeAI/local-inference-optimizer-skill/main/install.sh | bash
127+
hermes -s local-inference-optimizer
128+
```
129+
130+
### Use with Claude Code / Codex
131+
132+
```bash
133+
./scripts/export-agent-instructions.sh
134+
```
135+
136+
This generates `AGENTS.md` and `CLAUDE.md` from `SKILL.md`.
137+
138+
### Prompt
81139

82140
```text
83141
Use the local-inference-optimizer skill.
@@ -87,33 +145,59 @@ Model: <HF repo or local path>
87145
Goal: <local chat / OpenAI API / production / long context / throughput / latency>
88146
```
89147

90-
## Repo contents
148+
### Engine map
91149

92-
```text
93-
.
94-
├── SKILL.md # Main Hermes skill
95-
├── install.sh # Installs the skill into ~/.hermes/skills/mlops/
96-
├── scripts/
97-
│ ├── validate-skill.py # Frontmatter/content sanity checks
98-
│ └── export-agent-instructions.sh # Generates AGENTS.md + CLAUDE.md
99-
├── templates/
100-
│ ├── AGENTS.md # Codex/agent integration template
101-
│ └── CLAUDE.md # Claude Code integration template
102-
└── README.md
150+
- Apple Silicon: MLX first, llama.cpp fallback.
151+
- GGUF / edge / unusual hardware: llama.cpp.
152+
- Single consumer RTX: ExLlamaV2 or llama.cpp/vLLM.
153+
- Multi-GPU NVIDIA: ExLlamaV3, vLLM, or SGLang.
154+
- Production API: vLLM.
155+
- Long context / MoE / structured output: SGLang.
156+
- NVIDIA datacenter max performance: TensorRT-LLM benchmark path.
157+
158+
---
159+
160+
## 🇨🇳 中文
161+
162+
**Local Inference Optimizer** 是一个适用于 Hermes / Claude Code / Codex 类代理的 skill。它让代理先检查真实硬件，再选择合适的 LLM 推理引擎，创建 `uv` + `.venv` 项目，调整 kernels、量化、batching、KV cache，并通过真实 smoke test 验证服务。
163+
164+
### 安装
165+
166+
```bash
167+
curl -fsSL https://raw.githubusercontent.com/ForgetMeAI/local-inference-optimizer-skill/main/install.sh | bash
168+
hermes -s local-inference-optimizer
103169
```
104170

105-
## Validate
171+
### 用于 Claude Code / Codex
106172

107173
```bash
108-
python3 scripts/validate-skill.py
174+
./scripts/export-agent-instructions.sh
109175
```
110176

111-
Expected result:
177+
该脚本会根据 `SKILL.md` 生成 `AGENTS.md` 和 `CLAUDE.md`。
178+
179+
### Prompt 示例
112180

113181
```text
114-
OK local-inference-optimizer skill is valid
182+
Use the local-inference-optimizer skill.
183+
Determine the best inference engine for my hardware, set up the project with uv + venv, choose kernels/quantization, tune flags, batching, KV cache, and optimize launch for my model and machine.
184+
185+
Model: <HF repo or local path>
186+
Goal: <local chat / OpenAI API / production / long context / throughput / latency>
115187
```
116188

189+
### 引擎选择速览
190+
191+
- Apple Silicon：优先 MLX，备用 llama.cpp。
192+
- GGUF / 边缘设备 / 特殊硬件：llama.cpp。
193+
- 单张消费级 RTX：ExLlamaV2 或 llama.cpp/vLLM。
194+
- 多张 NVIDIA GPU：ExLlamaV3、vLLM 或 SGLang。
195+
- 生产级 OpenAI-compatible API：vLLM。
196+
- 长上下文 / MoE / 结构化输出：SGLang。
197+
- NVIDIA 数据中心极致性能：TensorRT-LLM benchmark path。
198+
199+
---
200+
117201
## License
118202

119203
MIT. Use it, fork it, improve it.

0 commit comments

Comments
 (0)