Hermes Rust Backend 🦀⚡

High-performance Rust backend for Hermes Agent — GGUF inference, Hailo-8 vision, MCP tools, task queue, OpenAI-compatible API.

Architecture

hermes-rust-backend/
├── src/
│   ├── gguf_engine.rs     — llama.cpp GGUF inference (llama-cpp-2)
│   ├── hailo_engine.rs    — Hailo-8 NPU: classify/detect/OCR (REST bridge to :8767)
│   ├── mcp_bridge.rs      — MCP protocol: tool discovery + invoke (v0.4.0)
│   ├── openai_api.rs      — /v1/chat/completions, /v1/models (SSE streaming)
│   ├── api_server.rs      — Status, infer, OCR, detect endpoints
│   ├── multi_gpu.rs       — Multi-GPU info (3 GPUs), best GPU selection
│   ├── onnx_engine.rs     — ONNX runtime engine
│   ├── auth.rs            — JWT authentication
│   └── main.rs            — Server entry point, auto-model-load
├── models/                — GGUF model files
└── Cargo.toml

API Endpoints

Health & Status

Endpoint	Method	Description
/health	GET	Service status, version, features
/engines	GET	Detailed engine status (GGUF, Hailo, CUDA, ONNX)
/status	GET	Uptime, engine availability, GPU count

OpenAI-compatible

Endpoint	Method	Description
/v1/models	GET	Available model IDs
/v1/chat/completions	POST	Chat completions (real GGUF inference)
/v1/chat/completions?stream=true	POST	SSE streaming completions

MCP Tools (v0.4.0)

Endpoint	Method	Description
/mcp/tools	GET	List all MCP tools with schemas
/mcp/tools/:name	GET	Get tool info
/mcp/tools/:name/invoke	POST	Invoke a tool (NEW v0.4.0)

Available tools: mcp_gguf_infer, mcp_hailo_classify, mcp_hailo_detect, mcp_hailo_ocr, mcp_engine_status

Hailo Vision (Legacy)

Endpoint	Method	Description
/infer	POST	GGUF text inference
/detect	POST	Object detection (placeholder)
/ocr	POST	OCR (placeholder)

MCP Invoke Examples (v0.4.0)

Remote image via base64 (cross-machine support)

# Classify an image (sent as base64 from another machine)
curl -X POST http://192.168.1.47:8769/mcp/tools/mcp_hailo_classify/invoke \
  -H "Content-Type: application/json" \
  -d "{\"image_b64\": \"$(base64 -w0 /path/to/image.jpg)\"}"

# Response: {"ok": true, "result": {"predictions": [{"label": "cat", "confidence": 0.95}]}, "duration_ms": 45}

Local image path

# Detect objects (local path on EUREKAI)
curl -X POST http://192.168.1.47:8769/mcp/tools/mcp_hailo_detect/invoke \
  -H "Content-Type: application/json" \
  -d '{"image_path": "/tmp/test.jpg"}'

# Response: {"ok": true, "result": {"detections": [...]}, "duration_ms": 32}

GGUF text inference

curl -X POST http://192.168.1.47:8769/mcp/tools/mcp_gguf_infer/invoke \
  -H "Content-Type: application/json" \
  -d '{"prompt": "What is the capital of France?", "max_tokens": 64}'

Engine status

curl -X POST http://192.168.1.47:8769/mcp/tools/mcp_engine_status/invoke \
  -H "Content-Type: application/json" \
  -d '{}'

Health check (after v0.4.0)

curl http://192.168.1.47:8769/health
# {"service":"hermes-rust-backend","version":"0.4.0","status":"ok",
#  "features":["gguf","openai_api","hailo8","flash_attention","mcp","mcp_invoke","image_b64","streaming","gguf-proxy"]}

When mcp_invoke and image_b64 appear in features, remote invocation is ready.

Build Requirements

• Rust 1.75+
• libclang (for llama-cpp-2 bindgen): sudo apt install libclang-dev
• CUDA Toolkit (optional): nvidia-cuda-toolkit
• Target: EUREKAI (Linux x86_64, GPU recommended)

cargo build --release
./target/release/hermes-rust-backend
# → Listening on 0.0.0.0:8769

Deployment (EUREKAI)

# Build on EUREKAI
cd /home/sylvain/hermes-rust-backend
cargo build --release

# Kill old process
pkill hermes-rust-backend

# Start new
nohup ./target/release/hermes-rust-backend > /tmp/hermes-rust.log 2>&1 &

# Verify
curl http://192.168.1.47:8769/health

Security (LAN-only by default)

The invoke routes listen on 0.0.0.0:8769 with CORS permissive. For production:

• Firewall the port to LAN only: sudo ufw allow from 192.168.1.0/24 to any port 8769
• JWT is available but disabled by default for LAN use
• To enable JWT: set JWT_SECRET env var and modify auth.rs

Quick Start

# Drop a GGUF model
mkdir -p models/
cp ~/models/gemma-4-e2b-it-Q4_K_M.gguf models/

# Start server
cargo run --release

Then use as OpenAI provider in Hermes:

hermes config set model.base_url http://192.168.1.47:8769/v1
hermes config set model.default gguf

Performance

• GGUF Engine — llama.cpp native, token sampling with temperature, CUDA/Vulkan/CPU auto-detect
• Hailo-8 — 26 TOPS NPU, REST bridge to Python MCP on :8767
• Timing — every inference returns tokens/sec + ms elapsed
• MCP Invoke — add duration_ms to every response

Projets liés

• cogniarc — Cognitive reasoning engine
• hermes-brain — Architecture cognitive
• kitten-tts — TTS local FR

Licence MIT