Skip to content

Commit 753257e

Browse files
committed
docs: comprehensive SKILL.md v2.0 + README update for multi-backend hardware acceleration
SKILL.md v2.0: - Hardware acceleration section (TensorRT/CoreML/OpenVINO/ONNX) - env_config.py integration documentation - Auto start parameter documentation - Performance monitoring (perf_stats) documentation - Requirements files reference table - Updated protocol examples with backend/format/gpu fields README.md: - Updated yolo-detection-2026 description with acceleration formats - Added hardware environment layer to roadmap (completed) config.yaml: renamed use_coreml → use_optimized (all backends) detect.py: backward-compatible config key migration
1 parent a37e2b7 commit 753257e

4 files changed

Lines changed: 80 additions & 18 deletions

File tree

README.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ Each skill is a self-contained module with its own model, parameters, and [commu
3030

3131
| Category | Skill | What It Does | Status |
3232
|----------|-------|--------------|:------:|
33-
| **Detection** | [`yolo-detection-2026`](skills/detection/yolo-detection-2026/) | Real-time 80+ class object detection ||
33+
| **Detection** | [`yolo-detection-2026`](skills/detection/yolo-detection-2026/) | Real-time 80+ class detection — auto-accelerated via TensorRT / CoreML / OpenVINO / ONNX ||
3434
| | [`dinov3-grounding`](skills/detection/dinov3-grounding/) | Open-vocabulary detection — describe what to find | 📐 |
3535
| | [`person-recognition`](skills/detection/person-recognition/) | Re-identify individuals across cameras | 📐 |
3636
| **Analysis** | [`home-security-benchmark`](skills/analysis/home-security-benchmark/) | [131-test evaluation suite](#-homesec-bench--how-secure-is-your-local-ai) for LLM & VLM security performance ||
@@ -52,6 +52,7 @@ Each skill is a self-contained module with its own model, parameters, and [commu
5252

5353
- [x] **Skill architecture** — pluggable `SKILL.md` interface for all capabilities
5454
- [x] **Full skill catalog** — 18 skills across 9 categories with working scripts
55+
- [x] **Hardware environment layer** — shared [`env_config.py`](skills/lib/env_config.py) for auto-detection + model optimization across NVIDIA, AMD, Apple Silicon, Intel, and CPU
5556
- [ ] **Skill Store UI** — browse, install, and configure skills from Aegis
5657
- [ ] **Custom skill packaging** — community-contributed skills via GitHub
5758
- [ ] **GPU-optimized containers** — one-click Docker deployment per skill

skills/detection/yolo-detection-2026/SKILL.md

Lines changed: 73 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,19 @@
11
---
22
name: yolo-detection-2026
33
description: "YOLO 2026 — state-of-the-art real-time object detection"
4-
version: 1.0.0
4+
version: 2.0.0
55
icon: assets/icon.png
66
entry: scripts/detect.py
7+
deploy: deploy.sh
78

89
parameters:
10+
- name: auto_start
11+
label: "Auto Start"
12+
type: boolean
13+
default: false
14+
description: "Start this skill automatically when Aegis launches"
15+
group: Lifecycle
16+
917
- name: model_size
1018
label: "Model Size"
1119
type: select
@@ -45,6 +53,13 @@ parameters:
4553
description: "auto = best available GPU, else CPU"
4654
group: Performance
4755

56+
- name: use_optimized
57+
label: "Hardware Acceleration"
58+
type: boolean
59+
default: true
60+
description: "Auto-convert model to optimized format for faster inference"
61+
group: Performance
62+
4863
capabilities:
4964
live_detection:
5065
script: scripts/detect.py
@@ -64,6 +79,50 @@ Real-time object detection using the latest YOLO 2026 models. Detects 80+ COCO o
6479
| medium | Moderate | High | Accuracy-focused deployments |
6580
| large | Slower | Highest | Maximum detection quality |
6681

82+
## Hardware Acceleration
83+
84+
The skill uses [`env_config.py`](../../lib/env_config.py) to **automatically detect hardware** and convert the model to the fastest format for your platform. Conversion happens once during deployment and is cached.
85+
86+
| Platform | Backend | Optimized Format | Expected Speedup |
87+
|----------|---------|------------------|:----------------:|
88+
| NVIDIA GPU | CUDA | TensorRT `.engine` | ~3-5x |
89+
| Apple Silicon (M1+) | MPS | CoreML `.mlpackage` | ~2x |
90+
| Intel CPU/GPU/NPU | OpenVINO | OpenVINO IR `.xml` | ~2-3x |
91+
| AMD GPU | ROCm | ONNX Runtime | ~1.5-2x |
92+
| CPU (any) | CPU | ONNX Runtime | ~1.5x |
93+
94+
### How It Works
95+
96+
1. `deploy.sh` detects your hardware via `env_config.HardwareEnv.detect()`
97+
2. Installs the matching `requirements_{backend}.txt` (e.g. CUDA → includes `tensorrt`)
98+
3. Pre-converts the default model to the optimal format
99+
4. At runtime, `detect.py` loads the cached optimized model automatically
100+
5. Falls back to PyTorch if optimization fails
101+
102+
Set `use_optimized: false` to disable auto-conversion and use raw PyTorch.
103+
104+
## Auto Start
105+
106+
Set `auto_start: true` in the skill config to start detection automatically when Aegis launches. The skill will begin processing frames from the selected camera immediately.
107+
108+
```yaml
109+
auto_start: true
110+
model_size: nano
111+
fps: 5
112+
```
113+
114+
## Performance Monitoring
115+
116+
The skill emits `perf_stats` events every 50 frames with aggregate timing:
117+
118+
```jsonl
119+
{"event": "perf_stats", "total_frames": 50, "timings_ms": {
120+
"inference": {"avg": 3.4, "p50": 3.2, "p95": 5.1},
121+
"postprocess": {"avg": 0.15, "p50": 0.12, "p95": 0.31},
122+
"total": {"avg": 3.6, "p50": 3.4, "p95": 5.5}
123+
}}
124+
```
125+
67126
## Protocol
68127

69128
Communicates via **JSON lines** over stdin/stdout.
@@ -75,10 +134,11 @@ Communicates via **JSON lines** over stdin/stdout.
75134

76135
### Skill → Aegis (stdout)
77136
```jsonl
78-
{"event": "ready", "model": "yolo2026n", "device": "mps", "classes": 80, "fps": 5}
137+
{"event": "ready", "model": "yolo2026n", "device": "mps", "backend": "mps", "format": "coreml", "gpu": "Apple M3", "classes": 80, "fps": 5}
79138
{"event": "detections", "frame_id": 42, "camera_id": "front_door", "timestamp": "...", "objects": [
80139
{"class": "person", "confidence": 0.92, "bbox": [100, 50, 300, 400]}
81140
]}
141+
{"event": "perf_stats", "total_frames": 50, "timings_ms": {"inference": {"avg": 3.4}}}
82142
{"event": "error", "message": "...", "retriable": true}
83143
```
84144

@@ -90,20 +150,20 @@ Communicates via **JSON lines** over stdin/stdout.
90150
{"command": "stop"}
91151
```
92152

93-
## Hardware Support
94-
95-
| Platform | Backend | Performance |
96-
|----------|---------|-------------|
97-
| Apple Silicon (M1+) | MPS | 20-30 FPS |
98-
| NVIDIA GPU | CUDA | 25-60 FPS |
99-
| AMD GPU | ROCm | 15-40 FPS |
100-
| CPU (modern x86) | CPU | 5-15 FPS |
101-
| Raspberry Pi 5 | CPU | 2-5 FPS |
102-
103153
## Installation
104154

105-
The `deploy.sh` bootstrapper handles everything — Python environment, GPU backend detection, and dependency installation. No manual setup required.
155+
The `deploy.sh` bootstrapper handles everything — Python environment, GPU backend detection, dependency installation, and model optimization. No manual setup required.
106156

107157
```bash
108158
./deploy.sh
109159
```
160+
161+
### Requirements Files
162+
163+
| File | Backend | Key Deps |
164+
|------|---------|----------|
165+
| `requirements_cuda.txt` | NVIDIA | `torch` (cu124), `tensorrt` |
166+
| `requirements_mps.txt` | Apple | `torch`, `coremltools` |
167+
| `requirements_intel.txt` | Intel | `torch`, `openvino` |
168+
| `requirements_rocm.txt` | AMD | `torch` (rocm6.2), `onnxruntime-rocm` |
169+
| `requirements_cpu.txt` | CPU | `torch` (cpu), `onnxruntime` |

skills/detection/yolo-detection-2026/config.yaml

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,9 +57,10 @@ params:
5757
- { value: mps, label: "Apple Silicon (MPS)" }
5858
- { value: rocm, label: "AMD ROCm" }
5959

60-
- key: use_coreml
61-
label: CoreML Acceleration
60+
- key: use_optimized
61+
label: Hardware Acceleration
6262
type: boolean
6363
default: true
64-
description: "Convert model to CoreML for ~2x faster inference on Apple Silicon (ANE)"
64+
description: "Auto-convert model to optimized format (TensorRT/CoreML/OpenVINO/ONNX) for faster inference"
65+
6566

skills/detection/yolo-detection-2026/scripts/detect.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -171,7 +171,7 @@ def main():
171171
model_size = config.get("model_size", "nano")
172172
confidence = config.get("confidence", 0.5)
173173
fps = config.get("fps", 5)
174-
use_optimized = config.get("use_coreml", True) # legacy key, now covers all backends
174+
use_optimized = config.get("use_optimized", config.get("use_coreml", True))
175175
if isinstance(use_optimized, str):
176176
use_optimized = use_optimized.lower() in ("true", "1", "yes")
177177

0 commit comments

Comments
 (0)