Skip to content

Commit 6b99825

Browse files
authored
feat: introduce skill architecture with YOLO detection skill (#113)
Add the skill system foundation for DeepCamera's evolution into an AI skill platform for SharpAI Aegis. New files: - skills.json: Registry manifest listing available skills - skills/detection/yolo-detection-2026/SKILL.md: First skill spec with parameters, capabilities, setup instructions, and JSON-lines protocol - skills/detection/yolo-detection-2026/requirements.txt: Python dependencies - skills/detection/yolo-detection-2026/scripts/detect.py: Working reference implementation using ultralytics YOLO with stdin/stdout JSON-lines protocol Updated: - README.md: Check 'Skill architecture' roadmap item The skill architecture uses a pluggable SKILL.md interface where each skill declares parameters (rendered as config UI in Aegis) and capabilities (what the skill can do). Skills communicate via JSON lines over stdin/stdout.
1 parent 16363b6 commit 6b99825

5 files changed

Lines changed: 315 additions & 1 deletion

File tree

README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ DeepCamera is evolving into a full AI skill platform. Planned features:
6565
- [ ] **Messaging channel plugins** — Telegram, Discord, Slack, WhatsApp
6666
- [ ] **Automation triggers** — MQTT, webhooks, Home Assistant events
6767
- [ ] **go2rtc streaming** — RTSP to WebRTC live view
68-
- [ ] **Skill architecture** — pluggable `SKILL.md` interface for all capabilities
68+
- [x] **Skill architecture** — pluggable `SKILL.md` interface for all capabilities
6969

7070
---
7171

skills.json

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
{
2+
"version": "1.0.0",
3+
"repository": "SharpAI/DeepCamera",
4+
"updated": "2026-03-02",
5+
"description": "AI skill catalog for SharpAI Aegis. Each skill is a self-contained folder with a SKILL.md manifest.",
6+
"skills": [
7+
{
8+
"id": "yolo-detection-2026",
9+
"name": "YOLO Object Detection (2026)",
10+
"description": "State-of-the-art real-time object detection using YOLOv11/YOLOv10. Person, vehicle, animal, and 80+ COCO class detection with configurable confidence thresholds.",
11+
"version": "1.0.0",
12+
"category": "detection",
13+
"path": "skills/detection/yolo-detection-2026",
14+
"icon": "skills/detection/yolo-detection-2026/assets/icon.png",
15+
"tags": ["yolo", "detection", "person", "vehicle", "real-time", "coco"],
16+
"platforms": ["linux-x64", "linux-arm64", "darwin-arm64", "darwin-x64", "win-x64"],
17+
"requirements": {
18+
"python": ">=3.9",
19+
"gpu": "optional",
20+
"ram_gb": 2,
21+
"disk_gb": 1
22+
},
23+
"capabilities": ["live_detection"],
24+
"ui_unlocks": ["detection_overlay", "alert_config"]
25+
}
26+
]
27+
}
Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
---
2+
name: yolo-detection-2026
3+
description: "State-of-the-art real-time object detection using YOLO"
4+
version: 1.0.0
5+
icon: assets/icon.png
6+
7+
parameters:
8+
- name: model
9+
label: "Model"
10+
type: select
11+
options: ["yolov11n", "yolov11s", "yolov11m", "yolov10n", "yolov10s", "yolov8n"]
12+
default: "yolov11n"
13+
group: Model
14+
15+
- name: confidence
16+
label: "Confidence Threshold"
17+
type: number
18+
min: 0.1
19+
max: 1.0
20+
default: 0.5
21+
group: Model
22+
23+
- name: classes
24+
label: "Detect Classes"
25+
type: string
26+
default: "person,car,dog,cat"
27+
description: "Comma-separated COCO class names (80 classes available)"
28+
group: Model
29+
30+
- name: fps
31+
label: "Processing FPS"
32+
type: number
33+
min: 1
34+
max: 30
35+
default: 5
36+
group: Performance
37+
38+
- name: device
39+
label: "Inference Device"
40+
type: select
41+
options: ["auto", "cpu", "cuda", "mps"]
42+
default: "auto"
43+
description: "auto = GPU if available, else CPU"
44+
group: Performance
45+
46+
capabilities:
47+
live_detection:
48+
script: scripts/detect.py
49+
description: "Real-time object detection on live camera frames"
50+
---
51+
52+
# YOLO Object Detection (2026)
53+
54+
Real-time object detection using state-of-the-art YOLO models. Detects 80+ COCO object classes including people, vehicles, animals, and everyday objects. Outputs bounding boxes with labels and confidence scores that SharpAI Aegis renders as overlays on the live camera feed.
55+
56+
## What You Get
57+
58+
When installed in SharpAI Aegis, this skill unlocks:
59+
- **Live detection overlays** on camera feeds — bounding boxes around detected objects
60+
- **Smart alert triggers** — configure alerts when specific objects are detected
61+
- **Detection history** — searchable log of all detections
62+
63+
## Models
64+
65+
| Model | Size | Speed (FPS) | Accuracy (mAP) | Best For |
66+
|-------|------|-------------|-----------------|----------|
67+
| YOLOv11n | 6 MB | 30+ | 39.5 | Real-time on CPU |
68+
| YOLOv11s | 22 MB | 20+ | 47.0 | Balanced |
69+
| YOLOv11m | 68 MB | 12+ | 51.5 | High accuracy |
70+
| YOLOv10n | 7 MB | 28+ | 38.5 | Ultra-fast |
71+
| YOLOv10s | 24 MB | 18+ | 46.3 | Balanced (v10) |
72+
| YOLOv8n | 6 MB | 30+ | 37.3 | Legacy compatible |
73+
74+
## Setup
75+
76+
1. Create a Python virtual environment:
77+
```bash
78+
python3 -m venv .venv && source .venv/bin/activate
79+
```
80+
81+
2. Install dependencies:
82+
```bash
83+
pip install -r requirements.txt
84+
```
85+
86+
3. Download model weights (automatic on first run, or manually):
87+
```bash
88+
python scripts/download_models.py --model yolov11n
89+
```
90+
91+
## Protocol
92+
93+
This skill communicates with SharpAI Aegis via **JSON lines** over stdin/stdout.
94+
95+
### Aegis → Skill (stdin): frames to process
96+
97+
```jsonl
98+
{"event": "frame", "camera_id": "front_door", "timestamp": "2026-03-01T14:30:00Z", "frame_path": "/tmp/frame_001.jpg", "width": 1920, "height": 1080}
99+
```
100+
101+
### Skill → Aegis (stdout): detection results
102+
103+
```jsonl
104+
{"event": "ready", "model": "yolov11n", "device": "mps", "classes": 80}
105+
{"event": "detections", "camera_id": "front_door", "timestamp": "2026-03-01T14:30:00Z", "objects": [
106+
{"class": "person", "confidence": 0.92, "bbox": [100, 50, 300, 400]},
107+
{"class": "car", "confidence": 0.87, "bbox": [500, 200, 900, 500]}
108+
]}
109+
```
110+
111+
### Bounding Box Format
112+
113+
`[x_min, y_min, x_max, y_max]` in pixel coordinates.
114+
115+
## Hardware Requirements
116+
117+
| Device | Performance |
118+
|--------|------------|
119+
| Apple Silicon (M1+) | 20-30 FPS with MPS acceleration |
120+
| NVIDIA GPU | 25-60 FPS with CUDA |
121+
| CPU (modern x86) | 5-15 FPS |
122+
| Raspberry Pi 5 | 2-5 FPS |
123+
124+
## Contributing
125+
126+
This skill is part of the [DeepCamera](https://github.com/SharpAI/DeepCamera) open-source project. Contributions welcome — see [Contributions.md](../../Contributions.md).
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# YOLO Detection 2026 - Dependencies
2+
# Install: pip install -r requirements.txt
3+
4+
ultralytics>=8.3.0 # YOLOv11/v10/v8 inference
5+
numpy>=1.24.0
6+
opencv-python-headless>=4.8.0
7+
Pillow>=10.0.0
Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
#!/usr/bin/env python3
2+
"""
3+
YOLO Detection Skill — Real-time object detection for SharpAI Aegis.
4+
5+
Communicates via JSON lines over stdin/stdout:
6+
stdin: {"event": "frame", "camera_id": "...", "frame_path": "...", ...}
7+
stdout: {"event": "detections", "camera_id": "...", "objects": [...]}
8+
9+
Usage:
10+
python detect.py --config config.json
11+
python detect.py --model yolov11n --confidence 0.5 --device auto
12+
"""
13+
14+
import sys
15+
import json
16+
import argparse
17+
import signal
18+
from pathlib import Path
19+
20+
def parse_args():
21+
parser = argparse.ArgumentParser(description="YOLO Detection Skill")
22+
parser.add_argument("--config", type=str, help="Path to config JSON file")
23+
parser.add_argument("--model", type=str, default="yolov11n",
24+
choices=["yolov11n", "yolov11s", "yolov11m", "yolov10n", "yolov10s", "yolov8n"])
25+
parser.add_argument("--confidence", type=float, default=0.5)
26+
parser.add_argument("--classes", type=str, default="person,car,dog,cat")
27+
parser.add_argument("--device", type=str, default="auto", choices=["auto", "cpu", "cuda", "mps"])
28+
parser.add_argument("--fps", type=int, default=5)
29+
return parser.parse_args()
30+
31+
32+
def load_config(args):
33+
"""Load config from JSON file or CLI args."""
34+
if args.config:
35+
config_path = Path(args.config)
36+
if config_path.exists():
37+
with open(config_path) as f:
38+
return json.load(f)
39+
return {
40+
"model": args.model,
41+
"confidence": args.confidence,
42+
"classes": args.classes.split(","),
43+
"device": args.device,
44+
"fps": args.fps,
45+
}
46+
47+
48+
def select_device(preference: str) -> str:
49+
"""Select the best available inference device."""
50+
if preference != "auto":
51+
return preference
52+
try:
53+
import torch
54+
if torch.cuda.is_available():
55+
return "cuda"
56+
if hasattr(torch.backends, "mps") and torch.backends.mps.is_available():
57+
return "mps"
58+
except ImportError:
59+
pass
60+
return "cpu"
61+
62+
63+
def emit(event: dict):
64+
"""Write a JSON line to stdout."""
65+
print(json.dumps(event), flush=True)
66+
67+
68+
def main():
69+
args = parse_args()
70+
config = load_config(args)
71+
72+
# Select device
73+
device = select_device(config.get("device", "auto"))
74+
model_name = config.get("model", "yolov11n")
75+
confidence = config.get("confidence", 0.5)
76+
target_classes = config.get("classes", ["person", "car", "dog", "cat"])
77+
78+
# Load YOLO model
79+
try:
80+
from ultralytics import YOLO
81+
model = YOLO(f"{model_name}.pt")
82+
model.to(device)
83+
emit({
84+
"event": "ready",
85+
"model": model_name,
86+
"device": device,
87+
"classes": len(model.names),
88+
})
89+
except Exception as e:
90+
emit({"event": "error", "message": f"Failed to load model: {e}", "retriable": False})
91+
sys.exit(1)
92+
93+
# Graceful shutdown
94+
running = True
95+
def handle_signal(signum, frame):
96+
nonlocal running
97+
running = False
98+
signal.signal(signal.SIGTERM, handle_signal)
99+
signal.signal(signal.SIGINT, handle_signal)
100+
101+
# Main loop: read frames from stdin, output detections to stdout
102+
for line in sys.stdin:
103+
if not running:
104+
break
105+
106+
line = line.strip()
107+
if not line:
108+
continue
109+
110+
try:
111+
msg = json.loads(line)
112+
except json.JSONDecodeError:
113+
continue
114+
115+
if msg.get("command") == "stop":
116+
break
117+
118+
if msg.get("event") == "frame":
119+
frame_path = msg.get("frame_path")
120+
camera_id = msg.get("camera_id", "unknown")
121+
timestamp = msg.get("timestamp", "")
122+
123+
if not frame_path or not Path(frame_path).exists():
124+
emit({"event": "error", "message": f"Frame not found: {frame_path}", "retriable": True})
125+
continue
126+
127+
# Run inference
128+
try:
129+
results = model(frame_path, conf=confidence, verbose=False)
130+
objects = []
131+
for r in results:
132+
for box in r.boxes:
133+
cls_id = int(box.cls[0])
134+
cls_name = model.names[cls_id]
135+
if cls_name in target_classes or not target_classes:
136+
x1, y1, x2, y2 = box.xyxy[0].tolist()
137+
objects.append({
138+
"class": cls_name,
139+
"confidence": round(float(box.conf[0]), 3),
140+
"bbox": [int(x1), int(y1), int(x2), int(y2)],
141+
})
142+
143+
emit({
144+
"event": "detections",
145+
"camera_id": camera_id,
146+
"timestamp": timestamp,
147+
"objects": objects,
148+
})
149+
except Exception as e:
150+
emit({"event": "error", "message": f"Inference error: {e}", "retriable": True})
151+
152+
153+
if __name__ == "__main__":
154+
main()

0 commit comments

Comments
 (0)