Skip to content

Commit c8ae2ac

Browse files
committed
feat: OpenVINO skill 1:1 parity with Coral TPU + README skill catalog update
OpenVINO detect.py: - Add file_read timing metric (matches Coral TPU) - Add frame-not-found guard in main loop (empty detections response) - Add invalid JSON log message instead of silent continue OpenVINO SKILL.md: - Add description fields to all parameters - Add Platform Setup (Linux/macOS/Windows) section - Add Model section with compile instructions - Add Bounding Box Format section OpenVINO deploy.sh: - Add find_docker() function pattern - Add exit code 2 for partial success (CPU-only) - Add architecture to platform progress event - Add accelerator_found field in complete event OpenVINO deploy.bat: - Add Docker version reporting - Add device probe result checking New: scripts/compile_model.py - Local model export (--model, --size, --precision, --output) - FP16/INT8/FP32 via YOLO.export(format=openvino) README.md: - Add Coral TPU and OpenVINO to Skill Catalog (🧪 Testing) - Add Detection & Segmentation Skills architecture section - Add mermaid diagram showing native vs Docker detection paths - Add LLM-Assisted Skill Installation explanation
1 parent c5c478a commit c8ae2ac

File tree

6 files changed

+317
-37
lines changed

6 files changed

+317
-37
lines changed

README.md

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,8 @@ Each skill is a self-contained module with its own model, parameters, and [commu
5858
| Category | Skill | What It Does | Status |
5959
|----------|-------|--------------|:------:|
6060
| **Detection** | [`yolo-detection-2026`](skills/detection/yolo-detection-2026/) | Real-time 80+ class detection — auto-accelerated via TensorRT / CoreML / OpenVINO / ONNX ||
61+
| | [`yolo-detection-2026-coral-tpu`](skills/detection/yolo-detection-2026-coral-tpu/) | Google Coral Edge TPU — ~4ms inference via USB accelerator ([Docker-based](#detection--segmentation-skills)) | 🧪 |
62+
| | [`yolo-detection-2026-openvino`](skills/detection/yolo-detection-2026-openvino/) | Intel NCS2 USB / Intel GPU / CPU — multi-device via OpenVINO ([Docker-based](#detection--segmentation-skills)) | 🧪 |
6163
| **Analysis** | [`home-security-benchmark`](skills/analysis/home-security-benchmark/) | [143-test evaluation suite](#-homesec-bench--how-secure-is-your-local-ai) for LLM & VLM security performance ||
6264
| **Privacy** | [`depth-estimation`](skills/transformation/depth-estimation/) | [Real-time depth-map privacy transform](#-privacy--depth-map-anonymization) — anonymize camera feeds while preserving activity ||
6365
| **Segmentation** | [`sam2-segmentation`](skills/segmentation/sam2-segmentation/) | Interactive click-to-segment with Segment Anything 2 — pixel-perfect masks, point/box prompts, video tracking ||
@@ -70,6 +72,54 @@ Each skill is a self-contained module with its own model, parameters, and [commu
7072
7173
> **Registry:** All skills are indexed in [`skills.json`](skills.json) for programmatic discovery.
7274
75+
### Detection & Segmentation Skills
76+
77+
Detection and segmentation skills process visual data from camera feeds — detecting objects, segmenting regions, or analyzing scenes. All skills use the same **JSONL stdin/stdout protocol**: Aegis writes a frame to a shared volume, sends a `frame` event on stdin, and reads `detections` from stdout. This means every detection skill — whether running natively or inside Docker — is interchangeable from Aegis's perspective.
78+
79+
```mermaid
80+
graph TB
81+
CAM["📷 Camera Feed"] --> GOV["Frame Governor (5 FPS)"]
82+
GOV --> |"frame.jpg → shared volume"| PROTO["JSONL stdin/stdout Protocol"]
83+
84+
PROTO --> NATIVE["🖥️ Native: yolo-detection-2026"]
85+
PROTO --> DOCKER["🐳 Docker: Coral TPU / OpenVINO"]
86+
87+
subgraph Native["Native Skill (runs on host)"]
88+
NATIVE --> ENV["env_config.py auto-detect"]
89+
ENV --> TRT["NVIDIA → TensorRT"]
90+
ENV --> CML["Apple Silicon → CoreML"]
91+
ENV --> OV["Intel → OpenVINO IR"]
92+
ENV --> ONNX["AMD / CPU → ONNX"]
93+
end
94+
95+
subgraph Container["Docker Container"]
96+
DOCKER --> CORAL["Coral TPU → pycoral"]
97+
DOCKER --> OVIR["OpenVINO → Ultralytics OV"]
98+
DOCKER --> CPU["CPU fallback"]
99+
CORAL -.-> USB["USB/IP passthrough"]
100+
OVIR -.-> DRI["/dev/dri · /dev/bus/usb"]
101+
end
102+
103+
NATIVE --> |"stdout: detections"| AEGIS["Aegis IPC → Live Overlay + Alerts"]
104+
DOCKER --> |"stdout: detections"| AEGIS
105+
```
106+
107+
- **Native skills** run directly on the host — [`env_config.py`](skills/lib/env_config.py) auto-detects the GPU and converts models to the fastest format (TensorRT, CoreML, OpenVINO IR, ONNX)
108+
- **Docker skills** wrap hardware-specific runtimes in a container — cross-platform USB/device access without native driver installation
109+
- **Same output** — Aegis sees identical JSONL from all skills, so detection overlays, alerts, and forensic analysis work with any backend
110+
111+
#### LLM-Assisted Skill Installation
112+
113+
Skills are installed by an **autonomous LLM deployment agent** — not by brittle shell scripts. When you click "Install" in Aegis, a focused mini-agent session reads the skill's `SKILL.md` manifest and figures out what to do:
114+
115+
1. **Probe** — reads `SKILL.md`, `requirements.txt`, and `package.json` to understand what the skill needs
116+
2. **Detect hardware** — checks for NVIDIA (CUDA), AMD (ROCm), Apple Silicon (MPS), Intel (OpenVINO), or CPU-only
117+
3. **Install** — runs the right commands (`pip install`, `npm install`, `docker build`) with the correct backend-specific dependencies
118+
4. **Verify** — runs a smoke test to confirm the skill loads before marking it complete
119+
5. **Determine launch command** — figures out the exact `run_command` to start the skill and saves it to the registry
120+
121+
This means community-contributed skills don't need a bespoke installer — the LLM reads the manifest and adapts to whatever hardware you have. If something fails, it reads the error output and tries to fix it autonomously.
122+
73123

74124
## 🚀 Getting Started with [SharpAI Aegis](https://www.sharpai.org)
75125

skills/detection/yolo-detection-2026-openvino/SKILL.md

Lines changed: 51 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ parameters:
1616
label: "Auto Start"
1717
type: boolean
1818
default: false
19+
description: "Start this skill automatically when Aegis launches"
1920
group: Lifecycle
2021

2122
- name: confidence
@@ -24,26 +25,30 @@ parameters:
2425
min: 0.1
2526
max: 1.0
2627
default: 0.5
28+
description: "Minimum detection confidence (0.1–1.0)"
2729
group: Model
2830

2931
- name: classes
3032
label: "Detect Classes"
3133
type: string
3234
default: "person,car,dog,cat"
35+
description: "Comma-separated COCO class names (80 classes available)"
3336
group: Model
3437

3538
- name: fps
3639
label: "Processing FPS"
3740
type: select
3841
options: [0.2, 0.5, 1, 3, 5, 15]
3942
default: 5
43+
description: "Frames per second — OpenVINO on GPU/NCS2 handles 15+ FPS"
4044
group: Performance
4145

4246
- name: input_size
4347
label: "Input Resolution"
4448
type: select
4549
options: [320, 640]
4650
default: 640
51+
description: "640 is recommended for GPU/CPU accuracy, 320 for fastest inference"
4752
group: Performance
4853

4954
- name: device
@@ -59,12 +64,13 @@ parameters:
5964
type: select
6065
options: ["FP16", "INT8", "FP32"]
6166
default: "FP16"
67+
description: "FP16 is fastest on GPU/NCS2; INT8 is fastest on CPU; FP32 is most accurate"
6268
group: Performance
6369

6470
capabilities:
6571
live_detection:
6672
script: scripts/detect.py
67-
description: "Real-time object detection via OpenVINO"
73+
description: "Real-time object detection via OpenVINO runtime"
6874

6975
category: detection
7076
mutex: detection
@@ -99,6 +105,44 @@ Real-time object detection using Intel OpenVINO runtime. Runs inside Docker for
99105
└─────────────────────────────────────────────────────┘
100106
```
101107

108+
1. Aegis writes camera frame JPEG to shared `/tmp/aegis_detection/` volume
109+
2. Sends `frame` event via stdin JSONL to Docker container
110+
3. `detect.py` reads frame, runs inference via OpenVINO
111+
4. Returns `detections` event via stdout JSONL
112+
5. Same protocol as `yolo-detection-2026` — Aegis sees no difference
113+
114+
## Platform Setup
115+
116+
### Linux
117+
```bash
118+
# Intel GPU and NCS2 auto-detected via /dev/dri and /dev/bus/usb
119+
# Docker uses --device flags for direct device access
120+
./deploy.sh
121+
```
122+
123+
### macOS (Docker Desktop 4.35+)
124+
```bash
125+
# Docker Desktop USB/IP handles NCS2 passthrough
126+
# CPU fallback always available
127+
./deploy.sh
128+
```
129+
130+
### Windows
131+
```powershell
132+
# Docker Desktop 4.35+ with USB/IP support
133+
# Or WSL2 backend with usbipd-win for NCS2
134+
.\deploy.bat
135+
```
136+
137+
## Model
138+
139+
Ships without a pre-compiled model by default. On first run, `detect.py` will auto-download `yolo26n.pt` and export to OpenVINO IR format. To pre-export:
140+
141+
```bash
142+
# Runs on any platform (unlike Edge TPU compilation)
143+
python scripts/compile_model.py --model yolo26n --size 640 --precision FP16
144+
```
145+
102146
## Supported Devices
103147

104148
| Device | Flag | Precision | ~Speed |
@@ -113,14 +157,20 @@ Real-time object detection using Intel OpenVINO runtime. Runs inside Docker for
113157

114158
Same JSONL as `yolo-detection-2026`:
115159

160+
### Skill → Aegis (stdout)
116161
```jsonl
117162
{"event": "ready", "model": "yolo26n_openvino", "device": "GPU", "format": "openvino_ir", "classes": 80}
118163
{"event": "detections", "frame_id": 42, "camera_id": "front_door", "objects": [{"class": "person", "confidence": 0.85, "bbox": [100, 50, 300, 400]}]}
119164
{"event": "perf_stats", "total_frames": 50, "timings_ms": {"inference": {"avg": 8.1, "p50": 7.9, "p95": 10.2}}}
120165
```
121166

167+
### Bounding Box Format
168+
`[x_min, y_min, x_max, y_max]` — pixel coordinates (xyxy).
169+
122170
## Installation
123171

124172
```bash
125173
./deploy.sh
126174
```
175+
176+
The deployer builds the Docker image locally, probes for OpenVINO devices, and sets the runtime command. No packages pulled from external registries beyond Docker base images and pip dependencies.
Lines changed: 33 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,10 @@
11
@echo off
22
REM deploy.bat — Docker-based bootstrapper for OpenVINO Detection Skill (Windows)
3+
REM
4+
REM Builds the Docker image locally and verifies device availability.
5+
REM Called by Aegis skill-runtime-manager during installation.
6+
REM
7+
REM Requires: Docker Desktop 4.35+ with USB/IP support (for NCS2)
38

49
setlocal enabledelayedexpansion
510

@@ -8,40 +13,59 @@ set "IMAGE_NAME=aegis-openvino-detect"
813
set "IMAGE_TAG=latest"
914
set "LOG_PREFIX=[openvino-deploy]"
1015

11-
REM ─── Check Docker ────────────────────────────────────────────────────────
16+
REM ─── Step 1: Check Docker ────────────────────────────────────────────────
17+
1218
where docker >nul 2>&1
1319
if %errorlevel% neq 0 (
14-
echo %LOG_PREFIX% ERROR: Docker not found. 1>&2
15-
echo {"event": "error", "stage": "docker", "message": "Docker not found"}
20+
echo %LOG_PREFIX% ERROR: Docker not found. Install Docker Desktop 4.35+ 1>&2
21+
echo {"event": "error", "stage": "docker", "message": "Docker not found. Install Docker Desktop 4.35+"}
1622
exit /b 1
1723
)
1824

1925
docker info >nul 2>&1
2026
if %errorlevel% neq 0 (
21-
echo %LOG_PREFIX% ERROR: Docker daemon not running. 1>&2
27+
echo %LOG_PREFIX% ERROR: Docker daemon not running. Start Docker Desktop. 1>&2
2228
echo {"event": "error", "stage": "docker", "message": "Docker daemon not running"}
2329
exit /b 1
2430
)
2531

26-
echo {"event": "progress", "stage": "docker", "message": "Docker ready"}
32+
for /f "tokens=*" %%v in ('docker version --format "{{.Server.Version}}" 2^>nul') do set "DOCKER_VER=%%v"
33+
echo %LOG_PREFIX% Using Docker (version: %DOCKER_VER%) 1>&2
34+
echo {"event": "progress", "stage": "docker", "message": "Docker ready (%DOCKER_VER%)"}
35+
36+
REM ─── Step 2: Build Docker image ──────────────────────────────────────────
2737

28-
REM ─── Build Docker image ──────────────────────────────────────────────────
29-
echo %LOG_PREFIX% Building Docker image... 1>&2
38+
echo %LOG_PREFIX% Building Docker image: %IMAGE_NAME%:%IMAGE_TAG% ... 1>&2
3039
echo {"event": "progress", "stage": "build", "message": "Building Docker image..."}
3140

3241
docker build -t %IMAGE_NAME%:%IMAGE_TAG% "%SKILL_DIR%"
3342
if %errorlevel% neq 0 (
43+
echo %LOG_PREFIX% ERROR: Docker build failed 1>&2
3444
echo {"event": "error", "stage": "build", "message": "Docker image build failed"}
3545
exit /b 1
3646
)
3747

3848
echo {"event": "progress", "stage": "build", "message": "Docker image ready"}
3949

40-
REM ─── Probe devices ──────────────────────────────────────────────────────
50+
REM ─── Step 3: Probe for OpenVINO devices ──────────────────────────────────
51+
52+
echo %LOG_PREFIX% Probing OpenVINO devices... 1>&2
53+
echo {"event": "progress", "stage": "probe", "message": "Checking OpenVINO devices..."}
54+
4155
docker run --rm --privileged %IMAGE_NAME%:%IMAGE_TAG% python3 scripts/device_probe.py >nul 2>&1
56+
if %errorlevel% equ 0 (
57+
echo %LOG_PREFIX% OpenVINO devices detected 1>&2
58+
echo {"event": "progress", "stage": "probe", "message": "OpenVINO devices detected"}
59+
) else (
60+
echo %LOG_PREFIX% WARNING: No accelerator detected - CPU fallback 1>&2
61+
echo {"event": "progress", "stage": "probe", "message": "No GPU/NCS2 detected - CPU fallback"}
62+
)
63+
64+
REM ─── Step 4: Set run command ──────────────────────────────────────────────
4265

43-
REM ─── Set run command ─────────────────────────────────────────────────────
4466
set "RUN_CMD=docker run -i --rm --privileged -v /tmp/aegis_detection:/tmp/aegis_detection --env AEGIS_SKILL_ID --env AEGIS_SKILL_PARAMS --env PYTHONUNBUFFERED=1 %IMAGE_NAME%:%IMAGE_TAG%"
4567

4668
echo {"event": "complete", "status": "success", "run_command": "%RUN_CMD%", "message": "OpenVINO skill installed"}
69+
70+
echo %LOG_PREFIX% Done! 1>&2
4771
exit /b 0

0 commit comments

Comments
 (0)