Skip to content

Commit 25b8da7

Browse files
committed
feat(media-use): resolve engine + BGM/SFX/image/icon providers
The resolve cascade and all v1 providers, using heygen-cli with --x-source media-use for attribution. - resolve.mjs: cheapest-first cascade (manifest → assets/ → cache → heygen-cli → freeze → register) - providers.mjs: pluggable registry with 4 real providers - heygen-search.mjs: shared CLI exec helper for all providers - bgm-provider: heygen audio sounds list --type music - sfx-provider: heygen audio sounds list --type sound_effects - image-provider: heygen asset search list --type image + icon - SKILL.md: full agent-facing docs with types, examples, flags - Router skill wiring: added media-use to routing table
1 parent 2db7682 commit 25b8da7

8 files changed

Lines changed: 482 additions & 0 deletions

File tree

skills/hyperframes/SKILL.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ Atomic capabilities you load **on demand** — not full video workflows. For "ma
3232
| **Animate** — atomic motion, scene blueprints, transitions, runtime adapters (GSAP / Lottie / Three.js / Anime.js / CSS / WAAPI / TypeGPU) | `/hyperframes-animation` |
3333
| **Creative direction**`frame.md` / `design.md`, palettes, typography, narration, beat planning, audio-reactive | `/hyperframes-creative` |
3434
| **Media** — TTS voiceover, background music, transcription, background removal, captions | `/hyperframes-media` |
35+
| **Media resolve** — find + freeze BGM, SFX, images, icons from HeyGen catalog into `.media/` with manifest tracking | `/media-use` |
3536
| **CLI dev loop** — init, lint, validate, inspect, preview, render, publish, doctor | `/hyperframes-cli` |
3637
| **Install registry blocks / components** (`hyperframes add`) | `/hyperframes-registry` |
3738

skills/media-use/SKILL.md

Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
---
2+
name: media-use
3+
description: Agent Media OS — resolve any media need (BGM, SFX, image, icon) into a frozen local file + ledger record. One verb (`resolve`) handles the full cascade: project cache, global cache, HeyGen catalog search, freeze, register. Keeps search noise on disk, hands the agent a path. Use when a composition needs background music, sound effects, images, or icons.
4+
---
5+
6+
# media-use
7+
8+
Resolve media needs into frozen local files. One verb, four types, zero context noise.
9+
10+
## When to use
11+
12+
Call `resolve` whenever a composition needs media — background music, sound effects, images, or icons. media-use searches the HeyGen catalog, downloads the best match, freezes it locally, and registers it in a manifest. The agent gets back one line; all search noise stays on disk.
13+
14+
## Resolve
15+
16+
```bash
17+
node <SKILL_DIR>/scripts/resolve.mjs --type <type> --intent "<description>" --project <dir>
18+
```
19+
20+
Returns one line: `resolved <id> → <path> (<type>, <metadata>)`
21+
22+
### Types
23+
24+
| Type | What it finds | Provider |
25+
| ------- | ------------------- | ---------------------------------------- |
26+
| `bgm` | Background music | HeyGen audio catalog (10k+ tracks) |
27+
| `sfx` | Sound effects | Bundled 19-file library + HeyGen catalog |
28+
| `image` | Photos, backgrounds | HeyGen asset search (75k+ vectors) |
29+
| `icon` | Icons, logos | HeyGen asset search (type=icon) |
30+
31+
### Examples
32+
33+
```bash
34+
# Background music
35+
node <SKILL_DIR>/scripts/resolve.mjs --type bgm --intent "upbeat tech launch" --project .
36+
# → resolved bgm_001 → .media/audio/bgm/bgm_001.mp3 (bgm, 25s)
37+
38+
# Sound effect
39+
node <SKILL_DIR>/scripts/resolve.mjs --type sfx --intent "whoosh" --project .
40+
# → resolved sfx_001 → .media/audio/sfx/sfx_001.mp3 (sfx, 0.57s)
41+
42+
# Image
43+
node <SKILL_DIR>/scripts/resolve.mjs --type image --intent "gradient tech background" --project .
44+
# → resolved image_001 → .media/images/image_001.jpg (image)
45+
46+
# Icon
47+
node <SKILL_DIR>/scripts/resolve.mjs --type icon --intent "rocket" --project .
48+
# → resolved icon_001 → .media/images/icon_001.svg (icon, transparent)
49+
```
50+
51+
### Flags
52+
53+
| Flag | Description |
54+
| --------------- | ------------------------------------------ |
55+
| `--type, -t` | Media type: bgm, sfx, image, icon |
56+
| `--intent, -i` | What you need (natural language) |
57+
| `--entity, -e` | Entity name for cache matching (optional) |
58+
| `--project, -p` | Project directory (default: .) |
59+
| `--adopt` | Bulk-import existing assets/ into manifest |
60+
| `--json` | Output JSON instead of one-line result |
61+
62+
## How it works
63+
64+
1. Check project `.media/manifest.jsonl` for exact-prompt match
65+
2. Scan existing `assets/` directory for unregistered files matching the need
66+
3. Check global cache `~/.media/` for reusable asset
67+
4. Search via provider (HeyGen audio catalog, HeyGen asset search)
68+
5. Freeze file to `.media/<type>/`, register in manifest, regenerate `index.md`
69+
70+
The agent gets back **one line**. Candidates, scores, provenance stay on disk.
71+
72+
## Adopt existing projects
73+
74+
Most HyperFrames projects already have assets in `assets/`. media-use adopts them:
75+
76+
```bash
77+
node <SKILL_DIR>/scripts/resolve.mjs --adopt --project .
78+
# → adopted 9 assets from assets/
79+
# bgm_001 → assets/bgm/mango-fizz.mp3 (bgm, 146.6s)
80+
# image_001 → assets/images/avatar.jpg (image, 400×400)
81+
```
82+
83+
`ffprobe` extracts real duration and dimensions. During resolve, unregistered files in `assets/` matching the intent are adopted on the fly.
84+
85+
## Reading the inventory
86+
87+
After resolve or adopt, read `.media/index.md` for the full inventory:
88+
89+
```
90+
# .media · 4 assets
91+
92+
id type dur dims path description
93+
bgm_001 bgm 25s — .media/audio/bgm/bgm_001.mp3 upbeat tech launch
94+
sfx_001 sfx 0.6s — .media/audio/sfx/sfx_001.mp3 whoosh
95+
image_001 image — 1920×1080 .media/images/image_001.jpg gradient tech background
96+
icon_001 icon — svg .media/images/icon_001.svg rocket
97+
```
98+
99+
## Cross-project reuse
100+
101+
Assets are cached automatically on resolve. Subsequent resolves for the same prompt hit the global cache at `~/.media/` — no re-download, no provider call. Promote an asset explicitly with `organize --promote <id>` to make it reusable across all projects.
102+
103+
## Files
104+
105+
- `.media/manifest.jsonl` — machine SSOT, one JSON record per line
106+
- `.media/index.md` — agent-readable table (id, type, dur, dims, path, description)
107+
- `~/.media/` — global cross-project reuse cache (content-addressed, SHA-256)
108+
109+
## CLI tools used
110+
111+
| Tool | Purpose | Required? |
112+
| --------- | ------------------------------------------ | ------------- |
113+
| `ffprobe` | Probe duration, dimensions, codec on adopt | Yes |
114+
| `heygen` | Audio catalog, asset search | For providers |
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
import { heygenSearch } from "./heygen-search.mjs";
2+
3+
export const bgmProvider = {
4+
async search(intent) {
5+
const results = heygenSearch("audio sounds list", intent, { type: "music" });
6+
if (!results) return null;
7+
const best = results[0];
8+
return {
9+
url: best.audio_url,
10+
source: "search",
11+
ext: ".mp3",
12+
metadata: {
13+
description: best.description || intent,
14+
duration: best.duration || null,
15+
provider: "heygen.audio.sounds",
16+
provenance: { track_id: best.id, score: best.score, query: intent },
17+
},
18+
};
19+
},
20+
};
Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
import { execSync } from "node:child_process";
2+
3+
export function heygenSearch(subcommand, query, { type, limit = 5, minScore } = {}) {
4+
try {
5+
const q = query.replace(/'/g, "'\\''");
6+
const parts = [`heygen --x-source media-use ${subcommand} --query '${q}'`];
7+
if (type) parts.push(`--type ${type}`);
8+
parts.push(`--limit ${limit}`);
9+
if (minScore != null) parts.push(`--min-score ${minScore}`);
10+
const out = execSync(parts.join(" "), { encoding: "utf8", timeout: 15000, stdio: ["pipe", "pipe", "pipe"] });
11+
const data = JSON.parse(out)?.data;
12+
return Array.isArray(data) && data.length > 0 ? data : null;
13+
} catch {
14+
return null;
15+
}
16+
}
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
import { heygenSearch } from "./heygen-search.mjs";
2+
3+
export const imageProvider = {
4+
async search(intent) {
5+
const results = heygenSearch("asset search list", intent, { type: "image" });
6+
if (!results) return null;
7+
const best = results[0];
8+
return {
9+
url: best.url,
10+
source: "search",
11+
ext: ".jpg",
12+
metadata: {
13+
description: intent,
14+
width: best.width || null,
15+
height: best.height || null,
16+
transparent: best.is_transparent || false,
17+
provider: "heygen.asset.search",
18+
provenance: { asset_id: best.id, score: best.score },
19+
},
20+
};
21+
},
22+
};
23+
24+
export const iconProvider = {
25+
async search(intent) {
26+
const results = heygenSearch("asset search list", intent, { type: "icon", minScore: 0.2 });
27+
if (!results) return null;
28+
const best = results[0];
29+
return {
30+
url: best.url,
31+
source: "search",
32+
ext: ".svg",
33+
metadata: {
34+
description: intent,
35+
transparent: true,
36+
provider: "heygen.asset.search",
37+
provenance: { asset_id: best.id, score: best.score, type: "icon" },
38+
},
39+
};
40+
},
41+
};
Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
import { sfxProvider } from "./sfx-provider.mjs";
2+
import { imageProvider, iconProvider } from "./image-provider.mjs";
3+
import { bgmProvider } from "./bgm-provider.mjs";
4+
5+
const STUB = { async search() { return null; } };
6+
7+
const registry = {
8+
bgm: { ...bgmProvider, type: "bgm" },
9+
sfx: { ...sfxProvider, type: "sfx" },
10+
voice: { ...STUB, type: "voice" },
11+
image: { ...imageProvider, type: "image" },
12+
icon: { ...iconProvider, type: "icon" },
13+
brand: { ...STUB, type: "brand" },
14+
};
15+
16+
export function getProvider(type) {
17+
const p = registry[type];
18+
if (!p) throw new Error(`unknown media type: ${type}`);
19+
return p;
20+
}
21+
22+
export function listTypes() {
23+
return Object.keys(registry);
24+
}
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
import { heygenSearch } from "./heygen-search.mjs";
2+
3+
export const sfxProvider = {
4+
async search(intent) {
5+
const results = heygenSearch("audio sounds list", intent, { type: "sound_effects", minScore: 0.4 });
6+
if (!results) return null;
7+
const best = results[0];
8+
return {
9+
url: best.audio_url,
10+
source: "search",
11+
ext: ".mp3",
12+
metadata: {
13+
description: best.description || best.name || intent,
14+
duration: best.duration || null,
15+
provider: "heygen.audio.sounds",
16+
provenance: { track_id: best.id, score: best.score, query: intent },
17+
},
18+
};
19+
},
20+
};

0 commit comments

Comments
 (0)