Skip to content

Commit cc90f37

Browse files
ConalMullanConalMullan
committed
Merge feature/acestep-music-gen: Add ACE-Step 1.5 music generation (v0.12.0)
2 parents 6060c43 + 3092bd8 commit cc90f37

8 files changed

Lines changed: 2037 additions & 8 deletions

File tree

‎.claude/skills/acestep/SKILL.md‎

Lines changed: 264 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,264 @@
1+
---
2+
name: acestep
3+
description: AI music generation with ACE-Step 1.5 — background music, vocal tracks, covers, stem extraction for video production. Use when generating music, soundtracks, jingles, or working with audio stems. Triggers include background music, soundtrack, jingle, music generation, stem extraction, cover, style transfer, or musical composition tasks.
4+
---
5+
6+
# ACE-Step 1.5 Music Generation
7+
8+
Open-source music generation (MIT license) via `tools/music_gen.py`. Runs on RunPod serverless.
9+
Requires `RUNPOD_API_KEY` and `RUNPOD_ACESTEP_ENDPOINT_ID` in `.env` (run `--setup` to create endpoint).
10+
11+
## Quick Reference
12+
13+
```bash
14+
# Basic generation
15+
python tools/music_gen.py --prompt "Upbeat tech corporate" --duration 60 --output bg.mp3
16+
17+
# With musical control
18+
python tools/music_gen.py --prompt "Calm ambient piano" --duration 30 --bpm 72 --key "D Major" --output ambient.mp3
19+
20+
# Scene presets (video production)
21+
python tools/music_gen.py --preset corporate-bg --duration 60 --output bg.mp3
22+
python tools/music_gen.py --preset tension --duration 20 --output problem.mp3
23+
python tools/music_gen.py --preset cta --brand digital-samba --duration 15 --output cta.mp3
24+
25+
# Vocals with lyrics
26+
python tools/music_gen.py --prompt "Indie pop jingle" --lyrics "[verse]\nBuild it better\nShip it faster" --duration 30 --output jingle.mp3
27+
28+
# Cover / style transfer
29+
python tools/music_gen.py --cover --reference theme.mp3 --prompt "Jazz piano version" --duration 60 --output jazz_cover.mp3
30+
31+
# Stem extraction
32+
python tools/music_gen.py --extract vocals --input mixed.mp3 --output vocals.mp3
33+
34+
# List presets
35+
python tools/music_gen.py --list-presets
36+
```
37+
38+
## Creating a Song (Step by Step)
39+
40+
### 1. Instrumental background track (simplest)
41+
```bash
42+
python tools/music_gen.py --prompt "Upbeat indie rock, driving drums, jangly guitar" --duration 60 --bpm 120 --key "G Major" --output track.mp3
43+
```
44+
45+
### 2. Song with vocals and lyrics
46+
Write lyrics in a temp file or pass inline. Use structure tags to control song sections.
47+
48+
```bash
49+
# Write lyrics to a file first (recommended for longer songs)
50+
cat > /tmp/lyrics.txt << 'LYRICS'
51+
[Verse 1]
52+
Walking through the morning light
53+
Coffee in my hand feels right
54+
Another day to build and dream
55+
Nothing's ever what it seems
56+
57+
[Chorus - anthemic]
58+
WE KEEP MOVING FORWARD
59+
Through the noise and doubt
60+
We keep moving forward
61+
That's what it's about
62+
63+
[Verse 2]
64+
Screens are glowing late at night
65+
Shipping code until it's right
66+
The deadline's close but so are we
67+
Almost there, just wait and see
68+
69+
[Chorus - bigger]
70+
WE KEEP MOVING FORWARD
71+
Through the noise and doubt
72+
We keep moving forward
73+
That's what it's about
74+
75+
[Outro - fade]
76+
(Moving forward...)
77+
LYRICS
78+
79+
# Generate the song
80+
python tools/music_gen.py \
81+
--prompt "Upbeat indie rock anthem, male vocal, driving drums, electric guitar, studio polish" \
82+
--lyrics "$(cat /tmp/lyrics.txt)" \
83+
--duration 60 \
84+
--bpm 128 \
85+
--key "G Major" \
86+
--output my_song.mp3
87+
```
88+
89+
### 3. Using a preset for video background
90+
```bash
91+
python tools/music_gen.py --preset tension --duration 20 --output problem_scene.mp3
92+
```
93+
94+
### Key tips for good results
95+
- **Caption = overall style** (genre, instruments, mood, production quality)
96+
- **Lyrics = temporal structure** (verse/chorus flow, vocal delivery)
97+
- **UPPERCASE in lyrics** = high vocal intensity
98+
- **Parentheses** = background vocals: "We rise (together)"
99+
- **Keep 6-10 syllables per line** for natural rhythm
100+
- **Don't describe the melody in the caption** — describe the *sound* and *feeling*
101+
- **Use `--seed`** to lock randomness when iterating on prompt/lyrics
102+
103+
## Scene Presets
104+
105+
| Preset | BPM | Key | Use Case |
106+
|--------|-----|-----|----------|
107+
| `corporate-bg` | 110 | C Major | Professional background, presentations |
108+
| `upbeat-tech` | 128 | G Major | Product launches, tech demos |
109+
| `ambient` | 72 | D Major | Overview slides, reflective content |
110+
| `dramatic` | 90 | D Minor | Reveals, announcements |
111+
| `tension` | 85 | A Minor | Problem statements, challenges |
112+
| `hopeful` | 120 | C Major | Solution reveals, resolutions |
113+
| `cta` | 135 | E Major | Call to action, closing energy |
114+
| `lofi` | 85 | F Major | Screen recordings, coding demos |
115+
116+
## Task Types
117+
118+
### text2music (default)
119+
Generate music from text prompt + optional lyrics.
120+
121+
### cover
122+
Style transfer from reference audio. Control blend with `--cover-strength` (0.0-1.0):
123+
- **0.2** — Loose style inspiration (more creative freedom)
124+
- **0.5** — Balanced style transfer
125+
- **0.7** — Close to original structure (default)
126+
- **1.0** — Maximum fidelity to source
127+
128+
### extract
129+
Stem separation — isolate individual tracks from mixed audio.
130+
Tracks: `vocals`, `drums`, `bass`, `guitar`, `piano`, `keyboard`, `strings`, `brass`, `woodwinds`, `other`
131+
132+
### repaint (future)
133+
Regenerate a specific time segment within existing audio while preserving the rest.
134+
135+
### lego (future, requires base model)
136+
Generate individual instrument tracks within an existing audio context.
137+
138+
### complete (future, requires base model)
139+
Extend partial compositions by adding specified instruments.
140+
141+
## Prompt Engineering
142+
143+
### Caption Writing — Layer Dimensions
144+
145+
Write captions by layering multiple descriptive dimensions rather than single-word descriptions.
146+
147+
**Dimensions to include:**
148+
- **Genre/Style**: pop, rock, jazz, electronic, lo-fi, synthwave, orchestral
149+
- **Emotion/Mood**: melancholic, euphoric, dreamy, nostalgic, intimate, tense
150+
- **Instruments**: acoustic guitar, synth pads, 808 drums, strings, brass, piano
151+
- **Timbre**: warm, crisp, airy, punchy, lush, polished, raw
152+
- **Era**: "80s synth-pop", "modern indie", "classical romantic"
153+
- **Production**: lo-fi, studio-polished, live recording, cinematic
154+
- **Vocal**: breathy, powerful, falsetto, raspy, spoken word (or "instrumental")
155+
156+
**Good**: "Slow melancholic piano ballad with intimate female vocal, warm strings building to powerful chorus, studio-polished production"
157+
**Bad**: "Sad song"
158+
159+
### Key Principles
160+
161+
1. **Specificity over vagueness** — describe instruments, mood, production style
162+
2. **Avoid contradictions** — don't request "classical strings" and "hardcore metal" simultaneously
163+
3. **Repetition reinforces priority** — repeat important elements for emphasis
164+
4. **Sparse captions = more creative freedom** — detailed captions constrain the model
165+
5. **Use metadata params for BPM/key** — don't write "120 BPM" in the caption, use `--bpm 120`
166+
167+
### Lyrics Formatting
168+
169+
**Structure tags** (use in lyrics, not caption):
170+
```
171+
[Intro]
172+
[Verse]
173+
[Chorus]
174+
[Bridge]
175+
[Outro]
176+
[Instrumental]
177+
[Guitar Solo]
178+
[Build]
179+
[Drop]
180+
[Breakdown]
181+
```
182+
183+
**Vocal control** (prefix lines or sections):
184+
```
185+
[raspy vocal]
186+
[whispered]
187+
[falsetto]
188+
[powerful belting]
189+
[harmonies]
190+
[ad-lib]
191+
```
192+
193+
**Energy indicators:**
194+
- UPPERCASE = high intensity ("WE RISE ABOVE")
195+
- Parentheses = background vocals ("We rise (together)")
196+
- Keep 6-10 syllables per line within sections for natural rhythm
197+
198+
**Example — Tech Product Jingle:**
199+
```
200+
[Verse]
201+
Build it better, ship it faster
202+
Every feature tells a story
203+
204+
[Chorus - anthemic]
205+
THIS IS YOUR PLATFORM
206+
Your vision, your stage
207+
Digital Samba, every page
208+
209+
[Outro - fade]
210+
(Build it better...)
211+
```
212+
213+
## Video Production Integration
214+
215+
### Music for Scene Types
216+
217+
| Scene | Preset | Duration | Notes |
218+
|-------|--------|----------|-------|
219+
| Title | `dramatic` or `ambient` | 3-5s | Short, mood-setting |
220+
| Problem | `tension` | 10-15s | Dark, unsettling |
221+
| Solution | `hopeful` | 10-15s | Relief, optimism |
222+
| Demo | `lofi` or `corporate-bg` | 30-120s | Non-distracting, matches demo length |
223+
| Stats | `upbeat-tech` | 8-12s | Building credibility |
224+
| CTA | `cta` | 5-10s | Maximum energy, punchy |
225+
| Credits | `ambient` | 5-10s | Gentle fade-out |
226+
227+
### Timing Workflow
228+
229+
1. Plan scene durations first (from voiceover script)
230+
2. Generate music to match: `--duration <scene_seconds>`
231+
3. Music duration is precise (within 0.1s of requested)
232+
4. For background music spanning multiple scenes: generate one long track
233+
234+
### Combining with Voiceover
235+
236+
Background music should be mixed at 10-20% volume in Remotion:
237+
```tsx
238+
<Audio src={staticFile('voiceover.mp3')} volume={1} />
239+
<Audio src={staticFile('bg-music.mp3')} volume={0.15} />
240+
```
241+
242+
For music under narration: use instrumental presets (`corporate-bg`, `ambient`, `lofi`).
243+
For music-forward scenes (title, CTA): can use higher volume or vocal tracks.
244+
245+
### Brand Consistency
246+
247+
Use `--brand <name>` to load hints from `brands/<name>/brand.json`.
248+
Use `--cover --reference brand_theme.mp3` to create variations of a brand's sonic identity.
249+
For consistent sound across a project: fix the seed (`--seed 42`) and vary only duration/prompt.
250+
251+
## Technical Details
252+
253+
- **Output**: 48kHz MP3/WAV/FLAC
254+
- **Duration range**: 10-600 seconds
255+
- **BPM range**: 30-300
256+
- **Inference**: ~2-3s on GPU (turbo, 8 steps), ~40-60s on Mac MPS
257+
- **Turbo model**: 8 steps, no CFG needed, fast and good quality
258+
- **Shift parameter**: 3.0 recommended for turbo (improves quality)
259+
260+
### When NOT to use ACE-Step
261+
- **Voice cloning** — use Qwen3-TTS or ElevenLabs instead
262+
- **Sound effects** — use ElevenLabs SFX (`tools/sfx.py`)
263+
- **Speech/narration** — use voiceover tools, not music gen
264+
- **Stem extraction from video** — extract audio first with FFmpeg, then use `--extract`

‎CLAUDE.md‎

Lines changed: 33 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with th
99
**Key capabilities:**
1010
- Programmatic video creation with Remotion (React-based)
1111
- AI voiceover generation with ElevenLabs or Qwen3-TTS
12+
- AI music generation with ACE-Step 1.5 (text-to-music, vocals, covers, stems)
1213
- Browser demo recording with Playwright
1314
- Asset processing with FFmpeg
1415

@@ -107,9 +108,9 @@ pip install -r tools/requirements.txt
107108

108109
| Type | Tools | When to Use |
109110
|------|-------|-------------|
110-
| **Project tools** | voiceover, music, sfx, sync_timing | During video creation workflow |
111+
| **Project tools** | voiceover, music, music_gen, sfx, sync_timing | During video creation workflow |
111112
| **Utility tools** | redub, addmusic, notebooklm_brand, locate_watermark | Quick transformations on existing videos |
112-
| **Cloud GPU** | image_edit, upscale, dewatermark, sadtalker, qwen3_tts | AI processing via RunPod |
113+
| **Cloud GPU** | image_edit, upscale, dewatermark, sadtalker, qwen3_tts, music_gen | AI processing via RunPod |
113114

114115
Utility tools work on any video file without requiring a project structure.
115116

@@ -159,6 +160,7 @@ echo "RUNPOD_API_KEY=your_key_here" >> .env
159160
python tools/image_edit.py --setup
160161
python tools/upscale.py --setup
161162
python tools/qwen3_tts.py --setup
163+
python tools/music_gen.py --setup
162164

163165
# Image editing (Qwen-Image-Edit)
164166
python tools/image_edit.py --input photo.jpg --prompt "Add sunglasses"
@@ -173,6 +175,35 @@ python tools/upscale.py --input photo.jpg --scale 2 --model anime --face-enhance
173175

174176
See `docs/qwen-edit-patterns.md` and `.claude/skills/qwen-edit/` for prompting guidance.
175177

178+
### AI Music Generation (ACE-Step 1.5)
179+
180+
```bash
181+
# Background music with precise control
182+
python tools/music_gen.py --prompt "Upbeat tech corporate" --duration 60 --bpm 128 --key "G Major" --output music.mp3
183+
184+
# Scene presets for video production
185+
python tools/music_gen.py --preset corporate-bg --duration 60 --output bg.mp3
186+
python tools/music_gen.py --preset tension --duration 20 --output problem.mp3
187+
python tools/music_gen.py --preset cta --brand digital-samba --output cta.mp3
188+
189+
# Song with vocals and lyrics (use structure tags for sections)
190+
python tools/music_gen.py \
191+
--prompt "Indie pop anthem, male vocal, bright guitar, studio polish" \
192+
--lyrics "[Verse]\nWalking through the morning light\nCoffee in my hand feels right\n\n[Chorus - anthemic]\nWE KEEP MOVING FORWARD\nThrough the noise and doubt\n\n[Outro - fade]\n(Moving forward...)" \
193+
--duration 60 --bpm 128 --key "G Major" --output song.mp3
194+
195+
# Cover / style transfer
196+
python tools/music_gen.py --cover --reference theme.mp3 --prompt "Jazz piano version" --output cover.mp3
197+
198+
# Stem extraction
199+
python tools/music_gen.py --extract vocals --input mixed.mp3 --output vocals.mp3
200+
201+
# List presets
202+
python tools/music_gen.py --list-presets
203+
```
204+
205+
8 scene presets: `corporate-bg`, `upbeat-tech`, `ambient`, `dramatic`, `tension`, `hopeful`, `cta`, `lofi`. See `.claude/skills/acestep/` for prompt engineering patterns and video production integration guide.
206+
176207
### Watermark Removal
177208

178209
```bash

‎README.md‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ An AI-native video production workspace for [Claude Code](https://claude.ai/code
2323
> - `ghcr.io/conalmullan/video-toolkit-sadtalker` — Talking head generation (SadTalker)
2424
> - `ghcr.io/conalmullan/video-toolkit-qwen3-tts` — Text-to-speech (Qwen3-TTS)
2525
> - `ghcr.io/conalmullan/video-toolkit-flux2` — Text-to-image & editing (FLUX.2 Klein 4B)
26+
> - `ghcr.io/conalmullan/video-toolkit-acestep` — AI music generation (ACE-Step 1.5)
2627
>
2728
> My motto: **Be brave. Experiment.** And please share any videos you create or ideas you have back with the project — it helps me keep improving this toolkit for everyone.
2829
@@ -48,7 +49,7 @@ Clone this repo, open it in Claude Code, and start creating videos.
4849
| [Claude Code](https://docs.anthropic.com/en/docs/claude-code) | **Required** | AI assistant for video creation |
4950
| [Python](https://python.org/) 3.9+ | Optional | AI voiceover, audio tools |
5051
| [FFmpeg](https://ffmpeg.org/) | Optional | Media conversion |
51-
| [RunPod account](https://runpod.io/) | Optional | Cloud GPU (TTS, image editing) |
52+
| [RunPod account](https://runpod.io/) | Optional | Cloud GPU (TTS, image editing, music gen) |
5253
| [ElevenLabs API key](https://elevenlabs.io/) | Optional | Premium AI voices |
5354

5455
### Try It Now
@@ -112,6 +113,7 @@ Claude Code has deep knowledge in:
112113
| **playwright-recording** | Browser automation — record demos as video |
113114
| **frontend-design** | Visual design refinement for distinctive, production-grade aesthetics |
114115
| **qwen-edit** | AI image editing — prompting patterns and best practices |
116+
| **acestep** | AI music generation — prompts, lyrics, scene presets, video integration |
115117
| **runpod** | Cloud GPU — setup, Docker images, endpoint management, costs |
116118

117119
### Commands
@@ -220,9 +222,13 @@ python tools/voiceover.py --script script.md --output voiceover.mp3
220222
python tools/voiceover.py --provider qwen3 --speaker Ryan --scene-dir public/audio/scenes --json
221223
python tools/qwen3_tts.py --text "Hello world" --tone warm --output hello.mp3
222224

223-
# Generate background music
225+
# Generate background music (ElevenLabs)
224226
python tools/music.py --prompt "Upbeat corporate" --duration 120 --output music.mp3
225227

228+
# Generate background music (ACE-Step — free, precise BPM/key control)
229+
python tools/music_gen.py --preset corporate-bg --duration 120 --output music.mp3
230+
python tools/music_gen.py --prompt "Dramatic cinematic" --duration 30 --bpm 90 --key "D Minor" --output reveal.mp3
231+
226232
# Generate sound effects
227233
python tools/sfx.py --preset whoosh --output sfx.mp3
228234

@@ -264,9 +270,9 @@ python tools/flux2.py --list-presets
264270

265271
| Type | Tools | Purpose |
266272
|------|-------|---------|
267-
| **Project** | voiceover, music, sfx | Used during video creation workflow |
273+
| **Project** | voiceover, music, music_gen, sfx | Used during video creation workflow |
268274
| **Utility** | redub, addmusic, notebooklm_brand, locate_watermark | Quick transformations, no project needed |
269-
| **Cloud GPU** | image_edit, upscale, dewatermark, sadtalker, qwen3_tts, flux2 | AI processing via RunPod (see below) |
275+
| **Cloud GPU** | image_edit, upscale, dewatermark, sadtalker, qwen3_tts, flux2, music_gen | AI processing via RunPod (see below) |
270276

271277
See [docs/runpod-setup.md](docs/runpod-setup.md) for Cloud GPU tool setup.
272278

@@ -282,6 +288,7 @@ Cloud GPU tools use pre-built Docker images deployed to RunPod serverless:
282288
| sadtalker | `ghcr.io/conalmullan/video-toolkit-sadtalker:latest` | 24GB (RTX 4090) |
283289
| qwen3_tts | `ghcr.io/conalmullan/video-toolkit-qwen3-tts:latest` | 24GB (ADA) |
284290
| flux2 | `ghcr.io/conalmullan/video-toolkit-flux2:latest` | 24GB (ADA) |
291+
| music_gen | `ghcr.io/conalmullan/video-toolkit-acestep:latest` | 24GB+ (AMPERE/ADA) |
285292

286293
Dockerfiles and handlers are in `docker/`. Run `python tools/<tool>.py --setup` to auto-deploy.
287294

0 commit comments

Comments
 (0)