feat(backends): audio generation — ThinkSound (SFX) + ACE-Step (music) by pwilkin · Pull Request #2530 · lemonade-sdk/lemonade

pwilkin · 2026-07-02T09:28:39Z

Per #2529 : adds support for the ThinkSound model engine (sound generation) and the ACE-Step engine (music generation).

jeremyfowers

I'm really excited to see these features land!

Please see requests above for GUI fixes and test infrastructure, which will help me further review the PR.

fl0rianr

Thanks for adding this — the audio generation direction looks useful and the backend integration is a good start. Jeremy already covered the GUI mode issue, ROCm version pins, tests, and error-buffer handling. I left one additional comment on response_format/MIME consistency below.

Add the thinksound and acestep recipes — standalone WrappedServer backends — plus the audio-generation surface they share: the IAudioGenerationServer capability, the AUDIO_GENERATION model type, and the POST /audio/generations endpoint (text -> audio clip, quad-prefix). ThinkSound wraps ts-server for sound effects; ACE-Step wraps ace-server's async job API for music. Models: ThinkSound-SFX, ACE-Step-Music.

- pin rocm-stable + cuda builds for thinksound/acestep (Linux + Windows) and add CUDA loader-path/env handling mirroring sd-cpp - surface backend error payloads from /audio/generations as HTTP 500 JSON instead of a 200 audio body; ACE-Step reports failure reasons - restrict response_format to what the backend natively produces (supported_audio_formats(), wav for now) with 400 otherwise; transcoding left as TODO - add test/server_audio_generation.py and CI matrix entries (thinksound/acestep x vulkan/rocm on Windows + Linux) - port the audio-generation panel + mode switch from feat/mm-ui so selecting an ACE-Step/ThinkSound model leaves LLM Chat Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv

pwilkin · 2026-07-03T11:55:35Z

Aight, I think that should be all the changes :)

fl0rianr

Thanks for the follow-up fixes — the response_format/MIME issue looks addressed now. I left two remaining comments that seem separate from Jeremy’s existing review: one around backend-option plumbing for the new selectable backends, and one around ROCm CI prewarming for the new audio-generation tests.

jeremyfowers

Once the API documentation is in I will be happy with the structure of this PR. I'm heading out on vacation soon, so I'm tagging a few more reviewers to help land this.

@ramkrishna2910 - new multimedia capabilities! Please try them out and ensure the API docs look good to you, then approve.

@superm1 - new ROCm integration! Please verify that it meets your patterns. If it doesn't I would suggest pulling the ROCm integration from this PR, then merge it, then add ROCm support back in later. It's still a very nice PR even if it only has Vulkan support.

@kenvandine / @fl0rianr - new CUDA support, please test it out on Nvidia hardware, then approve.

jeremyfowers · 2026-07-03T14:52:26Z

PS. @pwilkin I realized that my two main comments on this PR (needs tests, and needs API docs) can be traced back to holes in the adding-a-backend.md guide. Here is a PR to improve that documentation: #2541

If you have any other suggestions for improving that guide please comment on that PR!

pwilkin · 2026-07-03T15:07:51Z

@jeremyfowers I think @fl0rianr 's remarks regarding how to declare backends are also useful, maybe adding the SD.cpp code as an example.

…, document endpoint - declare thinksound_backend/acestep_backend descriptor options and select the backend from RecipeOptions in load() (sd-cpp pattern) so per-model options and option-change reloads carry it - add thinksound/acestep to the test harness TheRock prewarm allow-list - document POST /v1/audio/generations in docs/api/lemonade.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv

pwilkin · 2026-07-03T15:28:00Z

Aight, newest round of fixes is in.

The slim ROCm engine binaries link LLVM OpenMP, which TheRock ships under lib/llvm/lib rather than lib/, so ace-server exited with 127 (libomp.so not found) on CI runners. Prepend that dir to the loader path alongside the main TheRock lib dir. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv

pwilkin · 2026-07-03T16:10:47Z

Can someone please reapprove the workflows? Needed a fix for ROCm. @jeremyfowers @fl0rianr

ramkrishna2910 · 2026-07-03T16:14:49Z

Can someone please reapprove the workflows? Needed a fix for ROCm. @jeremyfowers @fl0rianr

Done

pwilkin · 2026-07-03T16:44:15Z

Looking good now, don't think the MacOS failure is caused by my changes?

fl0rianr

Great work so far. Vulkan works, but I'm not able to select CUDA. Typically we would prefer rocm and cuda first before vulkan (as the auto default). If you are aware of stability issues it's fine going for vulkan first.

Small UI polish issue: the native number-input spinner for Duration looks visually broken/misaligned in the dark audio-generation panel. The field is very narrow and the browser up/down buttons dominate the control. Very typical issue appearing with this UI element upon Initial implementation trials.

fl0rianr

Very nice, CUDA ist working as well!

A minor topic, we have this fully UI element again correctly separated now but still the dark mode issue.

My local test worked but additionally CI runs are failing now.
It looks related to this selection not actually taking effect for the current registry entry. ThinkSound-SFX is registered as a bare repo checkpoint (ilintar/thinksound-gguf) with no :variant, and the shared HF downloader only calls select_checkpoint_files() inside the !main_variant.empty() branch. For an empty variant it falls back to downloading all repo files.

That matches the failing CI server logs: the ThinkSound test still tries to download dit-q8.gguf and t5-q8.gguf, even though this backend intends to skip the redundant Q8 variants. Those extra large downloads then hit hash/resume/retry issues and make the audio-gen jobs flaky/slow.

Can we make backend-specific select_checkpoint_files() apply even when main_variant is empty, or give the ThinkSound registry entry a variant/sentinel that routes through the backend selector? Otherwise CI and users will still download the full repo instead of the intended ~6.4 GB subset.

- select_checkpoint_files() was only consulted when a GGUF variant was present, so bare-repo checkpoints (ThinkSound-SFX has none) fell through to "download every file in the repo" instead of the intended 6-file subset, re-adding the redundant q8 variants and the CI timeouts they caused. Check the backend's selection regardless of variant. - add `color-scheme: dark` (this app has no light theme): without it, native form controls like the audio Duration spinner render with browser-default light chrome no matter what our CSS says, which is what the fixed -webkit-appearance hack was still showing on some platforms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv

pwilkin · 2026-07-04T14:30:32Z

Aight, hopefully we're done now 😃

pwilkin · 2026-07-04T15:16:37Z

Argh. This reminds me why I hate DevOps / staging stuff. Will add the Windows libomp and that should fix the two remaining tests.

Confirmed via objdump on the actual release DLLs: ggml-cpu.dll and ggml-base.dll both import libomp140.x86_64.dll, and TheRock's Windows distribution keeps it under lib/llvm/bin -- a sibling of the main bin/ dir that get_therock_lib_path() returns, not a subdirectory of it (mirrors the existing Linux lib/llvm/lib fix, verified against TheRock's actual archive layout). The Windows PATH construction only added the main bin/ dir, so ace-server/ts-server exited within ~1s of starting on Windows ROCm CI runners -- exactly the failure seen in the Test .exe - audio-gen-acestep job (acestep-server failed to start). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv

fl0rianr · 2026-07-04T15:21:17Z

Yeah, nearly there but CI is still off.

Another minor thing:
The spinner fix works for dark theme, but this now forces native form controls to use dark browser chrome in light theme as well. styles.css sets color-scheme: dark globally on body, while the app still supports [data-theme="light"].

Can we make this theme-aware instead of hardcoding dark? For example, define --native-color-scheme: dark in variables-dark.css, --native-color-scheme: light in variables-light.css, and use color-scheme: var(--native-color-scheme, dark) on body. That keeps the dark-theme spinner fix while allowing light theme controls to render correctly.

fl0rianr · 2026-07-04T15:22:44Z

I let the workflow run, so you can take a look, the spinner fix should be an easy one you can do after CI is green (besides the MacOS one)

pwilkin · 2026-07-04T15:24:35Z

@fl0rianr yeah I swear they did something to nerf Opus, I don't remember it being so obtuse with respect to UX, that's twice now that it's done really bad CSS fixes. I honestly think I'll actually just code this one by hand to make sure ;)

fl0rianr · 2026-07-04T16:31:54Z

The remaining CI failure looks centered on the new audio-generation jobs. Since the workflow currently runs both new backends across vulkan and rocm on Windows and Linux, I’d suggest narrowing required CI first: keep full /audio/generations coverage on the known-stable backend/platform path, and move the other variants to load/install smoke tests or optional follow-up jobs until the external backend artifacts are stable.

Also, ThinkSound failures are hard to debug right now because /generate is just forwarded; unlike ACE-Step, the backend error body is not logged before Lemonade returns 500. Adding explicit logging for the ThinkSound backend response body/status would make the next CI failure actionable.

Windows ROCm builds no longer depend on libomp140.x86_64.dll (both now build with GGML_OPENMP=OFF; TheRock's Windows dist never shipped that DLL in the first place). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv

…/lemonade into feat/mm-audio-generation

pwilkin · 2026-07-04T19:44:58Z

Okay, so the way forward was I basically rewrote the engines to use GGML's native threadpool instead of OpenMP, which cleared the problem with OpenMP library dependencies. Hopefully it works now.

pwilkin · 2026-07-04T20:21:24Z

Nope: data type size mismatch with Linux vs Windows in the backends (overflow on 2GB+ reads). Needs another rebuild and version bump. Sigh :)

Fixes a 32-bit fseek truncation on Windows that broke loading of the 2.4-2.6GB bf16 checkpoints (dit-bf16.gguf, t5-bf16.gguf) on any backend -- surfaced in CI as "t5 load failed" on the Windows Vulkan test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv

fl0rianr

Good work, Thanks! CI fully clean, just a minor nit to fix dark theme with the spinner, see below.

@ramkrishna2910 @superm1 can you please take a look if you can approve as well

Add color-scheme styling for audio duration input.

Two /pull requests for the same model (e.g. a client that timed out and retried while the first download was still running) wrote the same .partial files concurrently, corrupting them so hash verification failed with a different digest on every retry-from-scratch cycle. Take a per-checkpoint-repo lock around download_registered_model so the second request waits and then validates the finished files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv

ramkrishna2910 · 2026-07-05T13:51:02Z

Thanks for the detailed review @fl0rianr. I will test drive this shortly.

ramkrishna2910

Thanks for the contribution @pwilkin, this is a great addition to list of modalities supported :)

…am/main Merges upstream's ThinkSound/ACE-Step backend rows (lemonade-sdk#2530) with our gfx942 additions in the generated table. gen_backend_boilerplate.py --check passes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

lemonade-sdk#2530) * feat(backends): audio generation — ThinkSound (SFX) + ACE-Step (music) Add the thinksound and acestep recipes — standalone WrappedServer backends — plus the audio-generation surface they share: the IAudioGenerationServer capability, the AUDIO_GENERATION model type, and the POST /audio/generations endpoint (text -> audio clip, quad-prefix). ThinkSound wraps ts-server for sound effects; ACE-Step wraps ace-server's async job API for music. Models: ThinkSound-SFX, ACE-Step-Music. * fix(audio-generation): address review feedback - pin rocm-stable + cuda builds for thinksound/acestep (Linux + Windows) and add CUDA loader-path/env handling mirroring sd-cpp - surface backend error payloads from /audio/generations as HTTP 500 JSON instead of a 200 audio body; ACE-Step reports failure reasons - restrict response_format to what the backend natively produces (supported_audio_formats(), wav for now) with 400 otherwise; transcoding left as TODO - add test/server_audio_generation.py and CI matrix entries (thinksound/acestep x vulkan/rocm on Windows + Linux) - port the audio-generation panel + mode switch from feat/mm-ui so selecting an ACE-Step/ThinkSound model leaves LLM Chat Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv * fix(audio-generation): declare backend options, prewarm TheRock in CI, document endpoint - declare thinksound_backend/acestep_backend descriptor options and select the backend from RecipeOptions in load() (sd-cpp pattern) so per-model options and option-change reloads carry it - add thinksound/acestep to the test harness TheRock prewarm allow-list - document POST /v1/audio/generations in docs/api/lemonade.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv * fix(audio-generation): resolve libomp from TheRock's llvm/lib on ROCm The slim ROCm engine binaries link LLVM OpenMP, which TheRock ships under lib/llvm/lib rather than lib/, so ace-server exited with 127 (libomp.so not found) on CI runners. Prepend that dir to the loader path alongside the main TheRock lib dir. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv * fix(audio-generation): UI backend selector, honest model sizes, spinner CSS - register thinksound/acestep in the frontend recipe-options config so the Model Options modal offers the backend selector (Auto/ROCm/CUDA/Vulkan) - prefer rocm/cuda over vulkan as the auto default (sd-cpp order) - ThinkSound: only download the six GGUFs ts-server loads by default, dropping the redundant q8 variants (11.4 -> 6.4 GiB; also keeps the CI model pull inside the harness timeout) - correct registry sizes (GiB) and refresh a downloaded model's size from its aggregate on-disk payload at cache build, so the UI reports what is actually installed - hide the native number-input spinner on the audio Duration field Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv * fix(audio-ui): widen Duration field and keep the native spinner The earlier fix hid the number-input spinner entirely; restore it and widen the field (64px -> 96px) so the control no longer looks cramped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DfH8NmdasFseZDiwg7D6Cc * fix(audio-generation): ThinkSound full-repo download + dark-mode spinner - select_checkpoint_files() was only consulted when a GGUF variant was present, so bare-repo checkpoints (ThinkSound-SFX has none) fell through to "download every file in the repo" instead of the intended 6-file subset, re-adding the redundant q8 variants and the CI timeouts they caused. Check the backend's selection regardless of variant. - add `color-scheme: dark` (this app has no light theme): without it, native form controls like the audio Duration spinner render with browser-default light chrome no matter what our CSS says, which is what the fixed -webkit-appearance hack was still showing on some platforms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv * fix(audio-generation): find libomp140.x86_64.dll on Windows ROCm Confirmed via objdump on the actual release DLLs: ggml-cpu.dll and ggml-base.dll both import libomp140.x86_64.dll, and TheRock's Windows distribution keeps it under lib/llvm/bin -- a sibling of the main bin/ dir that get_therock_lib_path() returns, not a subdirectory of it (mirrors the existing Linux lib/llvm/lib fix, verified against TheRock's actual archive layout). The Windows PATH construction only added the main bin/ dir, so ace-server/ts-server exited within ~1s of starting on Windows ROCm CI runners -- exactly the failure seen in the Test .exe - audio-gen-acestep job (acestep-server failed to start). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv * chore(audio-generation): bump thinksound/acestep pins to v0.1.1 Windows ROCm builds no longer depend on libomp140.x86_64.dll (both now build with GGML_OPENMP=OFF; TheRock's Windows dist never shipped that DLL in the first place). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv * fix: remove inline styles, add proper CSS with no theme forcing * feat: proper width for input * chore(thinksound): bump pin to v0.1.2 Fixes a 32-bit fseek truncation on Windows that broke loading of the 2.4-2.6GB bf16 checkpoints (dit-bf16.gguf, t5-bf16.gguf) on any backend -- surfaced in CI as "t5 load failed" on the Windows Vulkan test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv * Enhance audio duration input with color-scheme Add color-scheme styling for audio duration input. * fix: serialize concurrent downloads of the same checkpoint repo Two /pull requests for the same model (e.g. a client that timed out and retried while the first download was still running) wrote the same .partial files concurrently, corrupting them so hash verification failed with a different digest on every retry-from-scratch cycle. Take a per-checkpoint-repo lock around download_registered_model so the second request waits and then validates the finished files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EbpKMPt3rkoxTRJMaDwWXv --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

github-actions Bot added the audio label Jul 2, 2026

jeremyfowers requested review from fl0rianr and jeremyfowers July 2, 2026 15:19

jeremyfowers assigned pwilkin Jul 2, 2026

jeremyfowers reviewed Jul 2, 2026

View reviewed changes

Comment thread docs/assets/models.js

jeremyfowers reviewed Jul 2, 2026

View reviewed changes

Comment thread src/cpp/resources/backend_versions.json

jeremyfowers reviewed Jul 2, 2026

View reviewed changes

Comment thread docs/assets/models.js

jeremyfowers reviewed Jul 2, 2026

View reviewed changes

Comment thread src/cpp/server/server.cpp

jeremyfowers requested changes Jul 2, 2026

View reviewed changes

fl0rianr reviewed Jul 2, 2026

View reviewed changes

Comment thread src/cpp/server/server.cpp Outdated

pwilkin and others added 2 commits July 3, 2026 10:11

pwilkin force-pushed the feat/mm-audio-generation branch from f60119f to cc49266 Compare July 3, 2026 11:27

fl0rianr reviewed Jul 3, 2026

View reviewed changes

Comment thread src/cpp/include/lemon/backends/thinksound/thinksound.h Outdated

Comment thread test/utils/server_base.py

jeremyfowers reviewed Jul 3, 2026

View reviewed changes

Comment thread .github/workflows/cpp_server_build_test_release.yml

jeremyfowers mentioned this pull request Jul 3, 2026

docs: cover tests and API docs in adding-a-backend guide #2541

Merged

jeremyfowers requested review from kenvandine and ramkrishna2910 July 3, 2026 14:46

jeremyfowers approved these changes Jul 3, 2026

View reviewed changes

jeremyfowers requested a review from superm1 July 3, 2026 14:49

fl0rianr reviewed Jul 3, 2026

View reviewed changes

Comment thread src/cpp/resources/server_models.json Outdated

Comment thread src/cpp/include/lemon/backends/acestep/acestep_server.h

fl0rianr reviewed Jul 4, 2026

View reviewed changes

pwilkin and others added 5 commits July 4, 2026 20:17

fix: remove inline styles, add proper CSS with no theme forcing

62c4918

Merge branch 'feat/mm-audio-generation' of https://github.com/pwilkin…

c7c8f89

…/lemonade into feat/mm-audio-generation

feat: proper width for input

06aa40e

Merge branch 'main' into feat/mm-audio-generation

c475590

fl0rianr approved these changes Jul 4, 2026

View reviewed changes

Comment thread src/app/styles/styles.css

pwilkin and others added 2 commits July 5, 2026 12:43

Enhance audio duration input with color-scheme

0298758

Add color-scheme styling for audio duration input.

superm1 approved these changes Jul 6, 2026

View reviewed changes

ramkrishna2910 approved these changes Jul 7, 2026

View reviewed changes

ramkrishna2910 added this pull request to the merge queue Jul 7, 2026

Merged via the queue into lemonade-sdk:main with commit b1fd12e Jul 7, 2026
93 checks passed

Uh oh!

Conversation

pwilkin commented Jul 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jeremyfowers left a comment

Choose a reason for hiding this comment

Uh oh!

fl0rianr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

pwilkin commented Jul 3, 2026

Uh oh!

fl0rianr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jeremyfowers left a comment

Choose a reason for hiding this comment

Uh oh!

jeremyfowers commented Jul 3, 2026

Uh oh!

pwilkin commented Jul 3, 2026

Uh oh!

pwilkin commented Jul 3, 2026

Uh oh!

pwilkin commented Jul 3, 2026

Uh oh!

ramkrishna2910 commented Jul 3, 2026

Uh oh!

pwilkin commented Jul 3, 2026

Uh oh!

fl0rianr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

fl0rianr left a comment

Choose a reason for hiding this comment

Uh oh!

pwilkin commented Jul 4, 2026

Uh oh!

pwilkin commented Jul 4, 2026

Uh oh!

fl0rianr commented Jul 4, 2026

Uh oh!

fl0rianr commented Jul 4, 2026

Uh oh!

pwilkin commented Jul 4, 2026

Uh oh!

fl0rianr commented Jul 4, 2026

Uh oh!

pwilkin commented Jul 4, 2026

Uh oh!

pwilkin commented Jul 4, 2026

Uh oh!

fl0rianr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ramkrishna2910 commented Jul 5, 2026

Uh oh!

ramkrishna2910 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants