Skip to content

fix(qwen3-tts-cpp): hold qwentts.cpp at 35ebe537, upstream master hangs in synthesis - #11286

Merged
mudler merged 1 commit into
masterfrom
fix/qwentts-revert-hanging-pin
Aug 1, 2026
Merged

fix(qwen3-tts-cpp): hold qwentts.cpp at 35ebe537, upstream master hangs in synthesis#11286
mudler merged 1 commit into
masterfrom
fix/qwentts-revert-hanging-pin

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Problem

tests-qwen3-tts-cpp has been red on master since 2026-07-31. The suite loads every component successfully and then stops:

Will run 27 of 27 specs
••••••••••••••••[qwen3-tts-cpp] Loading talker=... codec=...
[Talker] Loaded: 28 layers, hidden 1024, ...
[CodePredictor] Loaded: 5 layers, ...
[Quantizer] Loaded: 16 codebooks ...
[Transformer] Loaded: 8 layers, ...
panic: test timed out after 20m0s

TTS() never returns from the native call:

goroutine 74 [syscall, 19 minutes]:
github.com/ebitengine/purego.RegisterFunc.func4
github.com/mudler/LocalAI/backend/go/qwen3-tts-cpp.(*Qwen3TtsCpp).TTS
    backend/go/qwen3-tts-cpp/goqwen3ttscpp.go:154
github.com/mudler/LocalAI/backend/go/qwen3-tts-cpp.init.func2.4
    backend/go/qwen3-tts-cpp/e2e_test.go:90

A job that normally takes about 5 minutes runs into the 20 minute Go test timeout instead.

Not a flake. Reproduced on master (run 30689716806) and again on an explicit re-run of the same job.

Which bump did it

Bisected across this cycle's eight qwentts.cpp bumps using each PR's own tests-qwen3-tts-cpp check:

Bump PR Pin tests-qwen3-tts-cpp
#10832 7bb91886 pass, 4m38s
#10850 9e11ce41 pass, 4m58s
#10902 95b4840a pass, 5m9s
#10964 e93292be pass, 5m15s
#11006 ba4c7f78 pass, 5m33s
#11039 82cd05b9 pass, 4m58s
#11127 35ebe537 pass, 5m1s
#11241 abab6b3 fail, 1h58m26s

#11241 was merged with this check already red, which is how the hang reached master.

Upstream cause

35ebe537..abab6b3 is three upstream commits:

  • 26dd8adb predictor: unroll the frame into one cgraph and sample in standard ops
  • 9a9a4252 tests: exercise the fused decode in every cossim harness
  • abab6b3b tests: refresh the clone grid logs

Only 26dd8adb is functional; the other two are test-only. Rewriting the predictor's sampling into standard ops is consistent with a generation loop that never reaches its stop condition, which is exactly what an unbounded native call looks like from the Go side.

Fix

Hold the pin at 35ebe537, the last known-good commit.

The bump_deps entry is commented out rather than left in place: it tracks upstream master, so the nightly job would put the pin straight back onto the hang. Both the Makefile and the workflow carry a comment pointing at the other, so the hold is discoverable rather than mysterious, and restoring it is uncommenting four lines once upstream is fixed.

Why this blocks the release

This is not only a CI problem. TTS() hanging means a real qwen3-tts-cpp synthesis request never returns, so the backend wedges for users on the current master. Found during the pre-release CI sweep for v4.8.0.

Follow-up (not in this PR)

Report 26dd8adb upstream to ServeurpersoCom/qwentts.cpp with this bisection and stack trace, then restore the bump once it is fixed.

…gs in synthesis

tests-qwen3-tts-cpp has been failing on master since 2026-07-31. The suite
loads every component fine and then stops: TTS() never returns from the
native call, so a job that takes ~5 minutes runs into the 20 minute Go test
timeout instead.

    goroutine 74 [syscall, 19 minutes]:
    github.com/ebitengine/purego.RegisterFunc.func4
    qwen3-tts-cpp.(*Qwen3TtsCpp).TTS  goqwen3ttscpp.go:154
    qwen3-tts-cpp.init.func2.4        e2e_test.go:90

Not a flake: reproduced on master and again on an explicit re-run.

Bisected across this cycle's eight qwentts.cpp bumps by their own check:
10832, 10850, 10902, 10964, 11006, 11039 and 11127 all pass in ~5 minutes;
11241 (abab6b3) fails at 1h58m. That PR was merged with this check already
red, which is how the hang reached master.

35ebe537..abab6b3 is three upstream commits, and the only functional one is
26dd8adb, "predictor: unroll the frame into one cgraph and sample in standard
ops", which is consistent with a generation loop that never reaches its stop
condition.

Hold the pin at the last known-good commit. The bump entry is commented out
rather than left in place, because it tracks upstream master and would put
the hang straight back on the next nightly run. Both spots carry a pointer to
the other so the hold is discoverable, and restoring it is uncommenting four
lines once upstream is fixed.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
@localai-bot

Copy link
Copy Markdown
Collaborator Author

Reported upstream: ServeurpersoCom/qwentts.cpp#19

CI has now confirmed the hold works: tests-qwen3-tts-cpp passes in 5m15s on this PR, against 1h58m (timeout) on the abab6b3 pin. That matches the historical baseline for this job exactly, so 35ebe537 is confirmed good rather than merely older.

Restoring the bump is: revert the pin, and uncomment the four bump_deps lines.

@mudler
mudler merged commit 21ecc79 into master Aug 1, 2026
65 of 71 checks passed
@mudler
mudler deleted the fix/qwentts-revert-hanging-pin branch August 1, 2026 21:40
@localai-bot

Copy link
Copy Markdown
Collaborator Author

Correcting the mechanism described in the commit message, now that the upstream fix is understood.

This commit (and issue #19 as originally filed) says TTS() never returns. That is imprecise. It is a runaway, not a deadlock, and the CI log says so directly if you read past the panic:

[Pipeline] Generated 2048 frames (slot 0)
[Pipeline] Generation done : 2048 frames
panic: test timed out after 20m0s

Generation runs to the 2048-frame cap instead of stopping at EOS around frame 17-28, so each synthesis does roughly 100x the work and the suite blows its 20 minute budget. The call does terminate, just far too late to be useful.

Root cause upstream, from ServeurpersoCom/qwentts.cpp#20: 26dd8ad moved the sub-talker draw into a graph tail that crosses the cdf in descending probability order, while the host sampler it replaced walks the cdf in vocabulary order. Both are valid multinomial samplers with the same marginal, but they map a given philox u to different tokens. Every stochastic acoustic code changed, the talker went off-manifold and stopped emitting codec_eos. Greedy was unaffected, which is why the commit's own greedy parity check passed.

None of this changes what this PR does or why. Holding the pin at 35ebe537 is still the right call for the release, and tests-qwen3-tts-cpp passing here in 5m15s is unchanged evidence. Once #20 lands upstream, restoring the bump is reverting the pin and uncommenting the four bump_deps lines.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants