Register client disconnects while streaming TTS audio.#634
Conversation
No need to continue generating for clients that have hung up. This reduces latency processing the next request.
orbitalquark
changed the title
| if sample_rate is None: | ||
| sample_rate = result.sample_rate | ||
|
|
||
| await asyncio.sleep(0) # register any disconnects |
There was a problem hiding this comment.
This can cause the server performance to drop significantly.
There was a problem hiding this comment.
@Blaizzy I don't know how else to allow Python to temporarily pass control back to fastapi to register client disconnects. If I remove that line, this PR will do nothing.
For what it's worth, I streamed generated audio for over an hour with a stream interval of 0.5 on a MacBook Air. No hiccups at all. If the server performance had dropped significantly, then normal performance might be overkill.
There was a problem hiding this comment.
Ok, let's merge for now and revisit it later.
Could you open a github issue with describing the issue and preferably a video so I we can reproduce and fix it later without using asyncio.sleep(0)?
Blaizzy
left a comment
Blaizzy
left a comment
There was a problem hiding this comment.
Mostly looks good to me, just have one nit before we merge
|
See #634 (comment). |
2 participants
No need to continue generating for clients that have hung up. This reduces latency processing the next request.
I have a client application that splits long-form text into paragraphs and feeds each paragraph to an mlx-audio server sequentially. The client has a "next" button that stops listening for the current paragraph and POSTs the next one for playback. However, without disconnect detection, the server continues generating the previous request and tries sending to the disconnected client, resulting in unnecessary latency. This PR fixes that.