Skip to content

Keeping audio and clock queue in sync.#4178

Closed
filipi87 wants to merge 4 commits into
mainfrom
filipi/sync_clock_audio_queue
Closed

Keeping audio and clock queue in sync.#4178
filipi87 wants to merge 4 commits into
mainfrom
filipi/sync_clock_audio_queue

Conversation

@filipi87
Copy link
Copy Markdown
Contributor

@filipi87 filipi87 commented Mar 27, 2026

Summary

  • When a TTSStoppedFrame carries a presentation timestamp (pts), it is ow enqueued in both the audio_queue and the clock_queue so that timed frames (e.g. word-boundary events) and audio playback stay in sync.
  • When the audio task finishes sending all preceding audio and dequeues the TTSStoppedFrame, it signals the clock task via _clock_flush_event to stop waiting on timestamps and drain all pending frames immediately.
  • The clock task signals back via _clock_drained_event once the drain is complete, so the audio task can push TTSStoppedFrame downstream.
  • A BOT_VAD_STOP_FALLBACK_SECS timeout guards the wait on _clock_drained_event to prevent a deadlock if the clock task fails before completing the drain.

@filipi87 filipi87 marked this pull request as draft March 27, 2026 21:16
@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 27, 2026

Codecov Report

❌ Patch coverage is 0% with 26 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/pipecat/transports/base_output.py 0.00% 26 Missing ⚠️
Files with missing lines Coverage Δ
src/pipecat/transports/base_output.py 17.67% <0.00%> (-1.05%) ⬇️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@filipi87 filipi87 marked this pull request as ready for review March 27, 2026 21:29
@filipi87
Copy link
Copy Markdown
Contributor Author

filipi87 commented Apr 6, 2026

I tested all TTS services that provide word timestamps:

  • Rime
  • ElevenLabs
  • Inworld
  • Gradium
  • Hume
  • ResembleAI
  • Azure
  • Cartesia

Only Rime is returning incorrect timestamps.

However, the fix in the PR is not specifically addressing the Rime issue. Instead, it acts as a safeguard to ensure both queues stay in sync by using the TTSStopped frame as a gatekeeper.

But closing it for now, since Rime is currently the only service that would benefit from this change.

@filipi87 filipi87 closed this Apr 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant