feat: adding ten-vad wasm to project, configuration options on audio#4047
feat: adding ten-vad wasm to project, configuration options on audio#4047emmkay440 wants to merge 1 commit into
Conversation
|
Thanks for the contribution. So the testing for this needs to involve mobile testing specifically to evaluate if we have perf issues. |
|
This issue fits: #562 |
We have been trying to get in touch with the devs to get #3892 looked at or at least merged, the developer who worked on that PR is equally frustrated by the lack of communication for such a highly requested feature |
Adding TEN-VAD voice activity detection gate for local microphone
What and why
This PR adds an optional voice activity detection (VAD) gate applied to the local microphone track before it is published to the SFU. When enabled, audio is silenced during non-speech segments, reducing unwanted noises from being transmitted.
The gate is implemented using TEN-VAD and it runs entirely inside an AudioWorklet on the audio thread, no IPC round-trip to the main thread giving approximately 16 ms detection latency(customizeable).
Before this change there was no client-side VAD gate.
After this change users can opt into a VAD gate from Audio Settings.
The setting is off by default and persisted in localStorage.
Architecture
public/vad/ten_vad.wasm+ten_vad.jsTEN-VAD Emscripten build.src/livekit/TenVadProcessor.worklet.tsAudioWorklet; wraps the WASM synchronously, decimates 48 kHz -> 16 kHz at 3:1, runs VAD every hop (10-16 ms), applies asymmetric gain ramp.src/livekit/TenVadTransformer.tsLiveKitTrackProcessoradapter; fetches and compiles the WASM once (module is cached and reused across restarts), wires the Web Audio graph.src/settings/settings.tsseven newSettingentries for VAD state and parameters.src/settings/SettingsModal.tsxVAD section added to the Audio tab with Disabled / Simple / Advanced radio buttons and the corresponding controls.src/state/CallViewModel/localMember/Publisher.tsapplyTenVad()subscribes to the local mic track and thevadEnabledsetting, attaches or detachesTenVadTransformerreactively, and pushes parameter changes to the live worklet.Testing
node /tmp/yarn4.js backendNotes
Claude helped me with the implementation but going forward with this PR I can do all changes by myself if asked to.
An enhancement to this PR would be adding some sort of voice feedback loop so users can test for themselves.
Reopening this PR.