Skip to content

feat: adding ten-vad wasm to project, configuration options on audio#4047

Open
emmkay440 wants to merge 1 commit into
element-hq:livekitfrom
emmkay440:feat/tenvad
Open

feat: adding ten-vad wasm to project, configuration options on audio#4047
emmkay440 wants to merge 1 commit into
element-hq:livekitfrom
emmkay440:feat/tenvad

Conversation

@emmkay440

Copy link
Copy Markdown

Adding TEN-VAD voice activity detection gate for local microphone

What and why

This PR adds an optional voice activity detection (VAD) gate applied to the local microphone track before it is published to the SFU. When enabled, audio is silenced during non-speech segments, reducing unwanted noises from being transmitted.

The gate is implemented using TEN-VAD and it runs entirely inside an AudioWorklet on the audio thread, no IPC round-trip to the main thread giving approximately 16 ms detection latency(customizeable).

Before this change there was no client-side VAD gate.
After this change users can opt into a VAD gate from Audio Settings.

The setting is off by default and persisted in localStorage.

Architecture

getUserMedia -> LocalAudioTrack
                   └─ TenVadTransformer (LiveKit TrackProcessor)
                         └─ AudioWorkletNode (TenVadProcessor)
                               ├─ TenVADRuntime (WASM, synchronous)
                               └─ gain ramp -> processedTrack -> SFU
  • public/vad/ten_vad.wasm + ten_vad.js TEN-VAD Emscripten build.
  • src/livekit/TenVadProcessor.worklet.ts AudioWorklet; wraps the WASM synchronously, decimates 48 kHz -> 16 kHz at 3:1, runs VAD every hop (10-16 ms), applies asymmetric gain ramp.
  • src/livekit/TenVadTransformer.ts LiveKit TrackProcessor adapter; fetches and compiles the WASM once (module is cached and reused across restarts), wires the Web Audio graph.
  • src/settings/settings.ts seven new Setting entries for VAD state and parameters.
  • src/settings/SettingsModal.tsx VAD section added to the Audio tab with Disabled / Simple / Advanced radio buttons and the corresponding controls.
  • src/state/CallViewModel/localMember/Publisher.ts applyTenVad() subscribes to the local mic track and the vadEnabled setting, attaches or detaches TenVadTransformer reactively, and pushes parameter changes to the live worklet.

Testing

  1. Run node /tmp/yarn4.js backend
  2. Open two instances of the same call to talk to yourself.
  3. Open a call and go to Settings -> Audio.
  4. Gently scratch microphone in a way you can hear it, breath heavily next to it and try hitting your table or putting a big cup on the desk next to the mic.
  5. Under Voice Activity Detection, select Simple.
  6. Perform step 3 again and you shouldn't be able to hear those noises anymore.
  7. Switch back to Disabled
  8. Reproduce step 3 again and confirm you can hear the unwanted noises again.

Notes

Claude helped me with the implementation but going forward with this PR I can do all changes by myself if asked to.
An enhancement to this PR would be adding some sort of voice feedback loop so users can test for themselves.
Reopening this PR.

@emmkay440 emmkay440 requested a review from a team as a code owner June 17, 2026 01:38
@emmkay440 emmkay440 requested a review from Half-Shot June 17, 2026 01:38
@toger5

toger5 commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Thanks for the contribution.
Can you create a issue as described in the CONTRIBUTING.md.
Or link this to an existing issue that this solves.
We are currently having trouble maintaining the blur background PR on mobile as the track processors are not performing well there.

So the testing for this needs to involve mobile testing specifically to evaluate if we have perf issues.

@emmkay440

emmkay440 commented Jun 26, 2026

Copy link
Copy Markdown
Author

This issue fits: #562
which links to this PR essentially implementing same functionality albeit a different implementation.
There has also been some discussion here #3892 (comment) on merging multiple voice filter functionalities together for a single big commit with VAD and denoisers, I think we could use a maintainer input on weather multiple smaller commits are preferred or one big one, and if all the options make sense for the project.
Contributors.md also mention features should be test covered, and I haven't as of now added any testing, so I'll mend that when I'm able to, that might take a bit.

@xaetacorenet

Copy link
Copy Markdown

this PR essentially implementing same functionality albeit a different implementation.
There has also been some discussion here #3892 (comment) on merging multiple voice filter functionalities together

We have been trying to get in touch with the devs to get #3892 looked at or at least merged, the developer who worked on that PR is equally frustrated by the lack of communication for such a highly requested feature

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants