Add experimental microphone denoise

[tototomate123]

Content

Adds an optional client-side microphone noise reduction feature for Element Call.
This introduces a new Background noise reduction setting, routes the local microphone through a browser-side processing pipeline before publishing to LiveKit, and uses an RNNoise worklet backend with fallback to the raw microphone track if setup fails.

Motivation and context

Potentially fixes Noise reduction #714.

Element and Matrix are great open alternatives for real-time communication, but Element Call currently lacks basic microphone processing features like noise reduction, which are common in other platforms such as Discord.

This PR introduces a simple, optional client-side noise reduction to improve call quality in everyday environments and take a first step towards a more polished calling experience.

Screenshots / GIFs

(Settings UI change only)

Tests

• Enabled and disabled Background noise reduction in a call and verified that the local microphone track is replaced with the processed track when available.
• Verified fallback to the raw microphone track when the denoise pipeline cannot be initialized.
• Verified the denoise path continues to work across microphone publish, unmute, and restart flows.
• Ran yarn tsc --noEmit.
• Ran yarn vitest src/state/CallViewModel/localMember/Publisher.test.ts.

Checklist

• I have read through CONTRIBUTING.md.
• Pull request includes screenshots or videos if containing UI changes
• Tests written for new code (and old code if feasible).
• Linter and other CI checks pass.
• I have licensed the changes to Element by completing the Contributor License Agreement (CLA)

[CrazyNicc]

See also #3819

[emmkay440]

@CrazyNicc @tototomate123, I have added a ten-vad to only open mic when voice is detected(although it doesn't clean up the voice like your PRs do), maybe instead or merging to master we could agree to merge it into a new "voice filters" branch and agree on how the filters code should flow, test it all together for a single voice filter PR to master. Idk if that'd be better, let me know what you think.
I'm yet to give your PRs a look

[fkwp]

@CrazyNicc @tototomate123, I have added a ten-vad to only open mic when voice is detected(although it doesn't clean up the voice like your PRs do), maybe instead or merging to master we could agree to merge it into a new "voice filters" branch and agree on how the filters code should flow, test it all together for a single voice filter PR to master. Idk if that'd be better, let me know what you think. I'm yet to give your PRs a look

if you would take the effort: What about also adding the DeepFilter net one #3819 as an option.

The rational offering three different solutions might be CPU cycles where VAD < RNNoise < DeepFilterNet. DeepFilterNet is also doing some kind of reverberation which is beneficial in open acoustic scenarios (aka no headset). Be aware that all assets (like AI models etc.) need to be bundled within EC rather than being downloaded and cached online.

Also we need proper testing to get it over the line.

[tototomate123]

I'd be in favor of a shared voice-filters branch/PR too. DeepFilterNet as an option sounds great, but I'm a bit worried about the asset size. If the weights are ~20 MB and have to be bundled with Element Call rather than downloaded/cached externally, we need to be careful about shipping that directly.

@fkwp how are you handling the DeepFilterNet model/assets in your implementation?

The tiered approach makes sense to me too.

[itsCodeAlchemist]

I'd be in favor of a shared voice-filters branch/PR too. DeepFilterNet as an option sounds great, but I'm a bit worried about the asset size. If the weights are ~20 MB and have to be bundled with Element Call rather than downloaded/cached externally, we need to be careful about shipping that directly.

@fkwp how are you handling the DeepFilterNet model/assets in your implementation?

The tiered approach makes sense to me too.

From a user perspective, one of Element’s main features is the ability to establish calls between users. For calls longer than 5 minutes, audio quality should be the priority. If that requires bundling a "heavier" package, that’s acceptable.

[fkwp]

I'd be in favor of a shared voice-filters branch/PR too. DeepFilterNet as an option sounds great, but I'm a bit worried about the asset size. If the weights are ~20 MB and have to be bundled with Element Call rather than downloaded/cached externally, we need to be careful about shipping that directly.

@fkwp how are you handling the DeepFilterNet model/assets in your implementation?

The tiered approach makes sense to me too.

This script is downloading the weights and then its packaged. The weights are in the order of 5-7 MB.

At least for the messenger apps its important to bundle it alongside as it's possible to operate the Apps without proper internet access.

In case of a SPA deployment the client would only download the blob once you enable noise reduction.

Another reason for bundling is that you do not rely on a CDN providing the weights.

[xaetacorenet]

Can this be implemented, it would be great if this would be in the app even as a experimental feature

[Lama-Thematique]

As a matrix user and host of an instance anything under 50MB for such a core feature should not be a problem at all.

Users of my matrix instance complained many times about audio quality and returned promptly to discord when any of the users in the call have any level of background noise, not implementing anything because it take 7MB or 20MB is honestly shooting yourself in the foot.

I think some level of denoising is a must have for a communication platform.

In my experience i prefer DeepFilterNet (implement here #3819) but either is massive step forward.

[FrozenAlex]

I tested it on my own system, seems to work well.

[xaetacorenet]

My suggestion is merging this PR so people have noise cancelling/filtering(Which is one of the biggest frustrations on element-call, hearing keyboards and background noises all the time) then later making another PR for the DeepFilter or further improvements.

This would make voice calls alot more bearable because the number one reason people refuse to use it right now in our and many other instances is because of the lack of noise filtering.

This PR in its current state has been tested and works great, what would be stopping us from taking this win and improving on it in the future?

[tototomate123]

@fkwp Sorry for the delayed response. Thanks, that clears up my concern. At 5–7 MB, the model size seems reasonable, and bundling it for embedded apps while lazyloading it in the SPA makes sense. Avoiding a runtime CDN dependency is a good argument too.

I also agree with @xaetacorenet that merging the experimental RNNoise implementation first and improving the processing layer afterward may be the most pragmatic path. It provides a useful improvement now without preventing DeepFilterNet, VAD, or a shared backend abstraction later.

At this point, I'm mainly waiting for guidance from the maintainers. What specifically would need to change or be tested before this PR could move forward? I'm happy to address concrete requirements, whether that means additional tests, architectural changes, or rebasing/updating the implementation.