Feature/Initial Livekit MatrixRTC Integration

[herrfeder]

I introduced an example rtc_livekit_join which shows how to use matrix-sdk-rtc-livekit to join/leave a LiveKit room with Element-Call based SFrame Encryption and also with MatrixRTC call memberships. Initially this was done to have a headless video streaming towards Element Call from embedded devices like Raspberry Pis and Jetson Nanos, therefore only video is covered here and no audio yet.

I could also structure this functionality into an self-contained example without wiring it directly into the sources of the Rust-SDK but I guess this feature is something at least some people are looking for and therefore I guess it's beneficial to bring it into the Rust-SDK Core.

To make this example work, I introduced the following:

• a new crate matrix-sdk-rtc-livekit which
  • holds all relevant structures that are necessary to maintain the state of the Livekit Room Driver during an Element Call Session and also a structure for handling when Livekit Room Events are happening (for example increasing ratchet key index)
  • trigger the relevant Element Call Events (Membership and Hangup) which works via also introduced functions in matrix-sdk/widget
  • organizes the per-participant key provision and also receiving and parsing of encryption_keys events of other participants

Please see the readme of the example https://github.com/herrfeder/matrix-rust-sdk/blob/feature/livekit-webrtc-integration/examples/rtc_livekit_join/README.md for more detailed information and also how to use the necessary environment variables for the example.

Current workarounds (I'm not happy with):

• It seems the newest Livekit SDK Source is broken livekit-api uses types that don't exist in livekit-protocol livekit/rust-sdks#1080 and therefore depending on the initial branch which made the livekit rust sdk compatible with Element Call

• The Livekit webrtc-sys doesn't like OpenSSL Bindings (at least on my machine) and therefore makes usage of rusttls-tls necessary and therefore needed to modify more Cargo dependencies than desired

• the introduction of Livekit SDK and especially webrtc-sys introduces some C++ dependencies that needs some special handling, after a lot of try and error and dead ends I had over multiple Linux platforms (x86, various ARM) the best results with g++-13 and gcc-13 which differs from the current dependencies for the runner setup for this repository (happy about recommendations to deal more gracefully with it)

• I've documented the public API Changes in the appropriate CHANGELOG.md files.

• This PR was made with the help of AI.

Signed-off-by: herrfeder 14598229+herrfeder@users.noreply.github.com

[codspeed-hq]

Merging this PR will not alter performance

✅ 50 untouched benchmarks

---

_{Comparing herrfeder:feature/livekit-webrtc-integration (28b20cf) with main (1fe2c88)}

[herrfeder]

It seems the mentioned C++ dependencies of webrtc-sys are breaking the current pipeline setup, I will introduce additional modifications to fix that.

[codecov]

Codecov Report

❌ Patch coverage is 11.63462% with 919 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.17%. Comparing base (777ce05) to head (28b20cf).
⚠️ Report is 28 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
crates/matrix-sdk/src/widget/element_call.rs	0.00%	363 Missing ⚠️
...ates/matrix-sdk-rtc-livekit/src/per_participant.rs	13.19%	281 Missing and 2 partials ⚠️
crates/matrix-sdk-rtc-livekit/src/lib.rs	22.80%	261 Missing and 3 partials ⚠️
crates/matrix-sdk/src/widget/capabilities.rs	0.00%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6564      +/-   ##
==========================================
- Coverage   89.92%   89.17%   -0.75%     
==========================================
  Files         381      384       +3     
  Lines      106860   107900    +1040     
  Branches   106860   107900    +1040     
==========================================
+ Hits        96095    96224     +129     
- Misses       7108     8023     +915     
+ Partials     3657     3653       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[Hywan]

Hello,
Thank you for your contribution!
I've a couple of questions before starting to review this.

As this is not a small contribution, I need a bit more context:

• Why do we need a livekit specific integration? What would we gain from that?
• Did you vibe-code this contribution? You're mentioning the use of AI/LLM (thanks for being honest): did you read our AI policy?
• Is it possible to split the giant first commit (+5000LOC) into smaller relevant, atomic, documented, commits please?

[herrfeder]

Hello, Thank you for your contribution! I've a couple of questions before starting to review this.

As this is not a small contribution, I need a bit more context:

* Why do we need a livekit specific integration? What would we gain from that?

* Did you vibe-code this contribution? You're mentioning the use of AI/LLM (thanks for being honest): did you read [our AI policy](https://github.com/matrix-org/matrix-rust-sdk/blob/main/CONTRIBUTING.md#ai-policy)?

* Is it possible to split the giant first commit (+5000LOC) into smaller relevant, atomic, documented, commits please?

yes, this was my first experiment to vibecode such a huge functionality but I took reasonable time and care to clean up the mess (strip down bloated function/trait signatures, remove dead code and distribute functionality to reasonable locations and files). Actually it's only about 1500 additional LOC. The big blob is the changed Cargo.lock, that has a lot of changes because of the Livekit/WebRTC dependencies.

The test coverage is yet not really satisfying but before doing even more effort into something you may not appreciate at all I decided to go public with this effort. Actually the questions on my side are:

• yeah, does it need a Livekit specific SDK functionality or should it be a self-contained SDK extension (for example)?
• I guess, as much existing Ruma utilities as possible should be used instead of own implementations if applicable?

With the feedback/decision if a Livekit specific SDK crate is a worthsome addition to the Rust-SDK, I can for sure break this down to smaller more digestable PRs?

[shaba]

I've been experimenting with adding audio/video call support to Fractal (the GNOME Matrix client, GTK4/Rust, no embedded WebView) in my spare time, and ran into exactly the gap this PR closes. Posting this in case it's useful context for the direction discussion above.

Pulled livekit and libwebrtc straight from crates.io, everything builds, we join the room, tracks get published and subscribed — but the audio is unintelligible in both directions. I spent some time chasing DSP hypotheses: gain reduction, WebRTC APM with various AGC/HPF/NS combinations, low-pass against clipping harmonics, RED on/off — none of it helped. Then it became clear why: what reaches the Opus decoder isn't damaged audio, it's encrypted bytes.

Confirmed against the livekit server source and logs:

• The publish log for the Element Call track shows encryption: "GCM".
• pkg/rtc/wrappedreceiver.go:115 in livekit-server explicitly skips the RED↔Opus translation when the source is encrypted, so the SFU proxies the encrypted RTP payload through as-is.
• My RoomOptions::default() has e2ee: None, no KeyProvider attached, so encrypted bytes get fed into the Opus decoder verbatim.

Without per-participant key distribution over Matrix (encryption_keys, encrypted to-device per MSC4143) wired into livekit::e2ee::KeyProvider, a native MatrixRTC client can't talk to Element Call intelligibly. Issue #3295 raised this exact point two years ago and explicitly named Fractal. As far as I can see, your matrix-sdk-rtc-livekit is the first upstream code that actually closes that gap.

A note on the widget-driver path (#4963, #4990): it's a reasonable option for Element X and similar hosts that can embed a WebView. But it doesn't generalise. A native GTK or Qt desktop client has no sensible way to ship a WebKit runtime plus the entire Element Call JS bundle just to make m.call.member work. The Raspberry Pi / Jetson use case in this PR's description makes the same point from the embedded side. If widget becomes the only path, native clients stay without RTC support — which is where Fractal is today.

matrix-rust-sdk has always been positioned as an SDK any Rust Matrix client can build on. For that to remain true for RTC, it'd be useful to have a native path alongside the widget path, not replaced by it.

If the maintainers had a moment, two questions from a downstream hobbyist's perspective:

• Is there any near-term intent to support some native MatrixRTC E2EE path upstream — not necessarily this PR's exact code? Even "intent yes, no ETA" would help downstream projects decide whether to wait, contribute review effort, or stop-gap locally.
• If the Livekit-specific scope here is the main concern, would a smaller PR adding only the matrix-side primitives for key exchange (independent of any specific WebRTC backend) be easier to review?

I'm not in a position to commit to anything, but if audio support lands in this branch I'll try it against Fractal locally and report back what I see.