Intermittent join/signal failures after moving from livekit-client-js 2.17.x to 2.18.1 against livekit-server 1.9.12

[dbastos]

Describe the bug

Our baseline was stable on livekit-client-js 2.17.x.

After moving to 2.18.1 (server still on 1.9.12), we started seeing errors like:

• could not handle new participant
• request canceled

We then tested server 1.10.1, but the problem did not clearly disappear in our environment. Rolling the client back to 2.17.x appeared to normalize behavior again on our side.

Because of that, I'm not sure whether this is:

1. a regression/change in client behavior in 2.18.1,
2. a compatibility problem with server 1.9.12,
3. or a transport/proxy path issue that 2.18.1 is exposing more easily.

Observed behavior:

• Intermittent participant join/restart failures
• Some sessions appear to connect and then stall
• Media freeze
• We also observed DTLS/data-channel related warnings during the bad periods

Reproduction

SDK js 2.18.1 with v1.9.12.

Logs

Earlier participant join failure:

ERROR livekit routing/localrouter.go:111 could not handle new participant
{"room": "sporting-c_0407193619645", "participant": "552a637b-d178-471b-86a8-b4ce714b25ab", "connID": "CO_ANHZbD7fTVmz", "error": "request canceled"}

During later testing we also saw transport/proxy related errors:

caddy-1 | {"level":"error","logger":"layer4.handlers.proxy","msg":"upstream connection","local_address":"127.0.0.1:57986","remote_address":"127.0.0.1:7880","error":"local error: tls: bad record MAC"}

livekit-1 | WARN livekit.transport rtc/transport.go:924 error reading data channel
{"transport":"PUBLISHER","label":"_reliable","error":"dtls timeout: read/write timeout: context deadline exceeded"}

livekit-1 | ERROR livekit service/signal.go:186 could not handle new participant
{"room":"sporting-c_0407014153277","participant":"7f0e7c25-6097-42e4-86c3-901d6e6f4f72","connID":"CO_gCxdsGVcZZSc","error":"could not restart participant"}

System Info

Ubuntu 24.04;
Chrome latest version;
Our deployment is Linux + Docker Compose, as per the official documentation.

Severity

annoyance

Additional Information

We also saw warnings like could not get packet from bucket / received packet too old on VP8 video tracks during the same bad periods, even when packet loss in the stats was reported as 0, while jitter and PLI counts were high.

Questions

• Is there any known behavior change in livekit-client-js 2.18.1 that could explain more fragile joins/restarts against livekit-server 1.9.12?
• Is 2.18.1 expected to be fully safe with server 1.9.12, especially behind caddyl4?
• Do the logs above suggest a known transport/signal issue rather than a pure client regression?

These errors start appearing a few seconds after the room is active. I don't know if it was a coincidence, but both rooms that had problems had more than 400 subscribers.

[dbastos]

Hi folks!
I need rollback to 2.15.3. Above this, it collapses into a room with more than 350,400 subscribers.
Cya!
Doug

[dbastos]

Need rollback server to 1.9.0. Very unstable the new versions.

[boks1971]

@dbastos sorry about that.

Could you explain the set up some more? When you mentioned 350 - 400 subscribers in a room, how many are publishing tracks and how many are just subscribing? Are the publishers sending both audio and video and what bit rates?

How do things fail at that scale with the latest server and latest SDK?

Would you happen to have server logs and client logs?

[lukasIO]

@dbastos you mention

Our baseline was stable on livekit-client-js 2.17.x.

and in a later comment

I need rollback to 2.15.3.

Is there anything that changed in your setup that would explain why 2.17.x appeared to be stable and then didn't any more?

[dbastos]

Hi folks! Thks for the feedback!

@boks1971

Could you explain the set up some more? When you mentioned 350 - 400 subscribers in a room, how many are publishing tracks and how many are just subscribing? Are the publishers sending both audio and video and what bit rates?

I have only 1 publisher by room. So in this mention i have 1 publisher and 350/400 subscribers. The publisher send audio and video at 640x360@1.5mpbs. Adaptative stream, dynacast disabled.

How do things fail at that scale with the latest server and latest SDK?

In the newer versions, the subscriber receives the image much faster than in older versions. Something introduced in the newer versions is changing the connection behavior enough to cause it to crash under "high load".

Would you happen to have server logs and client logs?

Unfortunately, I don't have the client logs because I disable them in production. Server log attached.

@lukasIO
Sorry for the misunderstanding. Actually, when I posted, things seemed normal, as I rarely have rooms with more than 350/400 subscribers. It took me a while to realize that what was actually causing the problem in the new versions was this number of subscribers.
I noticed that rooms with many subscribers started to exhibit this behavior, even in version 2.17.x. So, for safety, I reverted the server to 1.9.0 and the SDK to 2.15.3.
It took almost a week for me to notice this problem, as I had updated my server, and only after a week did I experience this high subscriber load.

The bare metal specs:
2x Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz
10GbE NIC
64 GB ram

202604071-log.txt

Cya!
Doug

[boks1971]

Thank you @dbastos .

Not seeing much in the server logs. The errors in there are due to trying to use data channel of a participant after it is closed. That is not catastrophic. Do not see any crashes. Have you profiled the server? Maybe, there is some other bottleneck.

[dbastos]

Hi @boks1971 and @lukasIO !

I think our production issue may sit at the intersection of #1852 and #1915 rather than matching only one of them.

What we are seeing is not just "some normal disconnects."" In large rooms, once the session starts getting busy, we get a dense burst of paired _lossy + _reliable data-channel aborts across many participants in the same room, very close together in time, and those bursts are mixed with repeated dtls timeout: read/write timeout: context deadline exceeded events. After that, the room starts degrading further and participant restarts/rejoins begin to fail.

That makes me think the sequence is something like this:

1. under load, the room enters a degraded state and starts producing negotiation pressure / transport instability,
2. that triggers concurrent renegotiation/reconnect behavior on many clients,
3. the behavior described in Coalesce burst negotiations and avoid full-reconnect on single NegotiationError under concurrent signaling #1915 may turn some of those negotiation failures into unnecessary full reconnects,
4. and then the behavior described in DefaultReconnectPolicy causes thundering herd in large rooms (650+ participants) #1852 can amplify that into a thundering-herd reconnect pattern in large rooms.

So from my side, #1915 looks like a plausible client-side trigger/escalation path, while #1852 looks like the large-room amplifier once many participants start reconnecting at nearly the same time.

This also matches our own version testing:

• we were stable before on livekit-server 1.9.0 + livekit-client 2.15.3,
• the problem became visible after moving to newer client/server combinations,
• and in production the failures become much more obvious once the room gets large.

I am not claiming that every single abort chunk log line is itself a bug. The concern is the pattern:

• many paired aborts,
• many participants,
• same room,
• short window,
• mixed with DTLS timeouts,
• followed by reconnect/rejoin instability.

This behavior shown in the logs happens as soon as the number of subscribers scales. In my case, the room grows from 1 to 350/400 in a matter of minutes. Then, after approximately 5 minutes, the session starts to degrade. In other words, it's a moment when users are entering the room, not when they are voluntarily leaving.

That pattern looks much more like an abnormal reconnect cascade than ordinary user churn.

My suggestion would be to check this as a combined failure mode:

• bursty renegotiation handling / concurrent negotiate() paths,
• escalation to full reconnect after a single negotiation failure,
• and reconnect backoff behavior in large rooms once many clients enter reconnecting at the same time.

The issues #1852 + #1915 together seem like the closest public explanation for the production behavior we are seeing.

CYA!
Doug