Fall back to longpoll when the websocket keeps dying after opening - #6768
Open
rdeese wants to merge 1 commit into
Open
Fall back to longpoll when the websocket keeps dying after opening#6768rdeese wants to merge 1 commit into
rdeese wants to merge 1 commit into
Conversation
`connectWithFallback` could only fall back before the primary transport had ever opened: the fallback timer, and the error handler guarded by `!established`. A websocket that opens and is then killed moments later escaped both, because `connectWithFallback` runs again on every reconnect and clears the pending fallback timer, so the client flapped forever and longpoll was never tried. Some enterprise middleboxes behave exactly this way: they let the handshake through, then kill the tunnel. Count primary connections that opened but closed again without ever delivering a message, and fall back once enough of them happen in a row. A connection that carried a message resets the count. The fallback is not memorized in this case, since the primary transport did connect. Closes phoenixframework#6766 Co-Authored-By: Claude <noreply@anthropic.com>
Member
|
So I've been thinking about this and I don't think we should go along with it for now. I'm worried that if we start to assume arbitrary middleboxes, we will end up in a rabbit hole of edge cases. Since this is the first time this comes up, I don't think it is a frequent enough problem to warrant a fix in Phoenix. If this comes up more frequently, we'll definitely reconsider. For now, I'd suggest to implement your fallback in your app.js: import {Socket, LongPoll} from "phoenix"
import {LiveSocket} from "phoenix_live_view"
const csrfToken = document
.querySelector("meta[name='csrf-token']")
.getAttribute("content")
const liveSocket = new LiveSocket("/live", Socket, {
params: {_csrf_token: csrfToken}
})
const socket = liveSocket.getSocket()
const maxUnusableConnections = 3
let opened = false
let receivedMessage = false
let unusableConnections = 0
let usingFallback = false
socket.onOpen(() => {
if(usingFallback) return
opened = true
receivedMessage = false
})
socket.onMessage(() => {
if(usingFallback) return
receivedMessage = true
unusableConnections = 0
})
socket.onClose(event => {
if(usingFallback) return
if(opened && !receivedMessage && event?.code !== 1000){
unusableConnections++
}
opened = false
receivedMessage = false
if(unusableConnections >= maxUnusableConnections){
usingFallback = true
liveSocket.replaceTransport(LongPoll)
}
})
liveSocket.connect()
window.liveSocket = liveSocket |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6766.
The bug
Socket.connectWithFallbackhas two paths to the fallback transport, and both are gated on the primary transport never having opened:fallbackTimer, andonErrorhandler, guarded byif(primaryTransport && !established).A websocket that opens and is then killed a few hundred milliseconds later escapes both.
establishedlatches on the first open, and the reconnect path re-entersconnectWithFallback, whose first statement clears the pending fallback timer — so every cycle starts the clock over.reconnectTimer.reset()inonConnOpenkeeps the backoff at its floor, and the client flaps indefinitely without ever trying longpoll.The timer that gets re-armed on open does cover the neighbouring case, where the connection stays up past
longPollFallbackMswithout answering the ping. The gap is the connection that dies before that threshold, over and over.The fix
Treat "opened but never proved useful" as primary-transport failure.
onConnClosecounts a connection that closed without ever having delivered a message; a connection that delivered one resets the count. OnceMAX_UNUSABLE_PRIMARY_CONNECTIONS(3) pile up in a row, the nextconnectWithFallbackfalls back immediately.Why "received a message" rather than time-alive: uptime is not evidence of usefulness — a middlebox can hold a socket open and drop every frame — while an inbound message proves the path works end to end. On a healthy connection that proof arrives within one RTT, because
connectWithFallbackalready pings on open, so a legitimately flaky-but-working network keeps reconnecting as it does today.The fallback is deliberately not memorized in
sessionStoragehere. The primary transport did connect, so the existing "only memorize LP if we never connected to primary" invariant stands and the next page load tries websockets again.Tests
Two tests in
assets/test/socket_test.jsdrive a fake transport under fake timers:replaceTransport(LongPoll)after 3 connections. Onmainthis test runs 11 connections in the same window and never falls back.npm testis green (180 passing, 3 skipped).Where this came from
We hit this in production with users behind enterprise security appliances that allow the websocket upgrade and then kill the established tunnel. One customer's client logged roughly 100 open/close cycles against 11 message arrivals over a week: the socket kept opening, almost never carried anything, and the fallback never engaged, so those users had no working transport path at all.
We have a reproduction harness (a node middlebox proxy that kills sockets a fixed delay after open, plus a Playwright driver) and are glad to share it or run a patch through it.
Noticed, not fixed here
errorRefis only removed when the fallback engages, so a long-lived socket accumulates one error callback per reconnect. Out of scope for this fix — happy to send it separately if you'd like it.Submitted on behalf of Rupert Deese by Claude, the AI engineering assistant at Gearflow.