Fix ineffective websocket heartbeat timeouts and reconnect stampedes

[ignatremizov]

Summary

Some websocket heartbeat paths appear to be ineffective or overly chatty across browser clients. This was noticed while inspecting repeated ping / pong! traffic from a local stack, then confirmed against hcengineering/platform develop.

Findings

1. HulypulseClient and PresenceClient clear and recreate a 5 minute ping timeout on every 30 second ping tick. Because the interval is shorter than the timeout, the timeout can be cleared before it fires.

   • packages/hulypulse-client/src/client.ts: PING_INTERVAL_MS = 30s, PING_TIMEOUT_MS = 5m, startPing() clears the previous timeout each interval.
   • plugins/presence-resources/src/client.ts: PING_INTERVAL = 30s, PING_TIMEOUT = 5m, startPing() has the same pattern.

2. In HulypulseClient, even if the ping timeout callback fires, it only closes when this.ws?.readyState !== WebSocket.OPEN. For a dead-but-open socket that stops returning pong, that condition is false, so the reconnect path may not run.

3. There are multiple heartbeat layers per browser session/tab:

   • Transactor client sends app-level ping every 10 seconds and expects pong!.
   • Server session manager can also send an app-level ping result for idle sessions.
   • HulyPulse sends text ping from the client while the Rust server also sends websocket protocol ping frames.

   This is expected keepalive traffic in small numbers, but it becomes noisy per tab and can stack across multiple sockets.

4. HulyPulse and presence reconnect on a fixed 1 second timer with no backoff or jitter. During a server restart or network incident, many browser tabs can reconnect in lockstep.

Expected behavior

• A missing pong should eventually close/reconnect a stale websocket even if the socket remains in OPEN state.
• Heartbeat ownership should be clear per socket, avoiding redundant client/server ping loops where one direction is enough.
• Reconnect loops should use bounded backoff and jitter to avoid reconnect stampedes.

Suggested direction

• Track an outstanding ping/deadline instead of clearing a long timeout every short interval.
• Close on timeout when the socket is still open but no pong was observed.
• Pick one heartbeat owner per socket where possible, or document why both directions are required.
• Add jitter/backoff to reconnect timers.
• Add focused tests for no-pong timeout and reconnect behavior.

Duplicate check

I searched existing issues before filing and did not find a matching report:

• repo:hcengineering/platform is:issue heartbeat websocket ping pong
• repo:hcengineering/platform is:issue HulyPulse ping timeout
• repo:hcengineering/platform is:issue "pong!"
• repo:hcengineering/platform is:issue "no response from server"
• repo:hcengineering/platform is:issue "no ping response from server"
• repo:hcengineering/platform is:issue websocket reconnect backoff
• repo:hcengineering/platform is:issue presence websocket ping

[huly-github-staging]

Connected to Huly®: UBERF-16576