libvncserver: add WebSockets handshake mode

[marcofortina]

Summary

Improve WebSockets/RFB handshake handling so slow normal RFB clients are no
longer forced through a fragile timing-only auto-detection path, while still
giving embedders an explicit way to select the expected transport.

Issue #438 reports that WebSockets auto-detection is fragile for slow normal
RFB clients. The server used a short fixed wait for a client-first WebSockets
header before falling back to normal RFB, which made mixed WebSockets/RFB
listeners sensitive to timing.

Changes

• Add rfbWebSocketsHandshakeMode to rfbScreenInfo:
  • rfbWebSocketsHandshakeAuto: default auto-detection behaviour.
  • rfbWebSocketsHandshakeRfb: skip WebSockets probing and proceed as normal RFB immediately.
  • rfbWebSocketsHandshakeWebSockets: expect a WebSockets client and wait up to the configured client wait timeout.
• Add command-line parsing for -websocketmode auto|rfb|ws.
• Keep the default mode as auto for compatibility with existing deployments.
• Document -websocketmode auto|rfb|ws in the WebSockets README section.
• Make the auto path tolerate slow normal RFB clients instead of treating the short WebSockets probe as the only decision point.
• Add regression coverage for:
  • explicit rfb mode bypassing WebSockets socket peeking;
  • slow normal RFB clients in auto mode;
  • WebSockets-only mode still requiring a WebSockets handshake.

Validation

cmake -S . -B build-438-patch \
  -DWITH_EXAMPLES=OFF \
  -DWITH_TESTS=ON \
  -DWITH_OPENSSL=OFF \
  -DWITH_GNUTLS=OFF \
  -DWITH_GCRYPT=OFF \
  -DWITH_SDL=OFF \
  -DWITH_GTK=OFF \
  -DWITH_QT=OFF \
  -DWITH_FFMPEG=OFF \
  -DWITH_XCB=OFF \
  -DWITH_LIBSSHTUNNEL=OFF \
  -DWITH_SYSTEMD=OFF \
  -DCMAKE_BUILD_TYPE=Debug

cmake --build build-438-patch --parallel 1
ctest --test-dir build-438-patch --output-on-failure

Result:

100% tests passed, 0 tests failed

Notes

This keeps the existing default auto mode, but makes it robust for the slow
normal-RFB-client case reported in #438. Deployments that know the expected
transport can also opt into -websocketmode rfb or -websocketmode ws.

The PR does not add any non-standard RFB wire messages, encodings, security
types, or pseudo-encodings. It only changes how the listener decides whether the
connection should enter the WebSockets handshake path or the normal RFB path.

Closes #438.

[bk138]

@marcofortina please lower the volume of incoming PRs, my AI can't keep up.

[marcofortina]

@marcofortina please lower the volume of incoming PRs, my AI can't keep up.

your AI ?