libvncclient: avoid Tight JPEG for deep pixel formats

[marcofortina]

Summary

Avoid requesting Tight JPEG when the client pixel format cannot safely consume 8-bit-per-channel JPEG samples.

Tight JPEG is an 8-bit RGB path. When the client format has more than 8 bits per RGB component, for example a 30-bit/10-bit-per-channel HDR format, requesting the Tight quality pseudo-encoding can make servers choose JPEG subencoding even though the negotiated client format cannot represent it losslessly and some servers cannot encode that source format as JPEG correctly.

Changes

• Add a small helper that decides whether Tight JPEG should be requested for the current client pixel format.
• Keep requesting Tight JPEG for true-colour formats with more than 8 bpp and RGB component maxima up to 8 bits.
• Do not request Tight JPEG for 10-bit-per-channel/deep pixel formats.
• Reuse the helper for both explicit tight encoding requests and the default encoding list.

Validation

Tested with a minimal local CMake configuration:

cmake -S . -B build-695-patch \
  -DWITH_EXAMPLES=OFF \
  -DWITH_TESTS=ON \
  -DWITH_OPENSSL=OFF \
  -DWITH_GNUTLS=OFF \
  -DWITH_GCRYPT=OFF \
  -DWITH_SDL=OFF \
  -DWITH_GTK=OFF \
  -DWITH_QT=OFF \
  -DWITH_FFMPEG=OFF \
  -DWITH_XCB=OFF \
  -DWITH_LIBSSHTUNNEL=OFF \
  -DWITH_SYSTEMD=OFF \
  -DCMAKE_BUILD_TYPE=Debug
cmake --build build-695-patch --parallel 1
ctest --test-dir build-695-patch --output-on-failure

Result:

100% tests passed, 0 tests failed out of 5

Notes

I could not run the original WayVNC HDR setup locally, so this should still be runtime-verified against the reported 10-bit framebuffer case.

Closes #695.