libvncserver: validate VNC auth response length

[marcofortina]

Summary

Reject invalid VNC auth response lengths before comparing them against the fixed-size authentication challenge buffer.

rfbCheckPasswordByList() builds a temporary encrypted challenge buffer of CHALLENGESIZE bytes, but compared it with the caller-provided length. Passing a length larger than CHALLENGESIZE could therefore make memcmp() read past the stack buffer.

The default password checker had the same implicit fixed-size response assumption, so this patch applies the same validation there as well.

Changes

• Reject NULL authentication responses.
• Reject VNC authentication responses whose length is not exactly CHALLENGESIZE.
• Apply the validation to both built-in password checkers.
• Add a regression test for the overlong rfbCheckPasswordByList() response path.

Validation

Tested with a minimal local CMake configuration:

cmake -S . -B build-680-patch \
  -DWITH_EXAMPLES=OFF \
  -DWITH_TESTS=ON \
  -DWITH_OPENSSL=OFF \
  -DWITH_GNUTLS=OFF \
  -DWITH_GCRYPT=OFF \
  -DWITH_SDL=OFF \
  -DWITH_GTK=OFF \
  -DWITH_QT=OFF \
  -DWITH_FFMPEG=OFF \
  -DWITH_XCB=OFF \
  -DWITH_LIBSSHTUNNEL=OFF \
  -DWITH_SYSTEMD=OFF \
  -DCMAKE_BUILD_TYPE=Debug
cmake --build build-680-patch --parallel 1
ctest --test-dir build-680-patch --output-on-failure

Result:

100% tests passed, 0 tests failed out of 6

Also ran the new regression test with AddressSanitizer enabled:

cmake -S . -B build-680-asan \
  -DWITH_EXAMPLES=OFF \
  -DWITH_TESTS=ON \
  -DWITH_OPENSSL=OFF \
  -DWITH_GNUTLS=OFF \
  -DWITH_GCRYPT=OFF \
  -DWITH_SDL=OFF \
  -DWITH_GTK=OFF \
  -DWITH_QT=OFF \
  -DWITH_FFMPEG=OFF \
  -DWITH_XCB=OFF \
  -DWITH_LIBSSHTUNNEL=OFF \
  -DWITH_SYSTEMD=OFF \
  -DCMAKE_BUILD_TYPE=Debug \
  -DCMAKE_C_FLAGS='-fsanitize=address -fno-omit-frame-pointer' \
  -DCMAKE_EXE_LINKER_FLAGS='-fsanitize=address'
cmake --build build-680-asan --target test_password_list_test --parallel 1
ASAN_OPTIONS=detect_leaks=0 ./build-680-asan/test/password_list_test

Closes #680.