clamp BIO_read length in encrypt/decrypt_rsa_message

[dxbjavid]

• Version of iperf3 (or development branch, such as master or
  3.1-STABLE) to which this pull request applies: master

• Issues fixed (if any):

• Brief description of code changes (suitable for use as a commit message):

both encrypt_rsa_message and decrypt_rsa_message warn that input longer than the rsa key size will be truncated, but they then hand the full length to BIO_read while rsa_buffer is only output_buffer_len bytes, so an over-long input writes past the allocation. on a server started with authentication the client-supplied authtoken is base64 decoded and passed straight into decrypt_rsa_message, so a long token overflows that heap buffer (asan reports a write of the full decoded length into the RSA_size buffer). this caps the read to output_buffer_len in both helpers, which is the truncation the existing warning already describes.

[bmah888]

Thanks for the PR! Sounds pretty reasonable, we'll take a look.

[dxbjavid]

no rush, just leaving the trigger details here in case they're useful when you get to it.

on a server built with authentication, the client authtoken is base64 decoded and the decoded bytes go straight into decrypt_rsa_message as encryptedtext/encryptedtext_len, both fully client controlled. because rsa_buffer is only OPENSSL_malloc(output_buffer_len) (RSA_size, 256 bytes for the 2048-bit key in src/private.pem) but BIO_read is handed the full decoded length, a token longer than the key size writes past the allocation before any authentication happens.

i reproduced it against src/private.pem under asan: build a 4096-byte blob, base64 encode it as the authtoken and feed it to decode_auth_setting (same entry the server uses). before the patch it prints the existing 'truncated to 256' warning and then asan reports a heap write of 4096 into the 256-byte region inside BIO_read at iperf_auth.c:349. after the patch the same warning prints, the read is capped to output_buffer_len, decode returns -1 and it exits clean with no asan error. a full make plus src/t_auth still pass.

happy to fold that into a t_auth case if you'd like a regression test alongside the fix.