Fix TLS1.2 error code correction

[embhorn]

Description

In DoTls13ServerHello, the no-extensions / no-downgrade branch now returns VERSION_ERROR instead of BUFFER_ERROR. The caller's existing TranslateErrorToAlert() maps that to protocol_version.

Fixes #10761

Testing

Added tests
test_tls13_no_ext_sh_alert
test_tls13_trunc_ext_sh_alert

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[Copilot]

Pull request overview

Updates TLS 1.3 client ServerHello parsing so that a syntactically valid TLS 1.2 ServerHello lacking extensions (i.e., no supported_versions) is treated as a version negotiation failure (mapping to protocol_version) rather than a decode error, aligning behavior with RFC 8446 §6.2 and the linked issue (#10761).

Changes:

• In DoTls13ServerHello, return VERSION_ERROR (instead of BUFFER_ERROR) when the ServerHello ends immediately after the compression method (no extensions field at all) and downgrade is disabled.
• Preserve BUFFER_ERROR behavior for genuinely malformed messages where the extensions length field is truncated (partial).
• Add API tests that assert the resulting fatal alert is protocol_version for the no-extensions case and decode_error for the truncated-length case.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
tests/api.c	Adds manual-memio tests to validate alert mapping for “no extensions” vs “truncated extensions length” ServerHello inputs.
src/tls13.c	Refines the no-extensions/no-downgrade branch to distinguish “absent extensions field” (version mismatch) from “truncated length” (decode error).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10764

Scan targets checked: wolfssl-bugs, wolfssl-src

No new issues found in the changed files. ✅