Add error when RPK is used with DANE stub

[embhorn]

Description

DANE support was stubbed for the compatibility layer. Ensure an actual connection errors if it is used.

Thanks to Peter Samarin for reporting the issue and reviewing the fix.

Fixes zd21825

Testing

Added test cases in test_tls13_rpk_handshake

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10486

Scan targets checked: wolfssl-bugs, wolfssl-src

No new issues found in the changed files. ✅

[Copilot]

Pull request overview

This PR adjusts wolfSSL’s OpenSSL-compat verification reporting for TLS Raw Public Keys (RPK, RFC 7250) so that connections using RPK without an out-of-band trust mechanism do not report WOLFSSL_X509_V_OK via wolfSSL_get_verify_result().

Changes:

• Introduces a new verification result code for “RPK not trusted” and exposes it in both wolfSSL and OpenSSL-compat headers.
• Updates peer-certificate processing to set peerVerifyRet to the new RPK-untrusted code when appropriate.
• Extends test_tls13_rpk_handshake to assert the expected verify-result behavior for RPK vs X.509 handshakes.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
wolfssl/ssl.h	Adds a new wolfSSL verification-result enum value for untrusted RPK.
wolfssl/openssl/x509.h	Exposes the corresponding OpenSSL-compat X509_V_ERR_RPK_UNTRUSTED define.
src/internal.c	Sets peerVerifyRet for RPK handshakes and adds an error-string mapping for the new code.
tests/api/test_tls13.c	Adds assertions that RPK handshakes produce a non-OK verify result while X.509 handshakes remain OK.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m3

• FLASH: .text +8 B (+0.0%, 122,449 B / 262,144 B, total: 47% used)

gcc-arm-cortex-m4

• FLASH: .rodata.wolfSSL_ERR_reason_error_string.str1.1 +36 B (+0.0%, 200,256 B / 262,144 B, total: 76% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .rodata +128 B, .text +64 B (+0.0%, 770,508 B / 1,048,576 B, total: 73% used)

gcc-arm-cortex-m4-pq

• FLASH: .rodata +32 B (+0.0%, 279,268 B / 1,048,576 B, total: 27% used)

gcc-arm-cortex-m4-rsa-only

• FLASH: .rodata +32 B (+0.0%, 324,864 B / 1,048,576 B, total: 31% used)

gcc-arm-cortex-m4-tls13

• FLASH: .rodata.wolfSSL_ERR_reason_error_string.str1.1 +36 B (+0.0%, 235,954 B / 262,144 B, total: 90% used)

gcc-arm-cortex-m7

• FLASH: .rodata.wolfSSL_ERR_reason_error_string.str1.1 +36 B, .text +64 B (+0.0%, 200,256 B / 262,144 B, total: 76% used)

gcc-arm-cortex-m7-pq

• FLASH: .rodata +32 B (+0.0%, 279,844 B / 1,048,576 B, total: 27% used)

gcc-arm-cortex-m7-tls13

• FLASH: .rodata.wolfSSL_ERR_reason_error_string.str1.1 +36 B (+0.0%, 236,018 B / 262,144 B, total: 90% used)

linuxkm-standard

• Data: __patchable_function_entries +32 B (+0.1%, 48,128 B)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• linuxkm-pie
• stm32-sim-stm32h753

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10486

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

No new issues found in the changed files. ✅

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (2)

src/internal.c:1

• Correct the typo in the comment: “skip following x509 version check” should be “skip following X.509 version check” (or at least “skip following x509 version check” with proper spacing was already addressed, but please also standardize “X.509” capitalization to match surrounding terminology).
  src/internal.c:1
• The handshake failure path uses ASN_NO_SIGNER_E (an X.509-oriented error) as the transport-level error for an untrusted RPK, while also introducing a distinct verify-result code (WOLFSSL_X509_V_ERR_RPK_UNTRUSTED) under OPENSSL_EXTRA*. This can be confusing for API consumers diagnosing failures (they may see ASN_NO_SIGNER_E rather than an RPK-specific failure unless they also inspect verify_result). Recommendation: consider propagating an RPK-specific error code at the handshake error layer as well (or otherwise ensure the mapping from ASN_NO_SIGNER_E to the new RPK_UNTRUSTED reason is discoverable), to make diagnostics consistent across builds (with/without OPENSSL_EXTRA).

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10486

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

No new issues found in the changed files. ✅

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #10486

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

No new issues found in the changed files. ✅

[dgarske]

Skoll Code Review

Scan type: reviewOverall recommendation: COMMENT
Findings: 7 total — 7 posted, 0 skipped
6 finding(s) posted as inline comments (see file-level comments below)

Posted findings

• [Medium] Nested '*/' inside doxygen \code block truncates the comment — doc/dox_comments/header_files/ssl.h:16038,16074
• [Medium] RPK now fails closed for all HAVE_RPK builds — notable behavior change — src/internal.c:16884-16911
• [Low] Append-only pin table has no clear/replace API — src/ssl_api_cert.c:446-507
• [Low] CTX-level pinning API NULL/arg-error path not directly tested — src/ssl_api_cert.c:475-485
• [Low] RPK reason-string regression check omitted for OPENSSL_EXTRA_X509_SMALL-only builds — tests/api.c:28062-28075
• [Low] DANE removal leaves useDANE field with no remaining reader/writer — wolfssl/internal.h:5301-5303
• [Low] x509.h comment may understate OpenSSL compatibility of value 95 — wolfssl/openssl/x509.h:213-215

Review generated by Skoll

[dgarske]

Please review these skoll reports -> #10486 (review)

[dgarske]

Skoll Code Review

Scan type: reviewOverall recommendation: COMMENT
Findings: 1 total — 1 posted, 0 skipped

Posted findings

• [Low] useDANE bitfield is now dead after removing its only reader — src/internal.c:16933-16940 (field at wolfssl/internal.h:5301-5303)

Review generated by Skoll

[embhorn]

retest this please