fixes for gracefully handling allocation failures in edge cases

[JacobBarthelmeh]

These fixes are for the nightly mem test which injects allocation failures.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Improves resilience to injected allocation failures (nightly mem tests) by fixing ownership/cleanup paths in crypto and adjusting tests to avoid dereferencing NULL or freeing unowned objects.

Changes:

• Fix/adjust ownership and cleanup on failure paths (EVP PKEY decode, OCSP CERTID, handshake hashes, CRL entry duplication).
• Harden tests against allocation-failure setups by adding guards and safer initialization/ownership transfer patterns.
• Prevent leaks/crashes under allocation-failure injection by improving frees and conditional execution.

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
wolfcrypt/src/evp_pk.c	Tracks EVP_PKEY ownership to free only when locally allocated; frees key on BIO write failure.
src/internal.c	Allocates handshake-hash state before freeing old state to preserve retry safety on alloc failure.
src/ocsp.c	Avoids freeing caller-owned OCSP CERTID on status allocation failure.
src/crl.c	Prevents aliasing of owned pointers after memcpy to avoid freeing original entry on partial dup failure.
tests/api/test_tls13.c	Adds NULL guard in a test to avoid use after setup alloc failure.
tests/api/test_pkcs7.c	Adds NULL guard around building an encrypted key package when inner buffer alloc fails.
tests/api/test_ossl_x509_str.c	Adjusts store freeing behavior when CTX creation fails under allocation failure.
tests/api/test_ossl_x509_ext.c	Transfers ownership only when stack push succeeds.
tests/api/test_ossl_p7p12.c	Transfers ownership only when stack push succeeds.
tests/api/test_mldsa.c	Skips negative checks when earlier expectations failed to avoid acting on freed/uninit state.
tests/api/test_lms_xmss.c	Zero-inits structs for safe cleanup; adjusts init flags under failure.
tests/api/test_dtls13.c	Adds some NULL guards around buffer/field access when setup alloc fails.
tests/api/test_dtls.c	Adds NULL guard around DTLS nonblock + hash computation in allocation-failure scenarios.
tests/api.c	Avoids dereferencing NULL ctx for heap; gates ECH checks behind EXPECT_SUCCESS.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m3

• FLASH: .text +12 B (+0.0%, 121,405 B / 262,144 B, total: 46% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text +64 B (+0.0%, 767,604 B / 1,048,576 B, total: 73% used)

gcc-arm-cortex-m4-tls12

• FLASH: .text +64 B (+0.1%, 122,189 B / 262,144 B, total: 47% used)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-dtls13
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-pq
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls13
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• gcc-arm-cortex-m7-tls13
• linuxkm-pie
• linuxkm-standard
• stm32-sim-stm32h753

[JacobBarthelmeh]

Retest this please Jenkins. FIPS 140-2 test fail, CAVP_AES_CCM

[JacobBarthelmeh]

Retest this please Jenkins. FIPS 140-2 test, CAVP_HMAC_SHA384

[JacobBarthelmeh]

Most of these are duplicates of what was done here #10652. Closing and will open a new PR for any failing reports after 10652 was merged.