fix: use EVP_DigestFinal_ex in rsa_signctx for OpenSSL 3.x compatibility

[mariosergiosl]

In OpenSSL 3.x, EVP_MD_CTX_set_update_fn() is deprecated and silently ignored, so digest_update() is never called and digest_finish() returns SHA-256 of empty string. Replace digest_finish() with EVP_DigestFinal_ex() for unrestricted keys to pass the correct hash to Esys_Sign().

Fixes EAP-TLS authentication failure with wpa_supplicant and NetworkManager. Verified on openSUSE Leap 15.6 (OpenSSL 3.1.4) and SLES 16 (OpenSSL 3.5.0).

PR_DESCRIPTION.md

[kalvdans]

Where in the documentation does OpenSSL say that the function is silently ignored? Deprecation is one thing, removing functionality is another thing.

[mariosergiosl]

Good point — and fair to ask for documentation.

You're right that deprecated doesn't mean removed, and the function call
itself still works. The issue is behavioral: in OpenSSL 3.x, when a digest
is initialized via the provider model (the default since 3.0), the custom
update function registered with EVP_MD_CTX_set_update_fn() is not called
by EVP_DigestUpdate(). The OpenSSL team described exactly this when they
deprecated it — see openssl/openssl#14003:

"These functions do not fit with the provider model where the update
function is not used. It is called only from the legacy code."

We confirmed this empirically. With debug logging enabled, digest_update()
never appears in the call sequence during EVP_DigestSignFinal(). The engine
goes directly from key load to rsa_signctx, skipping the update entirely:

ENGINE: engine initialized
TLS: Using private key from engine
rsa_signctx ... sig_data 0x... ← no digest_update before this

As a result, Esys_SequenceComplete receives no data and returns SHA-256 of
an empty string, which is the hash the TPM then signs:

e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Signature verification then fails with:
error:02000068:rsa routines:ossl_rsa_verify:bad signature

This was reproduced on OpenSSL 3.1.4 and 3.5.0. OpenSSL 1.x is not
affected because it does not use the provider path.

Also worth noting: Perry Werneck (PerryWerneck/tpmtest https://github.com/PerryWerneck/tpmtest), who originally
scoped this problem for SUSE engineering, observed the same engine
initialization failure in wpa_supplicant in issue #270 — further
confirming the behavior in real EAP-TLS scenarios.

Full debug logs and lab documentation are available at
https://github.com/mariosergiosl/8021x-eap-tls-ibmtss if useful.

Happy to provide the full debug log or any additional detail if helpful.