Skip to content

[✨ Triage] dotnet/runtime#127053 by rzikm - Support certificate validation TLS Alerts for OSX #199

@MihuBot

Description

@MihuBot

Triage for dotnet/runtime#127053.
Repo filter: All networking issues.
MihuBot version: 5a5ee7.
Ping MihaZupan for any issues.

This is a test triage report generated by AI, aimed at helping the triage team quickly identify past issues/PRs that may be related.
Take any conclusions with a large grain of salt.

Tool logs
dotnet/runtime#127053: Support certificate validation TLS Alerts for OSX by rzikm
Extracted 4 search queries: TLS alert propagation during certificate validation on macOS SslStreamPal.OSX ApplyAlertToken SSLSetError, VerifyRemoteCertificate returning TLS alert on macOS (SSLSetError) in SslStreamPal, certificate validation TLS Alerts macOS System.Net.Security SslStreamPal.OSX, ApplyAlertToken plumbing of TLS Alert using SSLSetError in .NET SslStreamPal.OSX
Found 23 candidate issues
  • PR #115996 (May 2025) - "Support certificate validation TLS alerts for Linux"
    Summary: Implements the core approach you propose but for OpenSSL/Linux: certificate validation is moved into the OpenSSL handshake via SSL_CTX_set_cert_verify_cb so OpenSSL can send the appropriate TLS alert (bad_certificate, unknown_ca, certificate_expired, certificate_required) when validation fails. The managed VerifyRemoteCertificate path is invoked from the native callback; failures are mapped to X509_V_ERR_* so OpenSSL emits the corresponding alert. Notes/caveats from the discussion that will be relevant for OSX work:

    • Resumed sessions: OpenSSL may skip the verify callback on resumed sessions; the PR preserves the previous behavior (post‑handshake revalidation) for resumed sessions, so no alert is sent on resume failures.
    • Client-certificate-required + user callback: if a server requires a client cert but a user-provided RemoteCertificateValidationCallback is present, OpenSSL won't be able to send a TLS alert in one of the code paths (the PR documents that case).
    • Connection info availability: the PR populates connection info before invoking user callbacks (pending-cipher fallback) so user callbacks get meaningful info during the mid-handshake validation.
    • Implementation review produced practical feedback (ensure distinct certificate instances for leaf vs intermediates, clear stored exception state per-handshake attempt, robust handling of weak GC handles, guard ALPN callback against nulls). These implementation details and pitfalls are directly applicable when doing a macOS native integration that invokes managed validation mid-handshake.
  • Issue #18837 (Oct 2016) - "Supports TLS alerts for cert validation for non-Windows platforms"
    Summary: Long-running tracking issue for sending TLS alerts on non-Windows platforms. Early conclusions: to get real TLS alerts you often need to run cert validation during the handshake (register a validation callback with the TLS library) rather than validating post-handshake. The discussion explored OpenSSL options (SSL_want_retry_verify, async verify) and pointed out limitations (server-side behavior, thread affinity for async OpenSSL modes). The key takeaway: the general approach is to validate inside the handshake callback when the TLS stack supports it; otherwise you cannot reliably cause the TLS library to emit an alert.

  • Issue #116305 (June 2025) - "SslStream indiscriminately uses ProtocolVersion TLS Alert for handshake failures"
    Summary: Calls out that SslStream previously sent a generic ProtocolVersion alert for many handshake failures. The recommendation was to map alerts more precisely to the underlying failure where possible rather than always using ProtocolVersion. Relevance to macOS: when you start plumbing alerts from macOS APIs you should map macOS/OSStatus/callback failures to the most appropriate TLS alert (the Linux PR is an example of mapping failure reasons to X509 error codes). There is an explicit note that some platform PALs (Windows) will do better mapping vs a generic placeholder; macOS work should aim to provide more specific alerts where available.

  • PR #117611 (July 2025) - "Unify certificate validation code on OSX between SslStream and QuicConnection"
    Summary: Unifies macOS certificate validation to call the shared managed BuildChainAndVerifyProperties logic (removes legacy macOS hostname-matching code). This refactor centralizes validation logic on macOS in managed code, which simplifies making platform-wide changes to validation behavior. Relevance: having a single managed validation entry point on macOS makes it easier to (a) call VerifyRemoteCertificate from a native callback and (b) ensure the same validation + error mapping is used for SslStream and QUIC. It does not itself implement TLS-alert plumbing, but it reduces duplication and is a good prerequisite.

  • PR #1513 (Jan 2020) - "improve cert validation diagnostic on OSX"
    Summary: Adds additional tracing to expose the OSStatus returned by the macOS SecTrust/validation APIs so failures are more diagnosable (e.g., exposing CSSM/errSec codes). This improved diagnostics work helps determine why the OS rejected a certificate, but it does not cause the OS to send a TLS alert. Important observation from later discussion: sometimes SecTrust/AppleCrypto returns 0 or otherwise doesn't provide a useful status, so diagnostics are sometimes limited.

  • Issue #89859 (Aug 2023 → closed May 2025) - "RemoteCertificateError TraceEvent Does Not Include the Underline TLS Error from macOS"
    Summary: User reported that the macOS trace event did not include a helpful OSStatus code; investigation showed the code attempts to surface SecTrust status but can get 0 when details are unavailable or when AppleCrypto reported 0. The issue was closed as non-actionable without a reproducible repro. Relevance: even when you extract macOS validation details, the platform may not always provide the exact underlying error; plan for "no detail available" paths.

  • Issue #121272 (Nov 2025 → closed Feb 2026) - "Improper handling of errSSLClosedGraceful returned from SSLWrite on macOS"
    Summary: Discusses Secure Transport (the legacy macOS TLS API) behavior differences and error mapping problems (errSSLClosedGraceful translated into misleading errno-based Win32Exception text). Notes mention Network.framework being a newer implementation path. Relevance: Secure Transport and macOS TLS semantics differ from OpenSSL/Schannel in important ways; when plumbing TLS alert behavior on macOS, be conscious of which native stack (Secure Transport vs Network.framework) you're targeting and of how macOS maps internal errors to status codes/strings.

Additional practical takeaways for the new OSX work (from the collected discussions)

  • The Linux/OpenSSL PR is a direct precedent: validation during handshake via native callback allows the TLS stack to emit proper alerts. Aim to reproduce the same idea on macOS if macOS TLS APIs let you run validation mid-handshake and specify an alert code. The new issue specifically mentions SSLSetError in ApplyAlertToken — investigate whether Secure Transport / Network.framework provide a supported way to set/send a specific TLS alert during handshake (and whether doing so is compatible with managed validation callbacks).
  • If macOS forces synchronous validation inside a native callback (as OpenSSL did), you need to call managed VerifyRemoteCertificate synchronously; that has the same caveats discussed earlier (it will block the native handshake thread and will not support arbitrary async work from user callbacks). The project discussions leaned toward performing validation synchronously inside the callback to get TLS alerts, accepting that async user callbacks would be problematic.
  • Even when macOS exposes an OSStatus to map to an alert, it may sometimes return 0 or otherwise withhold detail (see #89859 / #1513). Plan robust fallbacks: (a) send a reasonable generic alert when precise mapping isn't possible, (b) ensure good diagnostic logging/tracing of whatever macOS status is available.
  • There are several implementation pitfalls called out in the Linux PR reviews (certificate ownership / disposal, clearing stored exceptions per handshake, guarding weak handles) that are relevant for any native ↔ managed handshake-time callback integration. Make sure to handle lifetime and GCHandle/weak-handle safety and to clear per-handshake state.

If you want, I can:

  • point to the exact locations in the codebase that the Linux PR changed so you can mirror the same pattern on macOS (native shim + managed interop + validation mapping), or
  • collect the macOS Secure Transport / Network.framework APIs (SSLSetError, SSLSetPeerDomainName, SSLSetCertificate, SecTrustEvaluate variants) and summarize whether they support mid-handshake cert-verify callbacks and alert injection and what their limitations are.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions