src,lib: implement experimental DTLS API

Decided to take a short break from the work on QUIC
to implement a DTLS API. Very experimental at this
point but the basic API is there (inspired by the
QUIC API work).

The implementation is based on OpenSSL's built-in
DTLS support and no other dependencies are required.

DTLS is a datagram-based version of TLS that is used
for things like WebRTC and CoAP. It provides similar
security guarantees as TLS but is designed to work over
UDP instead of TCP.

This shouldn't be considered ready for production
but it is a good starting point for experimentation
and feedback.

```bash
./configure --experimental-dtls
make -j{nproc}
./node --experimental-dtls my-dtls-app.js
```

Signed-off-by: James M Snell <jasnell@gmail.com>
Assisted-by: Opencode:Opus 4.6

Author: jasnell

configure.py

@@ -1065,6 +1065,12 @@
     default=None,
     help='build with experimental QUIC support')
 
+parser.add_argument('--experimental-dtls',
+    action='store_true',
+    dest='experimental_dtls',
+    default=None,
+    help='build with experimental DTLS support')
+
 parser.add_argument('--ninja',
     action='store_true',
     dest='use_ninja',
@@ -2350,6 +2356,10 @@ def configure_quic(o):
   o['variables']['node_use_quic'] = b(options.experimental_quic and
                                       not options.without_ssl)
 
+def configure_dtls(o):
+  o['variables']['node_use_dtls'] = b(options.experimental_dtls and
+                                      not options.without_ssl)
+
 def configure_static(o):
   if options.fully_static or options.partly_static:
     if flavor == 'mac':
@@ -2808,6 +2818,7 @@ def make_bin_override():
 configure_v8(output, configurations)
 configure_openssl(output)
 configure_quic(output)
+configure_dtls(output)
 configure_intl(output)
 configure_static(output)
 configure_inspector(output)

doc/api/dtls.md

@@ -0,0 +1,349 @@
+# DTLS
+
+<!-- YAML
+added: REPLACEME
+-->
+
+<!-- introduced_in=REPLACEME -->
+
+> Stability: 1 - Experimental
+
+<!-- source_link=lib/dtls.js -->
+
+The `node:dtls` module provides an implementation of the Datagram Transport
+Layer Security (DTLS) protocol over UDP. DTLS provides TLS-equivalent
+security guarantees for datagram-based communication, including
+confidentiality, integrity, and authentication.
+
+To use this module, it must be enabled at build time with the
+`--experimental-dtls` configure flag and at runtime with the
+`--experimental-dtls` CLI flag.
+
+```bash
+node --experimental-dtls app.mjs
+```
+
+```mjs
+import { listen, connect } from 'node:dtls';
+```
+
+```cjs
+const { listen, connect } = require('node:dtls');
+```
+
+## DTLS vs TLS
+
+DTLS is designed for UDP transport and differs from TLS in several key ways:
+
+* No stream guarantees: Messages may arrive out of order or be lost.
+  DTLS preserves datagram semantics.
+* One socket, many peers: A single UDP socket can serve multiple DTLS
+  sessions. The `DTLSEndpoint` manages this multiplexing.
+* Cookie exchange: DTLS servers use a stateless cookie mechanism
+  (HelloVerifyRequest) to prevent denial-of-service amplification attacks.
+* Retransmission: DTLS handles handshake retransmission internally since
+  UDP does not guarantee delivery.
+
+## `dtls.listen(callback, options)`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+* `callback` {Function} Called for each new DTLS session accepted by the
+  server.
+  * `session` {DTLSSession} The new session.
+* `options` {Object}
+  * `cert` {string|Buffer} Server certificate in PEM format. **Required.**
+  * `key` {string|Buffer} Server private key in PEM format. **Required.**
+  * `port` {number} Port to bind to. **Required.**
+  * `host` {string} Address to bind to. **Default:** `'0.0.0.0'`.
+  * `ca` {string|Buffer|string\[]|Buffer\[]} CA certificates in PEM format.
+  * `ciphers` {string} OpenSSL cipher list string.
+  * `alpn` {string\[]|Buffer} ALPN protocol names.
+  * `srtp` {string} Colon-separated SRTP protection profile names
+    (e.g., `'SRTP_AES128_CM_SHA1_80:SRTP_AEAD_AES_128_GCM'`).
+  * `requestCert` {boolean} Request client certificate. **Default:** `false`.
+  * `mtu` {number} Maximum transmission unit for DTLS records.
+    **Default:** `1200`.
+* Returns: {DTLSEndpoint}
+
+Creates a DTLS server bound to the specified address and port. The server
+uses automatic HMAC-based cookie exchange for DoS protection.
+
+```mjs
+import { listen } from 'node:dtls';
+import { readFileSync } from 'node:fs';
+
+const endpoint = listen((session) => {
+  session.onmessage = (data) => {
+    console.log('Received:', data.toString());
+    session.send('pong');
+  };
+
+  session.onhandshake = (protocol) => {
+    console.log('Handshake complete:', protocol);
+  };
+}, {
+  cert: readFileSync('server-cert.pem'),
+  key: readFileSync('server-key.pem'),
+  port: 4433,
+});
+
+console.log('DTLS server listening on', endpoint.address);
+```
+
+## `dtls.connect(host, port[, options])`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+* `host` {string} Remote host to connect to.
+* `port` {number} Remote port to connect to.
+* `options` {Object}
+  * `ca` {string|Buffer|string\[]|Buffer\[]} CA certificates in PEM format.
+  * `cert` {string|Buffer} Client certificate in PEM format.
+  * `key` {string|Buffer} Client private key in PEM format.
+  * `rejectUnauthorized` {boolean} Reject connections with unverifiable
+    certificates. **Default:** `true`.
+  * `bindHost` {string} Local bind address. **Default:** `'0.0.0.0'`.
+  * `bindPort` {number} Local bind port. **Default:** `0` (ephemeral).
+  * `alpn` {string\[]|Buffer} ALPN protocol names.
+  * `srtp` {string} SRTP protection profile names.
+  * `mtu` {number} Maximum transmission unit. **Default:** `1200`.
+* Returns: {DTLSSession}
+
+Connects to a DTLS server. Returns a `DTLSSession` whose `opened` property
+is a `Promise` that resolves when the handshake completes.
+
+```mjs
+import { connect } from 'node:dtls';
+import { readFileSync } from 'node:fs';
+
+const session = connect('localhost', 4433, {
+  ca: [readFileSync('ca-cert.pem')],
+});
+
+await session.opened;
+session.send('hello');
+
+session.onmessage = (data) => {
+  console.log('Received:', data.toString());
+};
+```
+
+## Class: `DTLSEndpoint`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+Manages a UDP socket and multiplexes DTLS sessions.
+
+### `endpoint.address`
+
+* Returns: {Object} `{ address, family, port }`
+
+The local address the endpoint is bound to.
+
+### `endpoint.state`
+
+* Returns: {DTLSEndpointState}
+
+Shared state object with properties:
+
+* `bound` {boolean}
+* `listening` {boolean}
+* `closing` {boolean}
+* `destroyed` {boolean}
+* `sessionCount` {number}
+* `busy` {boolean}
+
+### `endpoint.busy`
+
+* {boolean}
+
+When `true`, the endpoint rejects new incoming connections. Can be set
+to implement backpressure.
+
+### `endpoint.close()`
+
+* Returns: {Promise} Resolves when the endpoint is fully closed.
+
+Gracefully closes the endpoint. All active sessions are closed with
+`close_notify` alerts before the UDP socket is released.
+
+### `endpoint.destroy([error])`
+
+Immediately destroys the endpoint without sending `close_notify` alerts.
+
+### `endpoint.closed`
+
+* {Promise} Resolves when the endpoint has fully closed.
+
+### `endpoint[Symbol.asyncDispose]()`
+
+Equivalent to calling `endpoint.close()`.
+
+## Class: `DTLSSession`
+
+<!-- YAML
+added: REPLACEME
+-->
+
+Represents a DTLS association with a single remote peer.
+
+### `session.send(data)`
+
+* `data` {string|Buffer} The data to send.
+* Returns: {number} The number of bytes written to the DTLS layer.
+
+Send application data to the peer. The data is encrypted by DTLS before
+being sent over UDP. Can only be called after the handshake completes
+(`session.opened` has resolved).
+
+### `session.close()`
+
+* Returns: {Promise} Resolves when the session is closed.
+
+Initiates a graceful DTLS shutdown by sending a `close_notify` alert.
+
+### `session.destroy([error])`
+
+Immediately destroys the session without sending `close_notify`.
+
+### `session.opened`
+
+* {Promise} Resolves with `{ protocol }` when the DTLS handshake completes.
+
+### `session.closed`
+
+* {Promise} Resolves when the session is fully closed.
+
+### `session.remoteAddress`
+
+* Returns: {Object} `{ address, family, port }`
+
+### `session.protocol`
+
+* Returns: {string} The negotiated DTLS protocol version
+  (e.g., `'DTLSv1.2'`).
+
+### `session.cipher`
+
+* Returns: {Object} `{ name, standardName, version }`
+
+### `session.peerCertificate`
+
+* Returns: {string|undefined} The peer's certificate in PEM format.
+
+### `session.alpnProtocol`
+
+* Returns: {string|undefined} The negotiated ALPN protocol.
+
+### `session.srtpProfile`
+
+* Returns: {string|undefined} The negotiated SRTP protection profile name.
+
+### `session.exportKeyingMaterial(length, label[, context])`
+
+* `length` {number} Number of bytes to export.
+* `label` {string} The label for the exported keying material.
+* `context` {Buffer} Optional context value.
+* Returns: {Buffer}
+
+Exports keying material from the DTLS session, as defined in
+[RFC 5705][]. This is commonly used with DTLS-SRTP to derive
+encryption keys for media streams.
+
+### Callback properties
+
+#### `session.onmessage`
+
+* {Function}
+  * `data` {Buffer}
+
+Set to receive application data from the peer.
+
+#### `session.onerror`
+
+* {Function}
+  * `error` {Error}
+
+Set to receive error notifications.
+
+#### `session.onhandshake`
+
+* {Function}
+  * `protocol` {string}
+
+Set to receive handshake completion notifications.
+
+#### `session.onkeylog`
+
+* {Function}
+  * `line` {string}
+
+Set to receive TLS key log lines (for debugging with Wireshark).
+
+### `session[Symbol.asyncDispose]()`
+
+Equivalent to calling `session.close()`.
+
+## DTLS-SRTP example
+
+DTLS-SRTP is used by WebRTC for media encryption. The DTLS handshake
+negotiates the SRTP protection profile and provides keying material.
+
+```mjs
+import { listen, connect } from 'node:dtls';
+import { readFileSync } from 'node:fs';
+
+// Server with SRTP
+const server = listen((session) => {
+  session.onhandshake = () => {
+    console.log('SRTP profile:', session.srtpProfile);
+    const keys = session.exportKeyingMaterial(
+      60,
+      'EXTRACTOR-dtls_srtp',
+    );
+    console.log('SRTP keying material:', keys);
+  };
+}, {
+  cert: readFileSync('server-cert.pem'),
+  key: readFileSync('server-key.pem'),
+  port: 5004,
+  srtp: 'SRTP_AES128_CM_SHA1_80:SRTP_AEAD_AES_128_GCM',
+});
+
+// Client with SRTP
+const session = connect('localhost', 5004, {
+  rejectUnauthorized: false,
+  srtp: 'SRTP_AEAD_AES_128_GCM:SRTP_AES128_CM_SHA1_80',
+});
+
+await session.opened;
+console.log('Negotiated SRTP:', session.srtpProfile);
+const keys = session.exportKeyingMaterial(60, 'EXTRACTOR-dtls_srtp');
+```
+
+## MTU considerations
+
+Since libuv does not currently support path MTU discovery, the DTLS module
+uses a conservative default MTU of 1200 bytes. This value works across
+virtually all network paths but may be suboptimal for local networks.
+
+The MTU can be configured via the `mtu` option:
+
+```mjs
+// For a local network where you know the path MTU
+const endpoint = listen(callback, {
+  // ...
+  mtu: 1400,
+});
+```
+
+The minimum allowed MTU is 256 bytes. The maximum is 65535.
+
+[RFC 5705]: https://www.rfc-editor.org/rfc/rfc5705