Remove openssl

[mamcx]

Description of Changes

Using the openssl crate significantly increases the build times on Windows. Switching to rustls improves it:

# Windows 11
# 1.79.0-x86_64-pc-windows-msvc 
# AMD Ryzen 9 7940HS w/ Radeon 780M Graphics        4.00 GHz
#  32.0 GB (25.8 GB usable)

   Compiling spacetimedb-core v0.12.0 (C:\Proyectos\space\SpacetimeDB\crates\core)
   Compiling spacetimedb-client-api v0.12.0 (C:\Proyectos\space\SpacetimeDB\crates\client-api)
   Compiling spacetimedb-standalone v0.12.0 (C:\Proyectos\space\SpacetimeDB\crates\standalone)   

OpenSSL:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 4m 01s
    Finished `release` profile [optimized] target(s) in 4m 39s    

RusTls:

    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 21s
    Finished `release` profile [optimized] target(s) in 2m 02s    

Expected complexity level and risk

*How complicated do you think these changes are? Grade on a scale from 1 to 5,

3

It looks correct to me the generation and reading of keys. However, even if trivial, changing libraries that do crypto requires careful eyes.

Testing

• * Check using https://8gwifi.org/PemParserFunctions.jsp that we use the same algorithms and settings as from openssl.*
• Publish quickstart and re-read the keys
• Manual inspection of the keys and config files

[gefjon]

I strongly request that we not do potentially dangerous changes which are not intended to be API breaks within the 0.12 release window. This can, and therefore should, wait.

[mamcx]

Changed to draft so we don't merge yet

[mamcx]

Btw, this adds cmake as a requirement on Windows.

[RReverser]

Btw, this adds [cmake](https://aws.github.io/aws-lc-rs/requirements/windows0 as a requirement on Windows.

Sigh. It's somewhat unfortunate to replace one external dependency for Windows devs (Perl or prebuilt OpenSSL) with another (CMake).

[mamcx]

Is now good time to change this from Draft?

[dare3path]

closes #1504

[CLAassistant]

All committers have signed the CLA.

[bfops]

openssl is a bunch of my build time on Ubuntu as well, so I was curious how this PR affects total build time (after rm -rf target/*).

master:

real	6m20.941s
user	7m28.604s
sys		1m14.316s

This branch:

real	4m48.625s
user	6m54.506s
sys		1m0.312s

So a meaningful improvement for me!

I don't really love that this swaps one outside-of-cargo dependency for another on Windows, but I can't speak to which one is more imposing for Windows users.

[clockwork-labs-bot]

Review: Remove openssl (#1700)

Summary: Replaces the openssl crate (with vendored openssl-src) with rcgen + p256 for key generation, and switches tokio-tungstenite/reqwest from native-tls to rustls. The motivation is clear: vendored OpenSSL is slow to compile, especially on Windows.

PEM Format Compatibility ✅

The critical question: do the new PEM outputs match what jsonwebtoken::DecodingKey::from_ec_pem() / EncodingKey::from_ec_pem() expect?

• Public key: Both old (openssl::EcKey::public_key_to_pem() via PEM_write_bio_EC_PUBKEY) and new (rcgen::KeyPair::public_key_pem()) produce -----BEGIN PUBLIC KEY----- (SPKI format). ✅
• Private key: Both old (PKey::private_key_to_pem_pkcs8()) and new (rcgen::KeyPair::serialize_pem()) produce -----BEGIN PRIVATE KEY----- (PKCS#8 format). ✅
• Backward compatibility: Existing key files on disk are read by the unchanged JwtKeys::new() path, which passes PEM bytes directly to jsonwebtoken. Since the PEM tags are identical, existing keys will continue to work. ✅
• Curve: Both use P-256 (Nid::X9_62_PRIME256V1 == PKCS_ECDSA_P256_SHA256). ✅

Swapped comments (cosmetic)

// Get the private key in PKCS#8 PEM format & write it.  // <-- says "private"
let public_key_bytes = key_pair.public_key_pem().into_bytes();  // <-- assigns public

// Get the public key in PEM format & write it.  // <-- says "public"
let private_key_bytes = key_pair.serialize_pem().into_bytes();  // <-- assigns private

The assignment is correct but the comments are swapped from the original and now misleading. Please swap them.

TLS backend change

• tokio-tungstenite: native-tls → rustls ✅
• reqwest: Added rustls-tls feature, but native-tls is still a transitive dependency (via reqwest's default features). Consider whether you want to disable default features and use only rustls-tls to fully remove the native-tls dependency.

Test changes ✅

The to_jwk_json helper in token_validation.rs tests was updated from openssl to p256 for extracting EC point coordinates. The logic is equivalent and cleaner.

Missing test coverage

There are no tests for EcKeyPair::generate() or the roundtrip of generating keys → creating JwtKeys → signing a JWT → verifying the JWT. The existing tests in token_validation.rs exercise JwtKeys::generate() (which calls EcKeyPair::generate()) indirectly, but a direct unit test would be valuable. Suggested tests:

1. Key generation roundtrip: Generate an EcKeyPair, convert to JwtKeys, sign a JWT, verify it with the public key.
2. PEM format assertions: Generate keys and verify the PEM headers (BEGIN PUBLIC KEY, BEGIN PRIVATE KEY).
3. Cross-library compatibility: Generate keys with the new code, verify the PEM can be parsed by jsonwebtoken (this is essentially what test 1 does).
4. Write/read roundtrip: Write keys to temp files, read them back, verify they produce valid JwtKeys.

I can add these tests if you'd like.

Verdict

The core change is sound. PEM formats are compatible, the curve is correct, and existing key files will continue to work. The swapped comments should be fixed. The TLS backend change is a nice bonus.

[bfops]

I added this since we now build in release mode here, and we started hitting this issue again.

I still have no idea why this happens since we no longer cache-on-failure, but.. /shrug.

[clockwork-labs-bot]

Closing this stale PR in favor of #4833, which tracks reimplementing the work on top of current master. Keeping this PR around as historical context and source material.