Tip: Working SSL/TLS certs for servers on ESP8266

[stonehippo]

Dropping this here, in the hopes of saving others some frustration and documenting this for my future self.

I am working on a device (a remote switch) based on an ESP8266. It implements a very simple HTTP server and of course I wanted to use SSL to secure the connection. But I kept running into issues with the RSA key I generated along with the certificate. Any connection to the server triggered an OSError -2: Invalid key. This occurred at key sizes from 512 to 2048 bits, with sha1 and sha256. These are pretty standard keys, so it should have worked.

The ESP8266 is a very constrained device and the SSL implementation for its MicroPython port is based on axTLS, not the more capable mBed TLS used by some other ports. It appears that axTLS doesn't like PKCS#8 packaged RSA private keys; they seem to fail axTLS's key validation code.

The solve turned out be telling openssl to use PKCS#1 packaging; modern releases of OpenSSL use PKCS#8 by default. After switching to this, current minimum best practice RSA keys at 2048 bits and sha256 worked fine.

When preparing your RSA private keys to deploy on an ESP8266 running micropython, you can use something like this:

openssl rsa -in original_key.pem -out key.der -traditional -outform DER

It's the -traditional parameter that I was missing with my keys. Note that you need to specify it when the key is output to DER format, whether that's when it is first created standalone or as part of a single command self-signed cert.

[stonehippo]

Here is a full working example for creating self-signed RSA key and x509 certificate that works on an ESP8266 running micropython. See the attached config files, too.

This is confirmed to work with v1.27.0 for ESP8266, downloaded from micropython.org, running on an Adafruit Huzzah breakout. I have not tried it with the 1MiB or 512KiB versions of the firware; boards like the ESP-01 are so resource constrained, I don't touch them these days.

#generate the RSA private key
openssl genpkey -outform PEM -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out priv.key

#Convert the private key to PKCS#1 and DER
#THIS IS THE CRUCIAL STEP FOR THE KEY ON AN ESP8266
openssl rsa -inform PEM -outform DER -traditional -passin pass: -in priv.key -out key.der

#Create the CSR (see csrconfig.txt)
openssl req -new -nodes -key priv.key -config csrconfig.txt -nameopt utf8 -utf8 -out cert.csr

#Self-sign your CSR (see certconfig.txt)
openssl req -x509 -nodes -in cert.csr -days 1095 -key priv.key -config certconfig.txt -extensions req_ext -nameopt utf8 -utf8 -out cert.crt

#convert cert to DER
openssl x509 -in cert.crt -out cert.der -outform DER

Copy key.der and cert.der to your ESP8266. You can read them in with something like

with open("cert.der","rb") as f:
    cert = f.read()
    f.close()
with open("key.der","rb") as f:
    key = f.read()
    f.close()

Pass cert and key to ssl.wrap_socket().

certconfig.txt
csrconfig.txt

[DoubleVee73]

Thanks for taking the time to post this - really useful

[stonehippo]

I should mention that while this works, it's sloooooow. As noted in the axTLS documentation, a TLS handshake with a 2048 bit RSA key can take significant time. A smaller key might work better, but 512 and 1024 keys are considered insecure and will be rejected by modern browsers and tools like curl unless they are instructed to allow such small keys.

Even 2048 bit RSA is only considered to be sufficiently secure for a few more years. The better answer would be to use ECC keys. I believe that's been shown to be doable on an ESP8266 with micropython by enabling mBed TLS, though it currently requires a custom build (I think the tradeoff is a better SSL/TLS stack for fewer resources available due to increase size/memory usage).

[stonehippo]

One more quick note on this: you'll probably want to use a 1024 bit RSA key. In my testing this, I found that an SSL handshake with a 512 bit key took ~1 second to complete with cURL generating the request. This required using the --insecure flag, or cURL would balk at the small key; trying this with the latest Firefox also triggered a security warning. A key at 1024 bits had a handshake time around ~8 seconds, which is slow but acceptable, but still required --insecure with cURL, but wasn't flagged by Firefox. A 2048 bit key took about 50 seconds (!), but didn't require --insecure or trigger any browser issues.

If better speed is needed in your application, switch from the limited ESP8266 to something with a bit more juice, like an ESP32 or RP2040 + WiFi radio (like on the RPi Pico W). For me, this is a small switch that isn't going to get called very often and response time is not that critical, so I think I'll go with a 1024 bit key.

[projectgus]

Thanks for posting all these steps, @stonehippo.

I think it is worth pointing out that anyone can factor an RSA-512 key in about 4 days on an ordinary desktop PC https://yurichev.com/news/20220210_RSA/ - or a few hours if you rent public cloud compute. RSA-1024 is, as far as I know, still uneconomical for folks like you and I to crack.

Still, I'd suggest that - given these ciphers are flagged as "insecure" anyway - it may be better to choose plain HTTP and accept the consequence of known insecurity rather than the possible illusion of security. Or go with your suggestion to switch to a newer micro than ESP8266.