No way to set `SSL_OP_IGNORE_UNEXPECTED_EOF` for ::HTTPClient, but Google services don't send `close_notify`

[louissobel]

Summary

• We want to set SSL_OP_IGNORE_UNEXPECTED_EOF to deal with Google server TLS implementation
• ::HTTPClient provides no way to do that, not even through global settings
• Monkeypatching BaseService#new_client seems like the only path forward

Background

My company—using Openssl 3—is seeing frequent SSL errors while using this library to interact with Google services that look like SSL_read: unexpected eof while reading. Specific services we've seen this from in the last few days are Bigquery and Storage.

We noticed that the public HTTPS load balancer service (https://cloud.google.com/load-balancing/docs/https) is documented to not send a close_notify, so we feel like it's pretty likely that at least some Google APIs also don't do this, and that's why we're observing this behavior.

HTTP clients can protect against truncation attacks using Content-Length header, so we're comfortable using the SSL_OP_IGNORE_UNEXPECTED_EOF option to ignore these unexpected EOFs for Google services.

Problem

However, there is no way we can see to actually set this option on the SSL sockets created by this library. It uses ::HTTPClient, which sets some default SSL options (https://github.com/nahi/httpclient/blob/master/lib/httpclient.rb#L446 --> https://github.com/nahi/httpclient/blob/master/lib/httpclient/ssl_config.rb#L162).

Ideally we'd be able to set SSL_OP_IGNORE_UNEXPECTED_EOF specifically when making requests to Google APIs, but even if we were comfortable always having it set, we still can't: ::HTTPClient doesn't check OpenSSL::SSL::SSLContext::DEFAULT_PARAMS or provide some other way to set SSL options globally.

Proposed fix

Give users of the library a way to further customize the SSL options used by ::HTTPClient instance used by BaseService. Some potential ideas:

• most direct: add a nilable ssl_options Integer to Google::Apis::ClientOptions, and in BaseServer#new_client check if its set and use it if so
• use faraday: HTTPClient is no longer maintained, switch to Faraday #2348 and add a hook to inject custom faraday middleware

Our workaround

In the meantime, we're probably going to monkeypatch BaseService#new_client to edit the SSL configuration. This will be brittle to library changes and hard to test (as we end up mocking out google services at higher level in most of our tests). We'd love any other ideas / or if there's some other way we're missing to set this option!

[louissobel]

@dazuma — any thoughts here?

[pdfrod]

I'm also seeing OpenSSL::SSL::SSLError: SSL_read: unexpected eof while reading errors when making requests to Cloud Storage.

It only happens occasionally, so I'm wondering if a simple retry would be enough. Maybe Google::Apis::Core::HttpCommand#error could check for that eof SSL error and turn into a Google::Apis::TransmissionError so that it can be retriable.