Handling of CVE-2023-44487 / HTTP2 Rapid Reset

[lkarsten]

Hi. Brought this up on IRC, but I think there should be some public record of it as well.

How well does Varnish handle the Rapid Reset attack?

Consensus seems to be that some new rate limiting should be added to Varnish.

Links:

• https://ubuntu.com/security/CVE-2023-44487
• https://blog.qualys.com/vulnerabilities-threat-research/2023/10/10/cve-2023-44487-http-2-rapid-reset-attack

Related fixes:

• h2o/h2o@28fe151

[bsdphk]

Or maybe we should simply violate this part of RFC7540 ?

SETTINGS_MAX_CONCURRENT_STREAMS (0x3): Indicates the maximum number
of concurrent streams that the sender will allow. This limit is
directional: it applies to the number of streams that the sender
permits the receiver to create. Initially, there is no limit to
this value. [...]

I'm pretty sure that a limit of 10 or 20 would cater for all non-hostile uses.

[kjetilho]

I'm pretty sure that a limit of 10 or 20 would cater for all non-hostile uses.

I think this was what Cloudflare had to back down on?

While responding to DoS attacks enabled by CVE-2023-44487, Cloudflare reduced maximum stream concurrency to 64. Before making this change, we were unaware that clients don't wait for SETTINGS and instead assume a concurrency of 100. Some web pages, such as an image gallery, do indeed cause a browser to send 100 requests immediately at the start of a connection. Unfortunately, the 36 streams above our limit all needed to be reset, which triggered our counting mitigations. This meant that we closed connections on legitimate clients, leading to a complete page load failure. As soon as we realized this interoperability issue, we changed the maximum stream concurrency to 100.

However, they employed blacklisting for IP of (inaccurately perceived) hostile users. Just sending resets without further punishment will not have the same kind of impact on user experience.

[jailbird777]

Looks like ATS went for rate limiting also: apache/trafficserver@8316188

[daghf]

I'm pretty sure that a limit of 10 or 20 would cater for all non-hostile uses.

SETTINGS_MAX_CONCURRENT_STREAMS is not a viable mitigation. The issue is that only streams that are in the "open" or either of the "half-closed" states will count towards this limit. Receving an RST puts the stream in state "closed", which is not counted towards SETTINGS_MAX_CONCURRENT_STREAMS.

Thus an attacker can endlessly spam us with new streams+RST regardless of the value of SETTINGS_MAX_CONCURRENT_STREAMS, and still be safely within the confines set by the h/2 spec ..

BTW we've done some experimentation with a rate-limiting facility for RSTs that looks very promising as a mitigation. I will open a PR here soon.

[dridi]

We currently have two complementary mitigations:

• h2: Add a rate limit facility for h/2 RST handling ("Rapid reset" mitigation) #3997
• vcl_vrt: Skip VCL execution if the client is gone #3998

[TomasKorbar]

Thanks a lot guys for your work on this, it is really appreciated. Do you have any idea when these 2 changes could be merged?

[nigoroll]

@TomasKorbar during bugwash today we agreed that we want to review again tomorrow and merge what is going to be merged for patch releases soon after. #3998 is planned to be merged even before the review session.