Cap HTTP response body size to bound adversarial-target memory use

The scanner's HTTP egress paths buffered response bodies into memory with
no size ceiling on three independent code paths:

  * common/.../OkHttpHttpClient.java :: parseResponse — used by every
    detector going through send / sendAsync. Calls
    okResponse.body().bytes(), which only enforces a 2 GB cap on bodies
    advertising a Content-Length and that ceiling is bypassed entirely
    by chunked encoding with no Content-Length header.
  * common/.../OkHttpHttpClient.java :: sendAsIs — used by the
    crafted-URL probes that fire against hostile targets. Calls
    ByteString.readFrom on the raw HttpURLConnection input stream with
    no ceiling at all.
  * plugin_server/py/.../requests_http_client.py :: send +
    _parse_response — uses session.send() with the default
    stream=False, which buffers the entire body into the Response
    object before returning, then reads res.content unconditionally.

Because Tsunami probes adversarial endpoints by definition, an
attacker-controlled target can detect the TsunamiSecurityScanner
User-Agent on inbound requests and serve an unbounded chunked response.
Each chunk arrives within the read-timeout window so the read timeout
never fires, and the body buffer grows until the JVM heap or Python
process is exhausted (CWE-770 / CWE-400). The result: a compromised
asset can deterministically crash the scanner that is probing it,
producing the false-negative "scanned, no findings" outcome an attacker
wants from a detection tool.

Fix: introduce a configurable max-response-body cap (default 100 MB)
enforced at every body-ingestion point. Reads are drained chunk by
chunk into a bounded buffer; once the cap is exceeded, the underlying
connection is closed and the call fails with IOException / IOError so
the affected probe errors out cleanly while the surrounding scan
continues.

Wiring:
  * Java: new HttpClientModule.@MaxResponseBodyBytes provider resolves
    --http-client-max-response-body-mb (CLI) → maxResponseBodyMb
    (config) → 100 MB default, mirroring the existing timeout flag
    pattern. New OkHttpHttpClient constructor + builder setter expose
    the cap; the legacy 6-arg constructor is preserved with the
    default value to keep external callers source-compatible.
  * Python: new --max_response_body_mb absl flag plumbed through
    RequestsHttpClientBuilder.set_max_response_body_bytes into
    RequestsHttpClient. session.send() now passes stream=True so the
    body never lands in the Response object before we get a chance to
    cap it.

Tests:
  * OkHttpHttpClientTest: send / sendAsIs throw IOException when the
    body exceeds the cap; bodies within the cap pass through unchanged.
  * requests_http_client_test: send raises IOError on advertised
    Content-Length over cap, on actual streamed bytes over cap, and
    passes through small bodies unchanged.

Disclosure note: the unbounded reads were originally surfaced by an
external security review. The patch is local to the HTTP-client layer;
no detector code changes and no detector behavior changes for any
response under the cap. Couldn't run gradle/pytest locally (no wrapper
checked in, no project-wide pytest harness configured); relying on CI.

Author: Ankit Prateek

common/src/main/java/com/google/tsunami/common/net/http/HttpClientCliOptions.java

@@ -65,12 +65,26 @@ public final class HttpClientCliOptions implements CliOption {
       description = "User-Agent to use in HTTP requests.")
   public String userAgent = HttpClient.TSUNAMI_USER_AGENT;
 
+  @Parameter(
+      names = "--http-client-max-response-body-mb",
+      description =
+          "Maximum size in megabytes of an HTTP response body the client will buffer in memory."
+              + " Responses larger than this cause the affected probe to fail with an IOException"
+              + " while the surrounding scan continues. Defaults to 100 MB.")
+  Integer maxResponseBodyMb;
+
   @Override
   public void validate() {
     validateTimeout("--http-client-call-timeout-seconds", callTimeoutSeconds);
     validateTimeout("--http-client-connect-timeout-seconds", connectTimeoutSeconds);
     validateTimeout("--http-client-read-timeout-seconds", readTimeoutSeconds);
     validateTimeout("--http-client-write-timeout-seconds", writeTimeoutSeconds);
+    if (maxResponseBodyMb != null && maxResponseBodyMb <= 0) {
+      throw new ParameterException(
+          String.format(
+              "--http-client-max-response-body-mb must be positive, received %d.",
+              maxResponseBodyMb));
+    }
   }
 
   private static void validateTimeout(String flagName, @Nullable Integer value) {

common/src/main/java/com/google/tsunami/common/net/http/HttpClientConfigProperties.java

@@ -50,4 +50,11 @@ public final class HttpClientConfigProperties {
    * more details.
    */
   Integer writeTimeoutSeconds;
+
+  /**
+   * Maximum size in megabytes of an HTTP response body the client will buffer in memory. Responses
+   * larger than this cause the affected probe to fail with an IOException while the surrounding
+   * scan continues. Defaults to 100 MB when unset.
+   */
+  Integer maxResponseBodyMb;
 }

common/src/main/java/com/google/tsunami/common/net/http/HttpClientModule.java

@@ -141,9 +141,16 @@ HttpClient provideOkHttpHttpClient(
       ConnectionFactory connectionFactory,
       @LogId String logId,
       @ConnectTimeout Duration connectTimeout,
-      @UserAgent String userAgent) {
+      @UserAgent String userAgent,
+      @MaxResponseBodyBytes long maxResponseBodyBytes) {
     return new OkHttpHttpClient(
-        okHttpClient, trustAllCertificates, connectionFactory, logId, connectTimeout, userAgent);
+        okHttpClient,
+        trustAllCertificates,
+        connectionFactory,
+        logId,
+        connectTimeout,
+        userAgent,
+        maxResponseBodyBytes);
   }
 
   @Provides
@@ -271,6 +278,23 @@ String provideUserAgent(HttpClientCliOptions httpClientCliOptions) {
     return HttpClient.TSUNAMI_USER_AGENT;
   }
 
+  @Provides
+  @MaxResponseBodyBytes
+  long provideMaxResponseBodyBytes(
+      HttpClientCliOptions httpClientCliOptions,
+      HttpClientConfigProperties httpClientConfigProperties) {
+    Integer megabytes = null;
+    if (httpClientCliOptions.maxResponseBodyMb != null) {
+      megabytes = httpClientCliOptions.maxResponseBodyMb;
+    } else if (httpClientConfigProperties.maxResponseBodyMb != null) {
+      megabytes = httpClientConfigProperties.maxResponseBodyMb;
+    }
+    if (megabytes == null) {
+      return OkHttpHttpClient.DEFAULT_MAX_RESPONSE_BODY_BYTES;
+    }
+    return ((long) megabytes) * 1024L * 1024L;
+  }
+
   @Qualifier
   @Retention(RetentionPolicy.RUNTIME)
   @Target({ElementType.PARAMETER, ElementType.METHOD, ElementType.FIELD})
@@ -326,6 +350,11 @@ String provideUserAgent(HttpClientCliOptions httpClientCliOptions) {
   @Target({ElementType.PARAMETER, ElementType.METHOD, ElementType.FIELD})
   @interface UserAgent {}
 
+  @Qualifier
+  @Retention(RetentionPolicy.RUNTIME)
+  @Target({ElementType.PARAMETER, ElementType.METHOD, ElementType.FIELD})
+  @interface MaxResponseBodyBytes {}
+
   /** Builder for {@link HttpClientModule}. */
   public static final class Builder {
     private static final int DEFAULT_CONNECTION_POOL_MAX_IDLE = 5;

common/src/main/java/com/google/tsunami/common/net/http/OkHttpHttpClient.java

@@ -30,7 +30,9 @@
 import com.google.protobuf.ByteString;
 import com.google.tsunami.common.net.http.javanet.ConnectionFactory;
 import com.google.tsunami.proto.NetworkService;
+import java.io.ByteArrayOutputStream;
 import java.io.IOException;
+import java.io.InputStream;
 import java.net.HttpURLConnection;
 import java.time.Duration;
 import java.util.List;
@@ -46,6 +48,8 @@
 import okhttp3.RequestBody;
 import okhttp3.Response;
 import okhttp3.ResponseBody;
+import okio.Buffer;
+import okio.BufferedSource;
 import org.checkerframework.checker.nullness.qual.Nullable;
 
 /**
@@ -55,12 +59,19 @@
 final class OkHttpHttpClient extends HttpClient {
   private static final GoogleLogger logger = GoogleLogger.forEnclosingClass();
 
+  // Default ceiling on response body bytes a single HTTP call may buffer in memory. Targets
+  // probed by the scanner are by definition adversarial, so an unbounded read is a remote DoS
+  // primitive (CWE-770 / CWE-400). 100 MB comfortably accommodates any HTML/JSON payload a
+  // detector should ever need while bounding the worst-case allocation per call.
+  static final long DEFAULT_MAX_RESPONSE_BODY_BYTES = 100L * 1024 * 1024;
+
   private final OkHttpClient okHttpClient;
   private final boolean trustAllCertificates;
   private final ConnectionFactory connectionFactory;
   private final String logId;
   private final Duration connectionTimeout;
   private final String userAgent;
+  private final long maxResponseBodyBytes;
 
   OkHttpHttpClient(
       OkHttpClient okHttpClient,
@@ -69,12 +80,31 @@ final class OkHttpHttpClient extends HttpClient {
       String logId,
       Duration connectionTimeout,
       String userAgent) {
+    this(
+        okHttpClient,
+        trustAllCertificates,
+        connectionFactory,
+        logId,
+        connectionTimeout,
+        userAgent,
+        DEFAULT_MAX_RESPONSE_BODY_BYTES);
+  }
+
+  OkHttpHttpClient(
+      OkHttpClient okHttpClient,
+      boolean trustAllCertificates,
+      ConnectionFactory connectionFactory,
+      String logId,
+      Duration connectionTimeout,
+      String userAgent,
+      long maxResponseBodyBytes) {
     this.okHttpClient = checkNotNull(okHttpClient);
     this.trustAllCertificates = trustAllCertificates;
     this.connectionFactory = checkNotNull(connectionFactory);
     this.logId = logId;
     this.connectionTimeout = connectionTimeout;
     this.userAgent = isNullOrEmpty(userAgent) ? TSUNAMI_USER_AGENT : userAgent;
+    this.maxResponseBodyBytes = maxResponseBodyBytes;
   }
 
   /**
@@ -134,7 +164,7 @@ public HttpResponse sendAsIs(HttpRequest httpRequest) throws IOException {
     return HttpResponse.builder()
         .setStatus(HttpStatus.fromCode(responseCode))
         .setHeaders(responseHeadersBuilder.build())
-        .setBodyBytes(ByteString.readFrom(connection.getInputStream()))
+        .setBodyBytes(readStreamWithCap(connection.getInputStream(), httpRequest.url()))
         .build();
   }
 
@@ -310,7 +340,7 @@ private static RequestBody buildRequestBody(HttpRequest httpRequest) {
         mediaType, httpRequest.requestBody().orElse(ByteString.EMPTY).toByteArray());
   }
 
-  private static HttpResponse parseResponse(Response okResponse) throws IOException {
+  private HttpResponse parseResponse(Response okResponse) throws IOException {
     logger.atInfo().log(
         "Received HTTP response with code '%d' for request to '%s'.",
         okResponse.code(), okResponse.request().url());
@@ -322,11 +352,65 @@ private static HttpResponse parseResponse(Response okResponse) throws IOExceptio
             .setResponseUrl(okResponse.request().url());
     if (!okResponse.request().method().equals(HttpMethod.HEAD.name())
         && okResponse.body() != null) {
-      httpResponseBuilder.setBodyBytes(ByteString.copyFrom(okResponse.body().bytes()));
+      httpResponseBuilder.setBodyBytes(
+          readBodyWithCap(okResponse.body(), okResponse.request().url().toString()));
     }
     return httpResponseBuilder.build();
   }
 
+  // Reads an OkHttp response body with a hard byte ceiling. Bypasses ResponseBody.bytes()
+  // because that buffers the entire body and only enforces a 2 GB ceiling on bodies whose
+  // Content-Length is advertised (a malicious chunked-encoded peer with no Content-Length
+  // bypasses it entirely).
+  private ByteString readBodyWithCap(ResponseBody body, String requestUrl) throws IOException {
+    long advertised = body.contentLength();
+    if (advertised > maxResponseBodyBytes) {
+      throw new IOException(
+          String.format(
+              "Response body for %s advertised %d bytes, exceeds cap of %d bytes.",
+              requestUrl, advertised, maxResponseBodyBytes));
+    }
+    Buffer sink = new Buffer();
+    long total = 0L;
+    try (BufferedSource source = body.source()) {
+      long n;
+      while ((n = source.read(sink, 8192L)) != -1L) {
+        total += n;
+        if (total > maxResponseBodyBytes) {
+          sink.clear();
+          throw new IOException(
+              String.format(
+                  "Response body for %s exceeds cap of %d bytes (read so far: %d).",
+                  requestUrl, maxResponseBodyBytes, total));
+        }
+      }
+    }
+    return ByteString.copyFrom(sink.readByteArray());
+  }
+
+  // Reads from a raw InputStream (used by sendAsIs, which goes through HttpURLConnection
+  // rather than OkHttp). Strictly worse than the OkHttp path absent a cap because the
+  // underlying ByteString.readFrom drains until EOF with no ceiling whatsoever.
+  private ByteString readStreamWithCap(InputStream stream, String requestUrl) throws IOException {
+    ByteArrayOutputStream sink = new ByteArrayOutputStream();
+    byte[] buffer = new byte[8192];
+    long total = 0L;
+    try (InputStream in = stream) {
+      int read;
+      while ((read = in.read(buffer)) != -1) {
+        total += read;
+        if (total > maxResponseBodyBytes) {
+          throw new IOException(
+              String.format(
+                  "Response body for %s exceeds cap of %d bytes (read so far: %d).",
+                  requestUrl, maxResponseBodyBytes, total));
+        }
+        sink.write(buffer, 0, read);
+      }
+    }
+    return ByteString.copyFrom(sink.toByteArray());
+  }
+
   private static HttpHeaders convertHeaders(Headers headers) {
     HttpHeaders.Builder headersBuilder = HttpHeaders.builder();
     for (int i = 0; i < headers.size(); i++) {
@@ -357,6 +441,7 @@ public static class OkHttpHttpClientBuilder extends Builder<OkHttpHttpClient> {
     private String logId;
     private Duration connectionTimeout;
     private String userAgent;
+    private long maxResponseBodyBytes;
 
     private OkHttpHttpClientBuilder(OkHttpHttpClient okHttpHttpClient) {
       this.okHttpClient = okHttpHttpClient.okHttpClient;
@@ -366,6 +451,7 @@ private OkHttpHttpClientBuilder(OkHttpHttpClient okHttpHttpClient) {
       this.logId = okHttpHttpClient.logId;
       this.connectionTimeout = okHttpHttpClient.connectionTimeout;
       this.userAgent = okHttpHttpClient.userAgent;
+      this.maxResponseBodyBytes = okHttpHttpClient.maxResponseBodyBytes;
     }
 
     @Override
@@ -392,6 +478,12 @@ public OkHttpHttpClientBuilder setUserAgent(String userAgent) {
       return this;
     }
 
+    @CanIgnoreReturnValue
+    public OkHttpHttpClientBuilder setMaxResponseBodyBytes(long maxResponseBodyBytes) {
+      this.maxResponseBodyBytes = maxResponseBodyBytes;
+      return this;
+    }
+
     @Override
     public OkHttpHttpClient build() {
       return new OkHttpHttpClient(
@@ -400,7 +492,8 @@ public OkHttpHttpClient build() {
           connectionFactory,
           logId,
           connectionTimeout,
-          userAgent);
+          userAgent,
+          maxResponseBodyBytes);
     }
   }
 }

common/src/test/java/com/google/tsunami/common/net/http/OkHttpHttpClientTest.java

@@ -710,6 +710,63 @@ protected void configure() {
     mockWebServer.shutdown();
   }
 
+  @Test
+  public void send_whenResponseBodyExceedsCap_throwsIOException() throws IOException {
+    String responseBody = "0123456789ABCDEF response body that comfortably exceeds 16 bytes";
+    mockWebServer.enqueue(
+        new MockResponse()
+            .setResponseCode(HttpStatus.OK.code())
+            .setHeader(CONTENT_TYPE, MediaType.PLAIN_TEXT_UTF_8.toString())
+            .setBody(responseBody));
+    mockWebServer.start();
+
+    HttpClient cappedClient = buildCappedClient(httpClient, 16L);
+
+    String requestUrl = mockWebServer.url("/test/get-oversized").toString();
+    IOException thrown =
+        assertThrows(
+            IOException.class,
+            () -> cappedClient.send(get(requestUrl).withEmptyHeaders().build()));
+    assertThat(thrown).hasMessageThat().contains("exceeds cap of 16 bytes");
+  }
+
+  @Test
+  public void sendAsIs_whenResponseBodyExceedsCap_throwsIOException() throws IOException {
+    mockWebServer.setDispatcher(new SendAsIsTestDispatcher());
+    mockWebServer.start();
+
+    HttpClient cappedClient = buildCappedClient(httpClient, 8L);
+
+    HttpUrl baseUrl = mockWebServer.url("/");
+    String requestUrl =
+        new URL(baseUrl.scheme(), baseUrl.host(), baseUrl.port(), "/send-as-is/oversized")
+            .toString();
+
+    IOException thrown =
+        assertThrows(
+            IOException.class,
+            () -> cappedClient.sendAsIs(get(requestUrl).withEmptyHeaders().build()));
+    assertThat(thrown).hasMessageThat().contains("exceeds cap of 8 bytes");
+  }
+
+  @Test
+  public void send_whenResponseBodyWithinCap_returnsExpectedHttpResponse() throws IOException {
+    String responseBody = "tiny";
+    mockWebServer.enqueue(
+        new MockResponse()
+            .setResponseCode(HttpStatus.OK.code())
+            .setHeader(CONTENT_TYPE, MediaType.PLAIN_TEXT_UTF_8.toString())
+            .setBody(responseBody));
+    mockWebServer.start();
+
+    HttpClient cappedClient = buildCappedClient(httpClient, 16L);
+
+    String requestUrl = mockWebServer.url("/test/within-cap").toString();
+    HttpResponse response = cappedClient.send(get(requestUrl).withEmptyHeaders().build());
+
+    assertThat(response.bodyBytes()).hasValue(ByteString.copyFrom(responseBody, UTF_8));
+  }
+
   @Test
   public void send_default_userAgent() throws IOException, InterruptedException {
     String responseBody = "test response";
@@ -761,6 +818,16 @@ protected void configure() {
     assertThat(mockWebServer.takeRequest().getHeader(USER_AGENT)).isEqualTo(userAgentOverride);
   }
 
+  // Returns a copy of the supplied client whose response-body cap is set to {@code capBytes}.
+  // The test only exercises the OkHttp builder, but the cast is safe because the injected
+  // HttpClient is always backed by OkHttpHttpClientBuilder in this module.
+  @SuppressWarnings("unchecked")
+  private static HttpClient buildCappedClient(HttpClient client, long capBytes) {
+    OkHttpHttpClient.OkHttpHttpClientBuilder builder =
+        (OkHttpHttpClient.OkHttpHttpClientBuilder) (HttpClient.Builder<?>) client.modify();
+    return builder.setMaxResponseBodyBytes(capBytes).build();
+  }
+
   private MockWebServer startMockWebServerWithSsl(InetAddress serverAddress)
       throws GeneralSecurityException, IOException {
     MockWebServer mockWebServer = new MockWebServer();