fix(otlp-http): limit response body size to prevent memory exhaustion by SATVIKsynopsis · Pull Request #3501 · open-telemetry/opentelemetry-rust

SATVIKsynopsis · 2026-05-07T13:03:39Z

Changes

I traced all HTTP response handling to export_http_once in opentelemetry-oltp/src/exporter/http/mod.rs and since all three signal exporters funnel through this one function, fixing it here covered everything. I also added a MAX_RESPONSE_BODY_BYTES (10 mb) cap on 2xx response bodies and stopped reading non-2xx bodies entirely. I also added a test with a mock client that returns an oversized body to verify the cap works.

Merge requirement checklist

CONTRIBUTING guidelines followed
Unit tests added/updated (if applicable)
Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
Changes in public API reviewed (if applicable)

linux-foundation-easycla · 2026-05-07T13:03:46Z

The committers listed above are authorized under a signed CLA.

✅ login: SATVIKsynopsis / name: SATVIKsynopsis (223f668, 3725278, 43b2fd2, 658b6d2, d426c90, d863d9d, ec69053, ef6bae3)

codecov · 2026-05-07T13:06:40Z

Codecov Report

❌ Patch coverage is 97.43590% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 82.9%. Comparing base (8882149) to head (2faf365).

Files with missing lines	Patch %	Lines
opentelemetry-http/src/lib.rs	97.4%	1 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##            main   #3501   +/-   ##
=====================================
  Coverage   82.9%   82.9%           
=====================================
  Files        130     130           
  Lines      27484   27523   +39     
=====================================
+ Hits       22800   22839   +39     
  Misses      4684    4684

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

SATVIKsynopsis · 2026-05-07T20:35:34Z

The current fix checks the size after the full body is already buffered. To truly prevent the allocation, the body needs to be read in chunks and aborted early as suggested earlier as well. I have an approach using bytes_stream() for reqwest async, Read::take() for reqwest blocking and body.frame() for hyper. should i go ahead with this or is there a preferred streaming pattern in this codebase i should follow?

lalitb · 2026-05-08T20:56:13Z

The current fix checks the size after the full body is already buffered. To truly prevent the allocation, the body needs to be read in chunks and aborted early as suggested earlier as well. I have an approach using bytes_stream() for reqwest async, Read::take() for reqwest blocking and body.frame() for hyper. should i go ahead with this or is there a preferred streaming pattern in this codebase i should follow?

Yes, that approach sounds right. The key requirement is that the limit is enforced while the body is being read, before it becomes Response<Bytes>. Using bytes_stream() for reqwest async, Read::take() for reqwest blocking, and body.frame() for hyper sounds like the right direction here. Please make sure each path aborts as soon as the accumulated bytes exceed MAX_RESPONSE_BODY_BYTES, so OtlpHttpClient only receives already-bounded Bytes.

Can we try to accommodate this in next release - #3508 - @cijothomas if it can be fixed soon

SATVIKsynopsis · 2026-05-09T10:06:14Z

Updated the fix to use chunked reading in all three HTTP client implementations as in reqwest async, reqwest blocking and hyper so the limit is enforced while the body is being read before it becomes Response<Bytes>. adding unit tests for the body limit in opentelemetry-http directly would require a mock HTTP server. happy to add one if you can suggest a preferred test crate for this codebase.

SATVIKsynopsis · 2026-05-11T06:35:43Z

I have added the Err case in hyper client and resolved the nitpick.

SATVIKsynopsis · 2026-05-16T10:48:15Z

Hi, just checking in. let me know if any further changes are needed or if there's anything blocking the review. Happy to address any feedback.

lalitb

LGTM. Since this is a behavior change in the built-in opentelemetry-http clients, we should add a short vNext changelog entry for opentelemetry-http before merging/releasing.

SATVIKsynopsis · 2026-06-03T17:25:39Z

LGTM. Since this is a behavior change in the built-in opentelemetry-http clients, we should add a short vNext changelog entry for opentelemetry-http before merging/releasing.

Thanks. will add a short vNext changelog entry for before merging/releasing.

SATVIKsynopsis · 2026-06-03T19:04:23Z

I have added the vNext changelog.

cijothomas · 2026-06-09T19:56:43Z

I also added a test with a mock client that returns an oversized body to verify the cap works.

Is that included in the PR? I am not seeing tests added.?

cijothomas · 2026-06-09T19:59:43Z

+            let status = response.status();
            let headers = std::mem::take(response.headers_mut());
-
+            let mut body_bytes = bytes::BytesMut::new();


could we start with some capacity other than 0 to avoid reallocation?

Good point. Let me check whether we can leverage the response body size hint or content length to preallocate an appropriate capacity rather than starting from 0.

i checked both implementations and was able to leverage the response size information for pre allocation and in both cases the capacity is capped at MAX_RESPONSE_BODY_BYTES before allocating.

let mut body_bytes = bytes::BytesMut::new();

I don't see pre-allocation still?

you're right, that's still showing the previous version. the pre allocation change is currently only in my local branch using the response size hint/content length); I'll push it shortly.

SATVIKsynopsis · 2026-06-09T20:03:46Z

I also added a test with a mock client that returns an oversized body to verify the cap works.

Is that included in the PR? I am not seeing tests added.?

I had added a test earlier in opentelemetry-otlp but forgot to move it when the logic was moved to opentelemetry-http. I'll add coverage in opentelemetry-http using a small Axum test server.

cijothomas · 2026-06-10T13:49:43Z

+                .take(MAX_RESPONSE_BODY_BYTES as u64 + 1)
+                .read_to_end(&mut body_bytes)?;
+            if body_bytes.len() > MAX_RESPONSE_BODY_BYTES {
+                return Err("response body too large".into());


I don't think this is correct. If we return Err from here, I think this would cause OTLP Exporter to consider the request as failed, and likely initiate retry waves.
If the response is too big, we don't need to retry (we shouldn't).

good point. returning an Err here would indeed make the exporter treat it as a request failure and potentially retry so then i will take a look at how the response body is consumed by the oltp exporter and adjust the handling so oversized responses don't trigger retries.

cijothomas

Thanks for working on this. We need to do a more holistic approach to handle this - when response size is bigger, this PR returns Err and the OTLP Exporter would retry.
We should prevent that. Most likely, this'd mean conditionally reading the response body (we only need the body for partial response case, and then for loggging purposes.)

(Also - small note on pre-allocating the capacity for reading response).

SATVIKsynopsis · 2026-06-10T14:23:05Z

Thanks for the feedback. looking through the oltp http flow, successful response bodies appear to be consumed primarily for partial success parsing so for oversized responses, would you prefer that we treat them similarly to an empty response body like skip partial success parsing and continue without retrying or would you rather move/condition the response body reading so it's only performed when needed for partial success or logging scenarios?

SATVIKsynopsis · 2026-06-10T19:47:28Z

I have added the pre-allocation step in all three, added the http axum mock server, and the oltp exporter retry fix. All tests pass now. happy to address any feedback.

cijothomas · 2026-06-19T03:51:58Z

            let mut response = self.execute(request)?.error_for_status()?;
+            let status = response.status();
            let headers = std::mem::take(response.headers_mut());
+            let mut body_bytes = Vec::new();


this can also be pre-allocated right?

cijothomas · 2026-06-19T03:52:44Z

 tokio = { workspace = true, features = ["time"], optional = true }

+[dev-dependencies]
+axum = { version = "0.8" }


I understand its dev-dev only, but wondering if we can avoid this ?

I'll switch to a minimal raw TCP test server instead so we don't need to add axum.

SATVIKsynopsis · 2026-06-19T06:40:55Z

I have added the pre-allocation on reqwest-blocking and added a minimal TCP test instead of axum.

SATVIKsynopsis added 2 commits May 7, 2026 18:04

oltp http change

43b2fd2

samll fix

223f668

SATVIKsynopsis requested a review from a team as a code owner May 7, 2026 13:03

SATVIKsynopsis changed the title ~~OLTP HTTP tracing for non-2xx bodies~~ fix(otlp-http): limit response body size to prevent memory exhaustion May 7, 2026

test for within limit

3725278

cijothomas reviewed May 7, 2026

View reviewed changes

Comment thread opentelemetry-otlp/src/exporter/http/mod.rs Outdated

cijothomas reviewed May 7, 2026

View reviewed changes

Comment thread opentelemetry-otlp/src/exporter/http/mod.rs Outdated

SATVIKsynopsis added 2 commits May 7, 2026 19:47

name change in comment

658b6d2

4 mb constant size and linting fix

ef6bae3

lalitb reviewed May 7, 2026

View reviewed changes

Comment thread opentelemetry-otlp/src/exporter/http/mod.rs Outdated

chunks reading for max response added

d426c90

lint fix

d863d9d

lalitb reviewed May 11, 2026

View reviewed changes

Comment thread opentelemetry-http/src/lib.rs Outdated

lalitb reviewed May 11, 2026

View reviewed changes

Comment thread opentelemetry-http/src/lib.rs Outdated

err case added and nitpick solved

ec69053

SATVIKsynopsis requested review from cijothomas and lalitb May 28, 2026 20:13

Merge branch 'main' into oltp-http

064750d

lalitb reviewed Jun 3, 2026

View reviewed changes

Comment thread opentelemetry-http/src/lib.rs

lalitb approved these changes Jun 3, 2026

View reviewed changes

changelog

a59bddf

SATVIKsynopsis added 2 commits June 5, 2026 15:06

trigger ci

3654073

Merge branch 'main' into oltp-http

0ce291c

cijothomas reviewed Jun 9, 2026

View reviewed changes

Comment thread opentelemetry-http/src/lib.rs Outdated

cijothomas reviewed Jun 9, 2026

View reviewed changes

Comment thread opentelemetry-http/CHANGELOG.md Outdated

cijothomas reviewed Jun 9, 2026

View reviewed changes

SATVIKsynopsis added 4 commits June 10, 2026 01:36

vnext changes

b9bf59e

vnext changes

8486b18

one level abstraction of body_bytes

628325d

lint fix

ccdaead

cijothomas reviewed Jun 10, 2026

View reviewed changes

cijothomas requested changes Jun 10, 2026

View reviewed changes

pre-allocation + axum-server + err case

a610c19

SATVIKsynopsis requested a review from cijothomas June 14, 2026 19:42

Merge branch 'main' into oltp-http

03bd90e

cijothomas reviewed Jun 19, 2026

View reviewed changes

tcp test added

2faf365

Uh oh!

Conversation

SATVIKsynopsis commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Merge requirement checklist

Uh oh!

linux-foundation-easycla Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SATVIKsynopsis commented May 7, 2026

Uh oh!

lalitb commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SATVIKsynopsis commented May 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SATVIKsynopsis commented May 11, 2026

Uh oh!

SATVIKsynopsis commented May 16, 2026

Uh oh!

Uh oh!

lalitb left a comment

Choose a reason for hiding this comment

Uh oh!

SATVIKsynopsis commented Jun 3, 2026

Uh oh!

SATVIKsynopsis commented Jun 3, 2026

Uh oh!

Uh oh!

Uh oh!

cijothomas commented Jun 9, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SATVIKsynopsis commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cijothomas left a comment

Choose a reason for hiding this comment

Uh oh!

SATVIKsynopsis commented Jun 10, 2026

Uh oh!

SATVIKsynopsis commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SATVIKsynopsis commented Jun 19, 2026

Uh oh!

Reviewers

Assignees

SATVIKsynopsis commented May 7, 2026 •

edited

Loading

linux-foundation-easycla Bot commented May 7, 2026 •

edited

Loading

codecov Bot commented May 7, 2026 •

edited

Loading

lalitb commented May 8, 2026 •

edited

Loading

SATVIKsynopsis commented May 9, 2026 •

edited

Loading

SATVIKsynopsis commented Jun 9, 2026 •

edited

Loading

SATVIKsynopsis commented Jun 10, 2026 •

edited

Loading