fix: Avoid overflows when reading json inputs by whoisj · Pull Request #8676 · triton-inference-server/server

whoisj · 2026-02-24T23:21:06Z

Check input from HTTP+JSON sizes before allocating memory for them.

Protect against integer overflows while we're at it.

mudit-eng · 2026-02-26T19:53:38Z

LGTM

mattwittwer · 2026-02-27T18:51:31Z

One edge case:
max_input_size_ is checked per-input rather than cumulatively, allowing up to a 4x memory limit bypass.
This PR checks byte_size > max_input_size_ to prevents excessive memory allocation. However, because this check is evaluated on individual inputs rather than the cumulative total of the entire HTTP request, a single request can still cause the server to allocate up to 4x the memory limit.

The scenario:
The upstream EVBufferToJson limits the physical size of the incoming HTTP JSON payload to max_input_size_.
However, the most compact JSON representation of an element is 0, (2 bytes), while an FP64 element allocates 8 bytes in memory.
With a limit set to --http-max-input-size=128000000 (128 MB), a single 128 MB JSON payload formatted like this can allocate 512MB:

{
  "parameters": {
    "in_1": {"datatype": "FP64", "data": [ ... 16 million zeros (32 MB JSON) ... ]},
    "in_2": {"datatype": "FP64", "data": [ ... 16 million zeros (32 MB JSON) ... ]},
    "in_3": {"datatype": "FP64", "data": [ ... 16 million zeros (32 MB JSON) ... ]},
    "in_4": {"datatype": "FP64", "data": [ ... 16 million zeros (32 MB JSON) ... ]}
  }
}

What happens:
The upstream check passes because the total HTTP payload is exactly 128 MB.
ExactMappingInput parses in_1. The calculated byte_size is 128 MB. The check if (byte_size > max_input_size_) passes. Triton allocates 128 MB. This repeats for in_2, in_3, and in_4. Resulting in the server allocating 512 MB of memory for a single request, bypassing the 128 MB max_input_size_ limit.

yinggeh

LGTM. Though I think it's a bit of over-engineering here.

whoisj · 2026-02-27T20:12:32Z

max_input_size_ is checked per-input rather than cumulatively, allowing up to a 4x memory limit bypass.

Agreed about exceeding the overall limit. I'll create a follow up to address that separately. Thanks for pointing it out.

yinggeh · 2026-03-04T10:56:56Z

PR title isn't following the convention.

yinggeh · 2026-03-04T10:59:43Z

I think test is required for such code change. Whether verify its functionality with existing ones or by creating a new test.

whoisj · 2026-03-06T01:43:45Z

I think test is required for such code change. Whether verify its functionality with existing ones or by creating a new test.

Added 2 new tests. Let me know if they look good to you.

yinggeh · 2026-03-06T16:57:43Z

+        )
+        error_msg = response.content.decode()
+        self.assertEqual(
+            '{"error":"Request JSON size of 89478576 bytes exceeds the maximum allowed value of 67108864 bytes. Use --http-max-input-size to increase the limit."}',


Would like to see tests that passing with size 67108864 but failed with 67108864+1.

changed the format.

yinggeh · 2026-03-06T16:58:51Z

 fi
+
+# Run test to verify that large inputs fail with default limit
+python http_input_size_limit_test.py InferSizeLimitTest.test_type_size_explosion >> $CLIENT_LOG 2>&1


I am confused with the naming here. Can you use a more descriptive name?

do you have a suggestion?

mudit-eng · 2026-03-24T06:09:50Z

          std::vector<char>& serialized = infer_req->serialized_data_.back();
          serialized.resize(byte_size);
-
+          char* serialized_base = &serialized[0];


&serialized[0] is undefined behavior if vector is empty. Safer to do something like:

char* serialized_base = serialized.data();

or

char* serialized_base = serialized.empty() ? nullptr : serialized.data();

sure. I was just using the existing code, but I'll fix up regardless.

mudit-eng · 2026-03-24T06:17:57Z

    const MappingSchema* request_schema_{nullptr};
    const MappingSchema* response_schema_{nullptr};
    const bool streaming_{false};
+    const size_t max_input_size_{0};


HTTPAPIServer already has protected member:size_t max_input_size_;. Can this create confusion?

I don't see how, they're both private fields.

mudit-eng · 2026-03-24T06:21:28Z

-         " bytes. Use --http-max-input-size to increase the limit.")
+        ("Request JSON size of " + std::to_string(length) + " + " +
+         std::to_string(overrun) +
+         " bytes exceeds the maximum allowed input size. "


Not sure if this message is intuitive. This does not convey what is the maximum input size.

the updated format was a request from @yinggeh. Can you two decide between you which you prefer?

Difference would be:

New log: Request JSON size of 1044 + 20 bytes exceeds the maximum allowed input size.

Old log: Request JSON size of 1044 bytes exceeds the maximum allowed value of 1024.

I talked to Yingge. May be there is a misunderstanding. We can keep the old message.

Plus, there is a bug in the new code. We should have printed max_input_size_ + overrun instead of 'length + overrun`. Can we validate the numbers too in the test?

Updated. Thanks for the clarity.

mudit-eng · 2026-03-24T06:23:28Z

    const std::string& name,
    triton::common::TritonJson::Value& generate_request,
-    std::map<std::string, triton::common::TritonJson::Value>& input_metadata)
+    std::map<std::string, triton::common::TritonJson::Value>& input_metadata,


Can we use unordered_map here?

no idea, but that would be changing existing code and I'd rather not because I feel that is outside the scope of this PR.

mudit-eng · 2026-03-24T06:33:27Z

    const MappingSchema* RequestSchema() { return request_schema_; }
    const MappingSchema* ResponseSchema() { return response_schema_; }

+    size_t MaxInputSize() { return max_input_size_; }


this is not being called anywhere

I'll remove.

mudit-eng · 2026-03-24T06:37:42Z

+    // allowed input size.
+    if (byte_size + consumed_input_byte_size > max_input_size_ ||
+        byte_size + consumed_input_byte_size < consumed_input_byte_size) {
+      auto overrun = byte_size + consumed_input_byte_size - max_input_size_;


Can byte_size + consumed_input_byte_size result in an overflow?

no because of the check immediately previous.

Yes, you checked for overflow here: byte_size + consumed_input_byte_size < consumed_input_byte_size) but can't there be an overflow here: auto overrun = byte_size + consumed_input_byte_size - max_input_size_

while true, this value is only to produce an error message.

not sure what we would do with an overrun of a size_t value.

update tests to match message changes

whoisj · 2026-04-03T20:23:04Z

@mudit-eng updated the error message per your comment. Updated tests to match.

CI-Pipeline-ID: 47658956

yinggeh · 2026-04-03T20:49:47Z

            self.generate_expect_failure(self._model_name, inputs, error_msg)
            self.generate_stream_expect_failure(self._model_name, inputs, error_msg)

+    def test_type_size_explosion(self):


Suggested change

def test_type_size_explosion(self):

def test_json_dtype_size_expansion_exceeds_limit_error(self):

yinggeh · 2026-04-03T20:50:12Z

            self.generate_expect_failure(self._model_name, inputs, error_msg)
            self.generate_stream_expect_failure(self._model_name, inputs, error_msg)

+    def test_type_size_explosion(self):


Please add a description comment to explain what's happening here.

mudit-eng · 2026-04-06T22:56:58Z

Thanks for addressing all the feedbacks. :)

whoisj requested review from mattwittwer, pskiran1 and yinggeh February 24, 2026 23:21

whoisj added the PR: fix A bug fix label Feb 24, 2026

yinggeh approved these changes Feb 27, 2026

View reviewed changes

yinggeh requested changes Feb 27, 2026

View reviewed changes

Comment thread src/http_server.cc

whoisj requested a review from yinggeh March 2, 2026 22:09

whoisj force-pushed the jwyman/tri-673 branch 2 times, most recently from c603e3d to 75c3c95 Compare March 3, 2026 00:09

yinggeh reviewed Mar 4, 2026

View reviewed changes

Comment thread src/http_server.cc

Comment thread src/http_server.cc

whoisj requested a review from yinggeh March 4, 2026 20:03

yinggeh reviewed Mar 4, 2026

View reviewed changes

Comment thread qa/common/gen.ONNXRuntime.gen_qa_model_repository.docker.v2.sh Outdated

whoisj changed the title ~~Avoid overflows when reading json inputs~~ fix: Avoid overflows when reading json inputs Mar 5, 2026

whoisj force-pushed the jwyman/tri-673 branch from 2586e7c to a48058d Compare March 6, 2026 01:44

whoisj requested a review from yinggeh March 6, 2026 01:48

yinggeh reviewed Mar 6, 2026

View reviewed changes

whoisj force-pushed the jwyman/tri-673 branch 4 times, most recently from 3f045a5 to 52f656f Compare March 9, 2026 15:49

github-advanced-security AI found potential problems Mar 9, 2026

View reviewed changes

Comment thread qa/L0_http/generate_endpoint_test.py Fixed

whoisj force-pushed the jwyman/tri-673 branch 2 times, most recently from db78671 to a67598d Compare March 11, 2026 20:39

whoisj force-pushed the jwyman/tri-673 branch from e793a58 to 77b3f26 Compare March 20, 2026 20:12

avoid adding zero sized buffer to TRITONSERVER_InferenceRequestAddInput

edb8aef

whoisj force-pushed the jwyman/tri-673 branch from 77b3f26 to edb8aef Compare March 20, 2026 22:29

whoisj marked this pull request as ready for review March 23, 2026 16:08

whoisj force-pushed the jwyman/tri-673 branch from 68df00c to 6df85d5 Compare March 23, 2026 16:09

whoisj requested a review from yinggeh March 23, 2026 16:09

fix test

98eabf9

whoisj force-pushed the jwyman/tri-673 branch 2 times, most recently from d864e63 to fcaabcf Compare March 23, 2026 20:08

print error when error

3d21dc1

whoisj force-pushed the jwyman/tri-673 branch from fcaabcf to 3d21dc1 Compare March 23, 2026 20:17

revert some error message changes

defd24a

whoisj force-pushed the jwyman/tri-673 branch from cec7b6e to b183ae7 Compare March 23, 2026 21:29

fix test match

5efc58b

whoisj force-pushed the jwyman/tri-673 branch from b183ae7 to 5efc58b Compare March 23, 2026 21:39

make pre-commit happy

8428658

whoisj requested a review from mudit-eng March 23, 2026 21:43

mudit-eng reviewed Mar 24, 2026

View reviewed changes

react to @mudit-eng comments

4b99ac0

whoisj requested a review from mudit-eng March 25, 2026 16:45

fixup error messages

148a4dd

update tests to match message changes

yinggeh reviewed Apr 3, 2026

View reviewed changes

whoisj added 2 commits April 3, 2026 17:06

rename tests

a329b10

make pre-commit happy

be97bed

yinggeh approved these changes Apr 6, 2026

View reviewed changes

whoisj merged commit e5aecb3 into main Apr 6, 2026
3 checks passed

whoisj deleted the jwyman/tri-673 branch April 6, 2026 22:20

	def test_type_size_explosion(self):
	def test_json_dtype_size_expansion_exceeds_limit_error(self):

Uh oh!

Conversation

whoisj commented Feb 24, 2026

Uh oh!

mudit-eng commented Feb 26, 2026

Uh oh!

mattwittwer commented Feb 27, 2026

Uh oh!

yinggeh left a comment

Choose a reason for hiding this comment

Uh oh!

whoisj commented Feb 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yinggeh commented Mar 4, 2026

Uh oh!

yinggeh commented Mar 4, 2026

Uh oh!

Uh oh!

whoisj commented Mar 6, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

whoisj commented Apr 3, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mudit-eng commented Apr 6, 2026