fix: Ensemble requests stuck indefinitely when the step max_queue_size exceeded

[pskiran1]

This PR addresses a critical issue where ensemble requests could hang indefinitely when the step max_queue_size limits were exceeded. The fix ensures proper error handling and cleanup by allowing completed steps to be consumed even after errors occur, preventing in-flight counter deadlocks.

Changes:

• Modified PrepareSteps to always process completed steps for proper cleanup, even when the ensemble has failed
• Added error_response_sent_ flag to prevent duplicate error responses
• Enhanced error handling to preserve original error status while still cleaning up in-flight steps

CI: triton-inference-server/server#8631

Background:
max_queue_size worked correctly, and the request was properly rejected with an error status. However, the ensemble scheduler had a critical bug that prevented proper cleanup after rejection, causing the hang. When ensemble_status_ was updated to unhealthy, UpdateEnsembleState() calls were skipped, and responses were not consumed, so inflight_step_counter_ never reached zero, and the request remained stuck indefinitely.

https://github.com/triton-inference-server/core/pull/466/changes#diff-ce8a574cf83a8bc8293e1be3573c9fb2d84cc681a8f6c6a66503bd844570934eR969

https://github.com/triton-inference-server/core/pull/466/changes#diff-ce8a574cf83a8bc8293e1be3573c9fb2d84cc681a8f6c6a66503bd844570934eR1338

[Copilot]

Pull request overview

This PR addresses a critical issue where ensemble requests could hang indefinitely when step max_queue_size limits were exceeded. The fix ensures proper error handling and cleanup by allowing completed steps to be consumed even after errors occur, preventing inflight counter deadlocks.

Changes:

• Modified PrepareSteps to always process completed steps for proper cleanup, even when ensemble has failed
• Added error_response_sent_ flag to prevent duplicate error responses
• Enhanced error handling to preserve original error status while still cleaning up in-flight steps

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[whoisj]

I am slightly confused here. Does this fix the problem or just turn full queues into error conditions?

[yinggeh]

Test case?

[pskiran1]

Test case?

Please refer to PR triton-inference-server/server#8631

[yinggeh]

How could max_queue_size be exceeded?

[pskiran1]

How could max_queue_size be exceeded?

We can exceed the max_queue_size of a step if we create a situation where more requests are queued at that step than it can handle.
I reproduced this with a similar test triton-inference-server/server#8631: the first step is a decoupled model that produces many requests and has a short processing delay, while the second step is slower and configured with max_queue_size = 4.
When we send an inference request, step 1 quickly generates more requests than step 2 can consume in time. As a result, the intermediate requests queued for step 2 exceed the configured limit. Our changes ensure that, in this case, the excess requests are rejected gracefully instead of causing the system to hang.

[yinggeh]

max_queue_size

Why did it hang before instead of rejecting the request (which line)? Did max_queue_size completely not work at all before?

[whoisj]

max_queue_size

Why did it hang before instead of rejecting the request (which line)? Did max_queue_size completely not work at all before?

Very good question and something I would think we should be concerned more about.

[pskiran1]

max_queue_size

Why did it hang before instead of rejecting the request (which line)? Did max_queue_size completely not work at all before?

max_queue_size worked correctly, and the request was properly rejected with an error status. However, the ensemble scheduler had a critical bug that prevented proper cleanup after rejection, causing the hang. When ensemble_status_ was updated to unhealthy, UpdateEnsembleState() calls were skipped, and responses were not consumed, so inflight_step_counter_ never reached zero, and the request remained stuck indefinitely.

https://github.com/triton-inference-server/core/pull/466/changes#diff-ce8a574cf83a8bc8293e1be3573c9fb2d84cc681a8f6c6a66503bd844570934eR969

https://github.com/triton-inference-server/core/pull/466/changes#diff-ce8a574cf83a8bc8293e1be3573c9fb2d84cc681a8f6c6a66503bd844570934eR1338

[yinggeh]

max_queue_size

Why did it hang before instead of rejecting the request (which line)? Did max_queue_size completely not work at all before?

max_queue_size worked correctly, and the request was properly rejected with an error status. However, the ensemble scheduler had a critical bug that prevented proper cleanup after rejection, causing the hang. When ensemble_status_ was updated to unhealthy, UpdateEnsembleState() calls were skipped, and responses were not consumed, so inflight_step_counter_ never reached zero, and the request remained stuck indefinitely.

https://github.com/triton-inference-server/core/pull/466/changes#diff-ce8a574cf83a8bc8293e1be3573c9fb2d84cc681a8f6c6a66503bd844570934eR969

https://github.com/triton-inference-server/core/pull/466/changes#diff-ce8a574cf83a8bc8293e1be3573c9fb2d84cc681a8f6c6a66503bd844570934eR1338

Should add that to the PR description.

[yinggeh]

Left some comments. LGTM in general

[yinggeh]

Nice work!