Skip to content

Commit b8b6670

Browse files
committed
[None][fix] Guard changeBeamWidth on mInflightReqIds.empty() in error handler
In pipeline-parallel multi-micro-batch mode, requests from other micro-batches may still be tracked in mInflightReqIds after the error handler erases only the current activeRequests. changeBeamWidth asserts mInflightReqIds.empty(), so calling it unconditionally would throw (caught by the inner try-catch) and skip the intended buffer reset. Add an explicit guard so the reset is skipped when other micro-batches are still in-flight. The next successful forwardAsync iteration will perform the reset via the normal changeBeamWidth call in verifyRequests once the set is clear. This makes the skip explicit rather than relying on TLLM_CHECK as control flow. Signed-off-by: Aurelien Chartier <2567591+achartier@users.noreply.github.com>
1 parent 1cfdc82 commit b8b6670

1 file changed

Lines changed: 5 additions & 2 deletions

File tree

cpp/tensorrt_llm/batch_manager/trtGptModelInflightBatching.cpp

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1223,8 +1223,11 @@ void TrtGptModelInflightBatching::forwardAsync(RequestList const& activeRequests
12231223
}
12241224
// Force buffer/decoder reset to clean up any partial state from the aborted batch
12251225
// (e.g. partially-filled cross-KV block offsets from mid-context-chunk processing).
1226-
// This prevents subsequent requests from reusing stale RuntimeBuffers.
1227-
if (mWorldConfig.isLastPipelineParallelRank())
1226+
// Guard on mInflightReqIds.empty(): in pipeline-parallel multi-micro-batch mode,
1227+
// other micro-batches may still have requests tracked here; changeBeamWidth asserts
1228+
// emptiness so we skip the reset and let the next successful forwardAsync iteration
1229+
// perform it when the set is clear.
1230+
if (mWorldConfig.isLastPipelineParallelRank() && mInflightReqIds.empty())
12281231
{
12291232
changeBeamWidth(mOperatingBeamWidth);
12301233
}

0 commit comments

Comments
 (0)