Skip to content

Commit 19ae053

Browse files
[None][fix] AutoDeploy: set enable_spec_decode on ADEngine for disagg (#15260)
Signed-off-by: Shixiaowei02 <39303645+Shixiaowei02@users.noreply.github.com> Co-authored-by: gramnarayan <105831528+govind-ramnarayan@users.noreply.github.com>
1 parent 646464b commit 19ae053

1 file changed

Lines changed: 5 additions & 0 deletions

File tree

tensorrt_llm/_torch/auto_deploy/shim/ad_executor.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -494,6 +494,11 @@ def __init__(
494494
else:
495495
self.max_total_draft_tokens = 0
496496

497+
# ADEngine skips PyTorchModelEngine.__init__, so set the spec-decode
498+
# flags that shared PyExecutor code expects on a ModelEngine.
499+
self.is_spec_decode = self.spec_config is not None
500+
self.enable_spec_decode = self.is_spec_decode
501+
497502
# For compatibility with PyTorchModelEngine utilities
498503
self.batch_size = cache_seq_interface.info.max_batch_size
499504

0 commit comments

Comments
 (0)