File tree Expand file tree Collapse file tree
tensorrt_llm/_torch/speculative Expand file tree Collapse file tree Original file line number Diff line number Diff line change @@ -522,17 +522,9 @@ def populate_sampling_params_for_one_model(
522522 """
523523 Set up topp/topk/temperatures for 1-model sampler.
524524
525- Two-phase implementation:
526- Phase 1: scan request sampling configs to compute skip_*/has_greedy
527- flags and decide whether the batch is all-greedy. No per-token
528- list construction.
529- Phase 2 (only when is_all_greedy_sample=False): build per-token /
530- per-request lists and issue 6 H->D copies into the sampling
531- parameter buffers.
532-
533- For all-greedy batches we skip phase 2 entirely (no list building, no
534- H->D copies). This is the common case when callers do not pass
535- temperature/top_k/top_p.
525+ Scans sampling configs to set skip_*/is_all_greedy_sample flags. When
526+ any request needs sampling, also builds per-token/per-request lists
527+ and copies them to GPU buffers; all-greedy batches skip this entirely.
536528 """
537529 from tensorrt_llm ._torch .pyexecutor .llm_request import LlmRequestState
538530 from tensorrt_llm .sampling_params import SamplingParams
You can’t perform that action at this time.
0 commit comments