You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
fix: disable GPU forcing/hijack on every token (1.48 t/s bottleneck)
The forcing + hijack + D2D copy + reallocation loop runs on every
single-token decode, adding ~600ms/token overhead. Kept as empty
if(do_cuda) block for future activation. Baseline hybrid path restored.
GPU infrastructure preserved and ready: force + hijack + D2D copy
+ get_tensor_async fallback all remain. Just needs graph reuse or
CUDA graph capture to amortize the per-token cost.
0 commit comments