You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
To avoid JIT re-compilation latency when testing multiple prompts, enable `interactive=True`. The model weights remain loaded in TPU memory, and after the initial warmup pass, each new prompt generates in **~0.9s**:
704
+
To avoid JIT re-compilation latency when testing multiple prompts, enable `interactive=True`. The model weights remain loaded in TPU memory, and after the initial warmup pass, each new prompt generates in **<1s**:
0 commit comments