How to use prefix caching

[Noha-Magdy]

Hi
I can't find a way to turn on the prefix caching

When I run any model, I always get:
Using prefix caching = False

Thanks a lot

[danieldk]

Which model did you try? Note that prefix caching is not supported for VLMs. The hardware also needs to support flashinfer or flash-attention2, so prefix caching is not supported on GPUs with a CUDA capability that is lower than 8.0.