Load/Unload model via API

[Jonseed]

It would be great if the API could load a model into vram upon receiving a request (if not already loaded), and unload a model from vram upon request (like keep_alive in Ollama). Having the model always loaded when Koboldcpp server is running is problematic when using ComfyUI workflows with other services, models, generations (image generation), that need vram. This might also allow model swapping/changing via API.

[pupphelper]

I also would like this. Both a timeout to automatically unload after a timeout, then reload on request. or API unload/load. Similar to ollama.

[LostRuins]

it should be coming soon

[LostRuins]

will be added in the upcoming version

[pupphelper]

Thanks so much for adding the feature. it properly times out and unloads the model, is there a setting to have it auto reload the last model used when it receives the next api query from silly tavern? Keep up the amazing work with this engine!

[LostRuins]

If you're using the Chat Completions mode in sillytavern, that should already happen. Just make sure the model selected matches the one you want to use.

[pupphelper]

thanks, I'll try that, have been using text completion for more control and DRY settings in kobold. Thanks again for your quick responses and excellent work!

[LostRuins]

it will work with text completions too. the model name just needs to be sent