Feature idea: More dynamic model server slots

[miloszkolber]

One thing I'm missing in llama.cpp is greater control over which models are loaded during requests. The services that I'm hosting locally prompt different models at different times, sometimes for brief moments.

For example, let's say we define a maximum of four models and we have 16 GB VRAM (or even less). This isn't an issue when four smaller models are loaded (0.6b embedding, 1b text, etc.). The problem arises when one larger model is performing some workload (for example, dense 27b at a smaller quant) and other services are requesting other models simultaneously.

It would be great to be able to assign a "weight" to each model – for example, the larger model could be assigned a weight of 3, meaning that only one other model with a weight of 1 could fit alongside it. To complement this approach, it would be good to assign a priority to the models, so that the larger model is not unloaded to fit the other two, but rather the other two wait until the smaller one has finished its work.

Not critical, but nice to have.

[wadealexc]

I think what you're looking for is similar to my proposal here: #25674

Your "weight" system sounds kinda like a policy engine to me. When a request comes in, it evaluates the resources required to serve the request, and attempts to free up enough resources by applying a series of configurable rules. For example, as you described, this engine might unload one model to serve the request of another - or, applying my proposal, we could offload just the kvcache of one model to serve another model's request, then replace that cache when the request completes.

Implementing a policy engine is probably a very opinionated take on model/request management -- my proposal is trying to describe the basic APIs/interfaces/features needed to support such an engine!

If this is something you need, please chime in on the discussion - it'll help signal boost a bit 😄