I saw you were interested in implementing this as a reverse proxy layer between a client and a LLM openai compatible backend, due to popular request on your Reddit post. Sessions would be managed through a user_id in each request. It would be awesome if this reverse proxy could bind to a port range instead of a single port, and each port would be tied to its own memory. This way, clients without the ability to add a user_id to each request could still work by selecting a unique port for each client instead of passing a user_id. That way, this reverse proxy could be implemented without requiring ANY client support whatsoever. Just point each client towards this proxy each using its own port and off you go. Thoughts?
I saw you were interested in implementing this as a reverse proxy layer between a client and a LLM openai compatible backend, due to popular request on your Reddit post. Sessions would be managed through a user_id in each request. It would be awesome if this reverse proxy could bind to a port range instead of a single port, and each port would be tied to its own memory. This way, clients without the ability to add a user_id to each request could still work by selecting a unique port for each client instead of passing a user_id. That way, this reverse proxy could be implemented without requiring ANY client support whatsoever. Just point each client towards this proxy each using its own port and off you go. Thoughts?