Hi,
I am trying to run the evaluations using Qwen3-4B and Qwen3-32B models for a public dataset. I am using vLLM to host my models locally. I noticed that the vllm custom model instructions in the (legacy) docs is outdated as there is no AgentWrapper anymore. I also learnt that I can't be using the public-evaluations branch as it does not support vllm hosting.
I am now trying to write the inference evaluation script in the main branch. Is there an example or tutorial that can help me utilize locally hosted vllm models?
Let me know if you need further details!
Thanks!
Hi,
I am trying to run the evaluations using Qwen3-4B and Qwen3-32B models for a public dataset. I am using vLLM to host my models locally. I noticed that the vllm custom model instructions in the (legacy) docs is outdated as there is no AgentWrapper anymore. I also learnt that I can't be using the public-evaluations branch as it does not support vllm hosting.
I am now trying to write the inference evaluation script in the main branch. Is there an example or tutorial that can help me utilize locally hosted vllm models?
Let me know if you need further details!
Thanks!