Skip to content

Latest commit

 

History

History
3 lines (2 loc) · 474 Bytes

File metadata and controls

3 lines (2 loc) · 474 Bytes

Online Serving Examples with trtllm-serve

We provide a CLI command, trtllm-serve, to launch a FastAPI server compatible with OpenAI APIs, here are some client examples to query the server, you can check the source code here or refer to the command documentation and examples for detailed information and usage guidelines.