[INFERENCE PROVIDERS] function calling tutorial#1832
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Wauplin
left a comment
Wauplin
left a comment
There was a problem hiding this comment.
Really cool guide, learnt a few things along the way ^^
|
|
||
| # Initialize client | ||
| client = OpenAI( | ||
| base_url="https://router.huggingface.co/nebius/v1", |
There was a problem hiding this comment.
This is slightly misleading since it won't work the same for all providers. Using URL "https://router.huggingface.co/v1" is more flexible and should work for all providers (provider selection when using this route is on its way server-side)
There was a problem hiding this comment.
Got it thanks. Agreed that the router makes sense here.
| - base_url="https://router.huggingface.co/together/v1", | ||
| + base_url="https://router.huggingface.co/nebius/v1", |
There was a problem hiding this comment.
Here is typically where using "https://router.huggingface.co/v1" would be better for server-side model resolution.
If you look at
- https://huggingface.co/deepseek-ai/DeepSeek-R1-0528?inference_api=true&language=python&inference_provider=together
- vs https://huggingface.co/deepseek-ai/DeepSeek-R1-0528?inference_api=true&language=python&inference_provider=nebius
you'll see the model id must be "deepseek-ai/DeepSeek-R1" for Together vs "deepseek-ai/DeepSeek-R1-0528" for Nebius. Once we have a proper "auto route with provider selection", this should hopefully be much simpler to use with the openai client.
There was a problem hiding this comment.
(Using InferenceClient it's much simpler since all the resolution is done client-side for you. We are currently closing the gap between the two clients)
There was a problem hiding this comment.
Here is typically where using "https://router.huggingface.co/v1" would be better for server-side model resolution.
Totally agree on this, but my main effort is to show the power of provider selection. I think this commit solves it, where I show a working example with provider="auto" and the router but diff trace for other providers.
pcuenca
left a comment
pcuenca
left a comment
There was a problem hiding this comment.
Very nice!
cc @sergiopaniego, we worked together on this notebook for tool calling with Llama (but most info should be general).
I also enjoyed @Rocketknight1's post, in particular the get_json_schema helper. Could we maybe leverage it here?
|
|
||
| # Get final response with function results | ||
| final_response = client.chat.completions.create( | ||
| model="meta-llama/Llama-3.1-8B-Instruct", |
There was a problem hiding this comment.
Why do we use a different model here? (Not saying it's wrong, just pointing out it could be potentially puzzling)
There was a problem hiding this comment.
Yeah. This is wrong.
Co-authored-by: Sergio Paniego Blanco <sergiopaniegoblanco@gmail.com> Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
…ttps://github.com/huggingface/hub-docs into function-calling-tutorial-for-inference-providers
|
@Wauplin @pcuenca @sergiopaniego Thanks for the reviews! I've responded to everything. Please could I get an approval? |
|
Nice, some small assorted tweaks in #1837 |
6 participants
This PR introduces a new tutorial on function calling.