|
2 | 2 |
|
3 | 3 | [](https://github.com/inftyai/amrs/releases/latest) |
4 | 4 |
|
5 | | -The Adaptive Model Routing System (AMRS) is a framework designed to select the best-fit model for exploration and exploitation. (still under development) |
| 5 | +The Adaptive Model Routing System (AMRS) is a framework designed to select the best-fit model for exploration and exploitation. Rust core with python bindings. Still under active development 🚧. |
6 | 6 |
|
7 | | -Thanks to [async-openai](https://github.com/64bit/async-openai), AMRS builds on top of it to provide adaptive model routing capabilities. |
| 7 | +AMRS builds on top of [async-openai](https://github.com/64bit/async-openai) to provide API services for quick setup. Thanks to open source 💙. |
8 | 8 |
|
9 | 9 | ## Features |
10 | 10 |
|
11 | | -- Flexible routing strategies, including: |
12 | | - - **Random**: Randomly selects a model from the available models. |
| 11 | +- **Endpoints Support** (only basic ones because of limited resources): |
| 12 | + - Chat Completions |
| 13 | + - Responses |
| 14 | + - More on the way |
| 15 | + |
| 16 | +- **Flexible Routing Strategies**: |
| 17 | + - **Random(default)**: Randomly selects a model from the available models. |
13 | 18 | - **WRR**: Weighted Round Robin selects models based on predefined weights. |
14 | | - - **UCB1**: Upper Confidence Bound based model selection (coming soon). |
| 19 | + - **UCB1**: Upper Confidence Bound for balancing exploration and exploitation (coming soon). |
15 | 20 | - **Adaptive**: Dynamically selects models based on performance metrics (coming soon). |
16 | 21 |
|
17 | | -- Broad provider support: |
18 | | - - OpenAI compatible providers (DeepInfra, OpenRouter, etc.) |
| 22 | +- **Various Providers Support**: |
| 23 | + - OpenAI compatible providers (OpenAI, DeepInfra, etc.) |
19 | 24 | - More on the way |
20 | 25 |
|
21 | 26 | ## How to use |
22 | 27 |
|
23 | | -Here's a simple example with the Weighted Round Robin (WRR) routing mode: |
| 28 | +Here's a simple example with the Weighted Round Robin (WRR) routing mode. Before running the code, make sure to set your provider API key in the environment variable by running `export <PROVIDER>_API_KEY="your_openai_api_key"`. |
| 29 | +Here we use OpenAI as an example. |
24 | 30 |
|
25 | 31 |
|
26 | 32 | ```rust |
27 | | -// Before running the code, make sure to set your OpenAI API key in the environment variable: |
28 | | -// export OPENAI_API_KEY="your_openai_api_key" |
| 33 | +# Make sure OPENAI_API_KEY is set in your environment variables before running this code. |
29 | 34 |
|
30 | 35 | use tokio::runtime::Runtime; |
31 | 36 | use arms::client; |
@@ -55,12 +60,22 @@ let mut client = client::Client::new(config); |
55 | 60 | let request = chat::CreateChatCompletionRequestArgs::default() |
56 | 61 | .messages([ |
57 | 62 | chat::ChatCompletionRequestSystemMessage::from("You are a helpful assistant.").into(), |
58 | | - chat::ChatCompletionRequestUserMessage::from("How is the weather today?").into(), |
| 63 | + chat::ChatCompletionRequestUserMessage::from("Who won the FIFA World Cup in 2025?").into(), |
59 | 64 | ]) |
60 | 65 | .build() |
61 | 66 | .unwrap(); |
62 | 67 |
|
63 | 68 | let result = Runtime::new().unwrap().block_on(client.create_completion(request)); |
| 69 | +match result { |
| 70 | + Ok(response) => { |
| 71 | + for choice in response.choices { |
| 72 | + println!("Response: {:?}", choice.message.content); |
| 73 | + } |
| 74 | + } |
| 75 | + Err(e) => { |
| 76 | + eprintln!("Error: {}", e); |
| 77 | + } |
| 78 | +} |
64 | 79 | ``` |
65 | 80 |
|
66 | 81 | ## Contributing |
|
0 commit comments