Skip to content

Commit fa7996c

Browse files
chore: make the tool-selection eval tolerant of Gemini's search-first pick (#95)
* chore: I hate em dash * chore: make the eval more tolerent * Update test_tool_selection.py * fix typo
1 parent 46f5c53 commit fa7996c

1 file changed

Lines changed: 4 additions & 4 deletions

File tree

tests/eval/test_tool_selection.py

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -107,17 +107,17 @@ class ToolChoice(BaseModel):
107107
{
108108
"id": "ambiguous-add-reaction-releases",
109109
"prompt": "Add a :tada: reaction to the latest message in #releases",
110-
"accepted_tools": ["slack_add_reaction", "slack_read_channel"],
110+
"accepted_tools": ["slack_add_reaction", "slack_read_channel", "slack_search_public", "slack_search_channels"],
111111
},
112112
{
113113
"id": "ambiguous-reply-in-thread",
114114
"prompt": "Reply 'we're on it' in the thread on the outage message in #incidents",
115-
"accepted_tools": ["slack_send_message", "slack_read_thread"],
115+
"accepted_tools": ["slack_send_message", "slack_read_thread", "slack_search_public", "slack_search_channels"],
116116
},
117117
{
118118
"id": "ambiguous-read-thread-replies",
119119
"prompt": "Show me all the replies in the thread on the latest message in #support",
120-
"accepted_tools": ["slack_read_thread", "slack_read_channel"],
120+
"accepted_tools": ["slack_read_thread", "slack_read_channel", "slack_search_public", "slack_search_channels"],
121121
},
122122
{
123123
"id": "ambiguous-lookup-user-by-email",
@@ -177,7 +177,7 @@ def build_prompt(available_tools: list[ToolCall], prompt: str) -> str:
177177
User request: {prompt}
178178
179179
Pick the single tool that performs the action the user is asking for. Any channel name,
180-
channel ID, or user ID already in the request is usable as-is — do not pick a search tool
180+
channel ID, or user ID already in the request is usable as-is. Do not pick a search tool
181181
just to resolve it into an ID first. Respond with the tool's exact name."""
182182

183183

0 commit comments

Comments
 (0)