Skip to content

Commit 55cdc0b

Browse files
asimurkaasimurka
committed
Added e2e tests for tool choices in responses endpoint
1 parent 09f03e9 commit 55cdc0b

4 files changed

Lines changed: 412 additions & 24 deletions

File tree

‎tests/e2e/features/responses.feature‎

Lines changed: 333 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ Feature: Responses endpoint API tests
55
Given The service is started locally
66
And REST API service prefix is /v1
77

8-
Scenario: Check if responses endpoint returns 200 for minimal request
8+
Scenario: Check if responses endpoint answers a minimal question
99
Given The system is in default state
1010
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
1111
When I use "responses" to ask question with authorization header
@@ -14,11 +14,341 @@ Feature: Responses endpoint API tests
1414
"""
1515
Then The status code of the response is 200
1616

17-
Scenario: Check if responses endpoint returns 200 for minimal streaming request
17+
Scenario: Check if responses endpoint streams a minimal answer
1818
Given The system is in default state
1919
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
2020
When I use "responses" to ask question with authorization header
2121
"""
2222
{"input": "Say hello", "model": "{PROVIDER}/{MODEL}", "stream": true}
2323
"""
24-
Then The status code of the response is 200
24+
Then The status code of the response is 200
25+
26+
Scenario: Check if responses endpoint with tool_choice none answers knowledge question without file search usage
27+
Given The system is in default state
28+
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
29+
And I capture the current token metrics
30+
When I use "responses" to ask question with authorization header
31+
"""
32+
{
33+
"input": "What is the title of the article from Paul?",
34+
"model": "{PROVIDER}/{MODEL}",
35+
"stream": false,
36+
"instructions": "You are an assistant. Always use the file_search tool to answer. Write only lowercase letters",
37+
"tools": [{"type": "file_search", "vector_store_ids": ["{VECTOR_STORE_ID}"]}],
38+
"tool_choice": "none"
39+
}
40+
"""
41+
Then The status code of the response is 200
42+
And The responses output should not include any tool invocation item types
43+
And The token metrics should have increased
44+
45+
Scenario: Check if responses endpoint with tool_choice auto answers a knowledge question using file search
46+
Given The system is in default state
47+
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
48+
And I capture the current token metrics
49+
When I use "responses" to ask question with authorization header
50+
"""
51+
{
52+
"input": "What is the title of the article from Paul?",
53+
"model": "{PROVIDER}/{MODEL}",
54+
"stream": false,
55+
"instructions": "You are an assistant. Always use the file_search tool to answer. Write only lowercase letters",
56+
"tools": [{"type": "file_search", "vector_store_ids": ["{VECTOR_STORE_ID}"]}],
57+
"tool_choice": "auto"
58+
}
59+
"""
60+
Then The status code of the response is 200
61+
And The responses output should include an item with type "file_search_call"
62+
And The responses output_text should contain following fragments
63+
| Fragments in LLM response |
64+
| great work |
65+
And The token metrics should have increased
66+
67+
Scenario: Check if responses endpoint with tool_choice required still invokes document search for a basic question
68+
Given The system is in default state
69+
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
70+
And I capture the current token metrics
71+
When I use "responses" to ask question with authorization header
72+
"""
73+
{
74+
"input": "Hello World!",
75+
"model": "{PROVIDER}/{MODEL}",
76+
"stream": false,
77+
"tools": [{"type": "file_search", "vector_store_ids": ["{VECTOR_STORE_ID}"]}],
78+
"tool_choice": "required"
79+
}
80+
"""
81+
Then The status code of the response is 200
82+
And The responses output should include an item with type "file_search_call"
83+
And The token metrics should have increased
84+
85+
Scenario: Check if responses endpoint with file search as the chosen tool answers using file search
86+
Given The system is in default state
87+
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
88+
And I capture the current token metrics
89+
When I use "responses" to ask question with authorization header
90+
"""
91+
{
92+
"input": "What is the title of the article from Paul?",
93+
"model": "{PROVIDER}/{MODEL}",
94+
"stream": false,
95+
"instructions": "You are an assistant. Always use the file_search tool to answer. Write only lowercase letters",
96+
"tools": [{"type": "file_search", "vector_store_ids": ["{VECTOR_STORE_ID}"]}],
97+
"tool_choice": {"type": "file_search"}
98+
}
99+
"""
100+
Then The status code of the response is 200
101+
And The responses output should include an item with type "file_search_call"
102+
And The responses output_text should contain following fragments
103+
| Fragments in LLM response |
104+
| great work |
105+
And The token metrics should have increased
106+
107+
Scenario: Check if responses endpoint with allowed tools in automatic mode answers using file search
108+
Given The system is in default state
109+
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
110+
And I capture the current token metrics
111+
When I use "responses" to ask question with authorization header
112+
"""
113+
{
114+
"input": "What is the title of the article from Paul?",
115+
"model": "{PROVIDER}/{MODEL}",
116+
"stream": false,
117+
"instructions": "You are an assistant. Always use the file_search tool to answer. Write only lowercase letters",
118+
"tools": [{"type": "file_search", "vector_store_ids": ["{VECTOR_STORE_ID}"]}],
119+
"tool_choice": {
120+
"type": "allowed_tools",
121+
"mode": "auto",
122+
"tools": [{"type": "file_search"}]
123+
}
124+
}
125+
"""
126+
Then The status code of the response is 200
127+
And The responses output should include an item with type "file_search_call"
128+
And The responses output_text should contain following fragments
129+
| Fragments in LLM response |
130+
| great work |
131+
And The token metrics should have increased
132+
133+
Scenario: Check if responses endpoint with allowed tools in required mode invokes file search for a basic question
134+
Given The system is in default state
135+
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
136+
And I capture the current token metrics
137+
When I use "responses" to ask question with authorization header
138+
"""
139+
{
140+
"input": "Hello World!",
141+
"model": "{PROVIDER}/{MODEL}",
142+
"stream": false,
143+
"tools": [{"type": "file_search", "vector_store_ids": ["{VECTOR_STORE_ID}"]}],
144+
"tool_choice": {
145+
"type": "allowed_tools",
146+
"mode": "required",
147+
"tools": [{"type": "file_search"}]
148+
}
149+
}
150+
"""
151+
Then The status code of the response is 200
152+
And The responses output should include an item with type "file_search_call"
153+
And The token metrics should have increased
154+
155+
@MCPClientAuthConfig
156+
Scenario: Check if responses endpoint with tool_choice none answers without tool usage even with explicit instructions
157+
Given The system is in default state
158+
And I set the "MCP-HEADERS" header to
159+
"""
160+
{"mcp-client": {"Authorization": "Bearer client-test-token"}}
161+
"""
162+
And I capture the current token metrics
163+
When I use "responses" to ask question with authorization header
164+
"""
165+
{
166+
"input": "Hello World!",
167+
"model": "{PROVIDER}/{MODEL}",
168+
"stream": false,
169+
"instructions": "You are an assistant. Always use the mcp tool to answer. Write only lowercase letters",
170+
"tools": [
171+
{
172+
"type": "mcp",
173+
"server_label": "mcp-client",
174+
"server_url": "http://mock-mcp:3001",
175+
"require_approval": "never"
176+
}
177+
],
178+
"tool_choice": "none"
179+
}
180+
"""
181+
Then The status code of the response is 200
182+
And The responses output should not include any tool invocation item types
183+
And The token metrics should have increased
184+
185+
@MCPClientAuthConfig
186+
Scenario: Check if responses endpoint with tool_choice auto answers using mcp without explicit instructions
187+
Given The system is in default state
188+
And I set the "MCP-HEADERS" header to
189+
"""
190+
{"mcp-client": {"Authorization": "Bearer client-test-token"}}
191+
"""
192+
And I capture the current token metrics
193+
When I use "responses" to ask question with authorization header
194+
"""
195+
{
196+
"input": "Hello World!",
197+
"model": "{PROVIDER}/{MODEL}",
198+
"stream": false,
199+
"instructions": "You are an assistant. Always use the mcp tool to answer. Write only lowercase letters",
200+
"tools": [
201+
{
202+
"type": "mcp",
203+
"server_label": "mcp-client",
204+
"server_url": "http://mock-mcp:3001",
205+
"require_approval": "never"
206+
}
207+
],
208+
"tool_choice": "auto"
209+
}
210+
"""
211+
Then The status code of the response is 200
212+
And The responses output should include an item with one of these types
213+
| item type |
214+
| mcp_call |
215+
| mcp_list_tools |
216+
And The token metrics should have increased
217+
218+
@MCPClientAuthConfig
219+
Scenario: Check if responses endpoint with tool_choice required still invokes mcp without explicit instructions
220+
Given The system is in default state
221+
And I set the "MCP-HEADERS" header to
222+
"""
223+
{"mcp-client": {"Authorization": "Bearer client-test-token"}}
224+
"""
225+
And I capture the current token metrics
226+
When I use "responses" to ask question with authorization header
227+
"""
228+
{
229+
"input": "Hello World!",
230+
"model": "{PROVIDER}/{MODEL}",
231+
"stream": false,
232+
"tools": [
233+
{
234+
"type": "mcp",
235+
"server_label": "mcp-client",
236+
"server_url": "http://mock-mcp:3001",
237+
"require_approval": "never"
238+
}
239+
],
240+
"tool_choice": "required"
241+
}
242+
"""
243+
Then The status code of the response is 200
244+
And The responses output should include an item with one of these types
245+
| item type |
246+
| mcp_call |
247+
| mcp_list_tools |
248+
And The token metrics should have increased
249+
250+
@MCPClientAuthConfig
251+
Scenario: Check if responses endpoint with MCP as the chosen tool answers using extension tools
252+
Given The system is in default state
253+
And I set the "MCP-HEADERS" header to
254+
"""
255+
{"mcp-client": {"Authorization": "Bearer client-test-token"}}
256+
"""
257+
And I capture the current token metrics
258+
When I use "responses" to ask question with authorization header
259+
"""
260+
{
261+
"input": "Hello World!",
262+
"model": "{PROVIDER}/{MODEL}",
263+
"stream": false,
264+
"instructions": "You are an assistant. Always use the mcp tool to answer. Write only lowercase letters",
265+
"tools": [
266+
{
267+
"type": "mcp",
268+
"server_label": "mcp-client",
269+
"server_url": "http://mock-mcp:3001",
270+
"require_approval": "never"
271+
}
272+
],
273+
"tool_choice": {"type": "mcp", "server_label": "mcp-client"}
274+
}
275+
"""
276+
Then The status code of the response is 200
277+
And The responses output should include an item with one of these types
278+
| item type |
279+
| mcp_call |
280+
| mcp_list_tools |
281+
And The token metrics should have increased
282+
283+
@MCPClientAuthConfig
284+
Scenario: Check if responses endpoint with allowed tools in automatic mode answers using extension tools
285+
Given The system is in default state
286+
And I set the "MCP-HEADERS" header to
287+
"""
288+
{"mcp-client": {"Authorization": "Bearer client-test-token"}}
289+
"""
290+
And I capture the current token metrics
291+
When I use "responses" to ask question with authorization header
292+
"""
293+
{
294+
"input": "Hello World!",
295+
"model": "{PROVIDER}/{MODEL}",
296+
"stream": false,
297+
"instructions": "You are an assistant. Always use the mcp tool to answer. Write only lowercase letters",
298+
"tools": [
299+
{
300+
"type": "mcp",
301+
"server_label": "mcp-client",
302+
"server_url": "http://mock-mcp:3001",
303+
"require_approval": "never"
304+
}
305+
],
306+
"tool_choice": {
307+
"type": "allowed_tools",
308+
"mode": "auto",
309+
"tools": [{"type": "mcp"}]
310+
}
311+
}
312+
"""
313+
Then The status code of the response is 200
314+
And The responses output should include an item with one of these types
315+
| item type |
316+
| mcp_call |
317+
| mcp_list_tools |
318+
And The token metrics should have increased
319+
320+
@MCPClientAuthConfig
321+
Scenario: Check if responses endpoint with allowed tools in required mode invokes mcp without explicit instructions
322+
Given The system is in default state
323+
And I set the "MCP-HEADERS" header to
324+
"""
325+
{"mcp-client": {"Authorization": "Bearer client-test-token"}}
326+
"""
327+
And I capture the current token metrics
328+
When I use "responses" to ask question with authorization header
329+
"""
330+
{
331+
"input": "Hello World!",
332+
"model": "{PROVIDER}/{MODEL}",
333+
"stream": false,
334+
"tools": [
335+
{
336+
"type": "mcp",
337+
"server_label": "mcp-client",
338+
"server_url": "http://mock-mcp:3001",
339+
"require_approval": "never"
340+
}
341+
],
342+
"tool_choice": {
343+
"type": "allowed_tools",
344+
"mode": "required",
345+
"tools": [{"type": "mcp"}]
346+
}
347+
}
348+
"""
349+
Then The status code of the response is 200
350+
And The responses output should include an item with one of these types
351+
| item type |
352+
| mcp_call |
353+
| mcp_list_tools |
354+
And The token metrics should have increased

‎tests/e2e/features/steps/common_http.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,7 @@ def check_status_code(context: Context, status: int) -> None:
127127
"""Check the HTTP status code for latest response from tested service."""
128128
assert context.response is not None, "Request needs to be performed first"
129129
if context.response.status_code != status:
130+
print("Output: ", context.response.json().get("output", []))
130131
# Include response body in error message for debugging
131132
try:
132133
error_body = context.response.json()

0 commit comments

Comments
 (0)