Skip to content

Commit 9b21300

Browse files
jrobertboosjrobertboos
authored
LCORE-1247: E2E Tests for MCP OAuth (#1177)
* Refactor MCP mock server configuration and update tests for OAuth support - Removed the old `mcp-mock-server` service and replaced it with `mock-mcp` in both `docker-compose` files. - Updated health check and dependencies to reflect the new service name. - Modified E2E test configurations to use the new `mock-mcp` service URL. - Added a minimal mock MCP server implementation with OAuth support for testing. - Updated feature tests to check for OAuth authentication requirements and response headers. * Refactor MCP mock server configuration and update tests for OAuth support - Removed the old `mcp-mock-server` service and replaced it with `mock-mcp` in both `docker-compose` files. - Updated health check and dependencies to reflect the new service name. - Modified E2E test configurations to use the new `mock-mcp` service URL. - Added a minimal mock MCP server implementation with OAuth support for testing. - Updated feature tests to check for OAuth authentication requirements and response headers. * Remove backup configuration file and update llama-stack service URL in E2E tests * fixed /tools in library mode and added e2e tests for query and streaming_query * Update E2E tests to standardize header checks - Modified feature files to change the wording for checking response headers from "contains the following" to "contains the following header". - Updated the corresponding step definition in common_http.py to reflect the new header check format. * Placed MCP tests into new feature - Introduced new feature tests for MCP authentication scenarios, including checks for tools, query, and streaming_query endpoints. - Added MCP configuration files for both library and server modes. - Updated the environment setup to switch configurations based on feature tags. - Removed outdated authentication checks from existing feature files to streamline tests. * Restored other mock mcp server * Fixed already allocated port * Fixed allocated port * Fixed `tools/list` for mock server and addressed coderabbit - Also removed authentication from `mcp.feature` * fixed black * Add new end-to-end tests for MCP authentication scenarios - Implemented scenarios to verify successful and error responses for the 'tools', 'query', and 'streaming_query' endpoints when valid and invalid MCP auth tokens are provided. - Enhanced the common HTTP step definitions to support setting headers dynamically. - Updated the mock MCP server to handle invalid tokens appropriately. * fixed black and ruff * addresed comments * addressed comments * fixed merge error
1 parent 3b00e40 commit 9b21300

11 files changed

Lines changed: 455 additions & 3 deletions

File tree

‎docker-compose-library.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@ services:
2828
depends_on:
2929
mcp-mock-server:
3030
condition: service_healthy
31+
mock-mcp:
32+
condition: service_healthy
3133
networks:
3234
- lightspeednet
3335
volumes:
@@ -93,6 +95,23 @@ services:
9395
retries: 3
9496
start_period: 2s
9597

98+
mock-mcp:
99+
build:
100+
context: ./tests/e2e/mock_mcp_server
101+
dockerfile: Dockerfile
102+
container_name: mock-mcp
103+
ports:
104+
- "3001:3001"
105+
networks:
106+
- lightspeednet
107+
healthcheck:
108+
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3001/health')"]
109+
interval: 5s
110+
timeout: 3s
111+
retries: 3
112+
start_period: 2s
113+
114+
96115
networks:
97116
lightspeednet:
98117
driver: bridge

‎docker-compose.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,8 @@ services:
9292
condition: service_healthy
9393
mcp-mock-server:
9494
condition: service_healthy
95+
mock-mcp:
96+
condition: service_healthy
9597
networks:
9698
- lightspeednet
9799
healthcheck:
@@ -118,6 +120,23 @@ services:
118120
retries: 3
119121
start_period: 2s
120122

123+
mock-mcp:
124+
build:
125+
context: ./tests/e2e/mock_mcp_server
126+
dockerfile: Dockerfile
127+
container_name: mock-mcp
128+
ports:
129+
- "3001:3001"
130+
networks:
131+
- lightspeednet
132+
healthcheck:
133+
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3001/health')"]
134+
interval: 5s
135+
timeout: 3s
136+
retries: 3
137+
start_period: 2s
138+
139+
121140
volumes:
122141
llama-storage:
123142

‎src/app/endpoints/tools.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44

55
from fastapi import APIRouter, Depends, HTTPException, Request
66
from llama_stack_client import APIConnectionError, BadRequestError, AuthenticationError
7+
from llama_stack.core.datatypes import AuthenticationRequiredError
78

89
from authentication import get_auth_dependency
910
from authentication.interface import AuthTuple
@@ -90,8 +91,7 @@ async def tools_endpoint_handler( # pylint: disable=too-many-locals,too-many-st
9091
except BadRequestError:
9192
logger.error("Toolgroup %s is not found", toolgroup.identifier)
9293
continue
93-
except AuthenticationError as e:
94-
logger.error("Authentication error: %s", e)
94+
except (AuthenticationError, AuthenticationRequiredError) as e:
9595
if toolgroup.mcp_endpoint:
9696
await probe_mcp_oauth_and_raise_401(
9797
toolgroup.mcp_endpoint.uri, chain_from=e

‎tests/e2e/configuration/library-mode/lightspeed-stack-mcp.yaml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
name: Lightspeed Core Service (LCS)
2+
service:
3+
host: 0.0.0.0
4+
port: 8080
5+
auth_enabled: false
6+
workers: 1
7+
color_log: true
8+
access_log: true
9+
llama_stack:
10+
# Library mode - embeds llama-stack as library
11+
use_as_library_client: true
12+
library_client_config_path: run.yaml
13+
user_data_collection:
14+
feedback_enabled: true
15+
feedback_storage: "/tmp/data/feedback"
16+
transcripts_enabled: true
17+
transcripts_storage: "/tmp/data/transcripts"
18+
authentication:
19+
module: "noop"
20+
mcp_servers:
21+
- name: "mcp-oauth"
22+
provider_id: "model-context-protocol"
23+
url: "http://mock-mcp:3001"
24+
authorization_headers:
25+
Authorization: "oauth"

‎tests/e2e/configuration/server-mode/lightspeed-stack-mcp.yaml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
name: Lightspeed Core Service (LCS)
2+
service:
3+
host: 0.0.0.0
4+
port: 8080
5+
auth_enabled: false
6+
workers: 1
7+
color_log: true
8+
access_log: true
9+
llama_stack:
10+
# Server mode - connects to separate llama-stack service
11+
use_as_library_client: false
12+
url: http://llama-stack:8321
13+
api_key: xyzzy
14+
user_data_collection:
15+
feedback_enabled: true
16+
feedback_storage: "/tmp/data/feedback"
17+
transcripts_enabled: true
18+
transcripts_storage: "/tmp/data/transcripts"
19+
authentication:
20+
module: "noop"
21+
mcp_servers:
22+
- name: "mcp-oauth"
23+
provider_id: "model-context-protocol"
24+
url: "http://mock-mcp:3001"
25+
authorization_headers:
26+
Authorization: "oauth"

‎tests/e2e/features/environment.py‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -297,6 +297,15 @@ def before_feature(context: Context, feature: Feature) -> None:
297297
context.port = os.getenv("E2E_LSC_PORT", "8080")
298298
context.feedback_conversations = []
299299

300+
if "MCP" in feature.tags:
301+
mode_dir = "library-mode" if context.is_library_mode else "server-mode"
302+
context.feature_config = (
303+
f"tests/e2e/configuration/{mode_dir}/lightspeed-stack-mcp.yaml"
304+
)
305+
context.default_config_backup = create_config_backup("lightspeed-stack.yaml")
306+
switch_config(context.feature_config)
307+
restart_container("lightspeed-stack")
308+
300309

301310
def after_feature(context: Context, feature: Feature) -> None:
302311
"""Run after each feature file is exercised.
@@ -324,3 +333,8 @@ def after_feature(context: Context, feature: Feature) -> None:
324333
url = f"http://{context.hostname}:{context.port}/v1/conversations/{conversation_id}"
325334
response = requests.delete(url, timeout=10)
326335
assert response.status_code == 200, f"{url} returned {response.status_code}"
336+
337+
if "MCP" in feature.tags:
338+
switch_config(context.default_config_backup)
339+
restart_container("lightspeed-stack")
340+
remove_config_backup(context.default_config_backup)

‎tests/e2e/features/mcp.feature‎

Lines changed: 204 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,204 @@
1+
@MCP
2+
Feature: MCP tests
3+
4+
Background:
5+
Given The service is started locally
6+
And REST API service prefix is /v1
7+
8+
Scenario: Check if tools endpoint reports error when MCP requires authentication
9+
Given The system is in default state
10+
When I access REST API endpoint "tools" using HTTP GET method
11+
Then The status code of the response is 401
12+
And The body of the response is the following
13+
"""
14+
{
15+
"detail": {
16+
"response": "Missing or invalid credentials provided by client",
17+
"cause": "MCP server at http://mock-mcp:3001 requires OAuth"
18+
}
19+
}
20+
"""
21+
And The headers of the response contains the following header "www-authenticate"
22+
23+
Scenario: Check if query endpoint reports error when MCP requires authentication
24+
Given The system is in default state
25+
When I use "query" to ask question
26+
"""
27+
{"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"}
28+
"""
29+
Then The status code of the response is 401
30+
And The body of the response is the following
31+
"""
32+
{
33+
"detail": {
34+
"response": "Missing or invalid credentials provided by client",
35+
"cause": "MCP server at http://mock-mcp:3001 requires OAuth"
36+
}
37+
}
38+
"""
39+
And The headers of the response contains the following header "www-authenticate"
40+
41+
Scenario: Check if streaming_query endpoint reports error when MCP requires authentication
42+
Given The system is in default state
43+
When I use "streaming_query" to ask question
44+
"""
45+
{"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"}
46+
"""
47+
Then The status code of the response is 401
48+
And The body of the response is the following
49+
"""
50+
{
51+
"detail": {
52+
"response": "Missing or invalid credentials provided by client",
53+
"cause": "MCP server at http://mock-mcp:3001 requires OAuth"
54+
}
55+
}
56+
"""
57+
And The headers of the response contains the following header "www-authenticate"
58+
59+
@skip # will be fixed in LCORE-1368
60+
Scenario: Check if tools endpoint succeeds when MCP auth token is passed
61+
Given The system is in default state
62+
And I set the "MCP-HEADERS" header to
63+
"""
64+
{"mcp-oauth": {"Authorization": "Bearer test-token"}}
65+
"""
66+
When I access REST API endpoint "tools" using HTTP GET method
67+
Then The status code of the response is 200
68+
And The body of the response is the following
69+
"""
70+
{
71+
"tools": [
72+
{
73+
"identifier": "",
74+
"description": "Insert documents into memory",
75+
"parameters": [],
76+
"provider_id": "",
77+
"toolgroup_id": "builtin::rag",
78+
"server_source": "builtin",
79+
"type": ""
80+
},
81+
{
82+
"identifier": "",
83+
"description": "Search for information in a database.",
84+
"parameters": [],
85+
"provider_id": "",
86+
"toolgroup_id": "builtin::rag",
87+
"server_source": "builtin",
88+
"type": ""
89+
},
90+
{
91+
"identifier": "",
92+
"description": "Mock tool for E2E",
93+
"parameters": [],
94+
"provider_id": "",
95+
"toolgroup_id": "mcp-oauth",
96+
"server_source": "http://localhost:3001",
97+
"type": ""
98+
}
99+
]
100+
}
101+
"""
102+
103+
@skip # will be fixed in LCORE-1366
104+
Scenario: Check if query endpoint succeeds when MCP auth token is passed
105+
Given The system is in default state
106+
And I set the "MCP-HEADERS" header to
107+
"""
108+
{"mcp-oauth": {"Authorization": "Bearer test-token"}}
109+
"""
110+
And I capture the current token metrics
111+
When I use "query" to ask question with authorization header
112+
"""
113+
{"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"}
114+
"""
115+
Then The status code of the response is 200
116+
And The response should contain following fragments
117+
| Fragments in LLM response |
118+
| hello |
119+
And The token metrics should have increased
120+
121+
@skip # will be fixed in LCORE-1366
122+
Scenario: Check if streaming_query endpoint succeeds when MCP auth token is passed
123+
Given The system is in default state
124+
And I set the "MCP-HEADERS" header to
125+
"""
126+
{"mcp-oauth": {"Authorization": "Bearer test-token"}}
127+
"""
128+
And I capture the current token metrics
129+
When I use "streaming_query" to ask question with authorization header
130+
"""
131+
{"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"}
132+
"""
133+
When I wait for the response to be completed
134+
Then The status code of the response is 200
135+
And The streamed response should contain following fragments
136+
| Fragments in LLM response |
137+
| hello |
138+
And The token metrics should have increased
139+
140+
@skip # will be fixed in LCORE-1368
141+
Scenario: Check if tools endpoint reports error when MCP invalid auth token is passed
142+
Given The system is in default state
143+
And I set the "MCP-HEADERS" header to
144+
"""
145+
{"mcp-oauth": {"Authorization": "Bearer invalid-token"}}
146+
"""
147+
When I access REST API endpoint "tools" using HTTP GET method
148+
Then The status code of the response is 401
149+
And The body of the response is the following
150+
"""
151+
{
152+
"detail": {
153+
"response": "Missing or invalid credentials provided by client",
154+
"cause": "MCP server at http://mock-mcp:3001 requires OAuth"
155+
}
156+
}
157+
"""
158+
And The headers of the response contains the following header "www-authenticate"
159+
160+
@skip # will be fixed in LCORE-1366
161+
Scenario: Check if query endpoint reports error when MCP invalid auth token is passed
162+
Given The system is in default state
163+
And I set the "MCP-HEADERS" header to
164+
"""
165+
{"mcp-oauth": {"Authorization": "Bearer invalid-token"}}
166+
"""
167+
When I use "query" to ask question with authorization header
168+
"""
169+
{"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"}
170+
"""
171+
Then The status code of the response is 401
172+
And The body of the response is the following
173+
"""
174+
{
175+
"detail": {
176+
"response": "Missing or invalid credentials provided by client",
177+
"cause": "MCP server at http://mock-mcp:3001 requires OAuth"
178+
}
179+
}
180+
"""
181+
And The headers of the response contains the following header "www-authenticate"
182+
183+
@skip # will be fixed in LCORE-1366
184+
Scenario: Check if streaming_query endpoint reports error when MCP invalid auth token is passed
185+
Given The system is in default state
186+
And I set the "MCP-HEADERS" header to
187+
"""
188+
{"mcp-oauth": {"Authorization": "Bearer invalid-token"}}
189+
"""
190+
When I use "streaming_query" to ask question with authorization header
191+
"""
192+
{"query": "Say hello", "model": "{MODEL}", "provider": "{PROVIDER}"}
193+
"""
194+
Then The status code of the response is 401
195+
And The body of the response is the following
196+
"""
197+
{
198+
"detail": {
199+
"response": "Missing or invalid credentials provided by client",
200+
"cause": "MCP server at http://mock-mcp:3001 requires OAuth"
201+
}
202+
}
203+
"""
204+
And The headers of the response contains the following header "www-authenticate"

0 commit comments

Comments
 (0)