Skip to content

Commit 8150a5f

Browse files
committed
LCORE-1860: Add E2E tests for degraded mode startup scenarios
Add end-to-end tests to verify LCORE behavior when starting without llama-stack and allow_degraded_mode is enabled.
1 parent ab86328 commit 8150a5f

3 files changed

Lines changed: 78 additions & 0 deletions

File tree

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
name: Lightspeed Core Service (LCS) - Degraded Mode Test
2+
service:
3+
host: 0.0.0.0
4+
port: 8080
5+
auth_enabled: false
6+
workers: 1
7+
color_log: true
8+
access_log: true
9+
llama_stack:
10+
# Server mode - connects to separate llama-stack service
11+
use_as_library_client: false
12+
url: http://${env.E2E_LLAMA_HOSTNAME}:8321
13+
api_key: xyzzy
14+
# Enable degraded mode to allow startup without llama-stack
15+
allow_degraded_mode: true
16+
user_data_collection:
17+
feedback_enabled: true
18+
feedback_storage: "/tmp/data/feedback"
19+
transcripts_enabled: true
20+
transcripts_storage: "/tmp/data/transcripts"
21+
authentication:
22+
module: "noop"
23+
inference:
24+
default_provider: openai
25+
default_model: gpt-4o-mini
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
@e2e_group_3 @skip-in-library-mode @Authorized
2+
Feature: Degraded mode startup
3+
4+
End-to-end scenarios that test LCORE startup behavior when llama-stack
5+
is NOT available at startup time and allow_degraded_mode is enabled.
6+
7+
These tests verify that LCORE metrics correctly reflect startup state
8+
in both healthy and degraded modes.
9+
10+
Background:
11+
Given The service is started locally
12+
And The system is in default state
13+
And REST API service prefix is /v1
14+
And the Lightspeed stack configuration directory is "tests/e2e/configuration"
15+
16+
Scenario: Degraded mode metric is set to 0.0 when started with llama-stack
17+
Given The service uses the lightspeed-stack-degraded-mode.yaml configuration
18+
And The service is restarted
19+
When I access endpoint "metrics" using HTTP GET method
20+
Then The status code of the response is 200
21+
And The response body contains "ls_started_in_degraded_mode 0.0"
22+
23+
Scenario: Degraded mode metric is set to 1.0 when started without llama-stack
24+
Given The llama-stack connection is disrupted
25+
And The service uses the lightspeed-stack-degraded-mode.yaml configuration
26+
And The service is restarted
27+
When I access endpoint "metrics" using HTTP GET method
28+
Then The status code of the response is 200
29+
And The response body contains "ls_started_in_degraded_mode 1.0"
30+
31+
Scenario: Readiness endpoint reports degraded state when started without llama-stack
32+
Given The llama-stack connection is disrupted
33+
And The service uses the lightspeed-stack-degraded-mode.yaml configuration
34+
And The service is restarted
35+
When I access endpoint "readiness" using HTTP GET method
36+
Then The status code of the response is 503
37+
And The body of the response, ignoring the "providers" field, is the following
38+
"""
39+
{"ready": false, "reason": "Cannot connect to backend service", "overall_status": "unhealthy", "impacts": ["LLM inference unavailable", "Provider health checks unavailable"]}
40+
"""

tests/e2e/features/steps/common_http.py

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -112,6 +112,19 @@ def check_response_body_does_not_contain(context: Context, substring: str) -> No
112112
), f"The response text '{context.response.text}' contains '{substring}'"
113113

114114

115+
@then('The response body contains "{substring}"')
116+
def check_response_contains_substring(context: Context, substring: str) -> None:
117+
"""Check that response body contains a specific substring.
118+
119+
This step handles quoted strings and performs exact matching
120+
(case-sensitive) for metrics and structured output validation.
121+
"""
122+
assert context.response is not None, "Request needs to be performed first"
123+
assert (
124+
substring in context.response.text
125+
), f"Expected '{substring}' in response, but got: {context.response.text[:200]}"
126+
127+
115128
@then("The body of the response is the following")
116129
def check_prediction_result(context: Context) -> None:
117130
"""Check the content of the response to be exactly the same.

0 commit comments

Comments
 (0)