Skip to content

Commit fb4f030

Browse files
tisniktisnik
authored
Merge pull request #1077 from tisnik/updated-e2e-scenarios
Updated e2e scenarios
2 parents d58d9bf + c38257e commit fb4f030

1 file changed

Lines changed: 158 additions & 5 deletions

File tree

‎docs/e2e_scenarios.md‎

Lines changed: 158 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,169 @@
11
# List of scenarios
22

3-
## [`smoketests.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/smoketests.feature)
3+
## [`authorized_noop.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/authorized_noop.feature)
44

5-
* Check if the main endpoint is reachable
5+
* Check if the authorized endpoint works fine when user_id and auth header are not provided
6+
* Check if the authorized endpoint works when auth token is not provided
7+
* Check if the authorized endpoint works when user_id is not provided
8+
* Check if the authorized endpoint works when providing empty user_id
9+
* Check if the authorized endpoint works when providing proper user_id
610

7-
## [`rest_api.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/rest_api.feature)
11+
## [`authorized_noop_token.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/authorized_noop_token.feature)
12+
13+
* Check if the authorized endpoint fails when user_id and auth header are not provided
14+
* Check if the authorized endpoint works when user_id is not provided
15+
* Check if the authorized endpoint works when providing empty user_id
16+
* Check if the authorized endpoint works when providing proper user_id
17+
* Check if the authorized endpoint works with proper user_id but bearer token is not present
18+
* Check if the authorized endpoint works when auth token is malformed
19+
20+
## [`authorized_rh_identity.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/authorized_rh_identity.feature)
21+
22+
* Request fails when x-rh-identity header is missing
23+
* Request fails when identity field is missing
24+
* Request succeeds with valid User identity and required entitlements
25+
* Request succeeds with valid System identity and required entitlements
26+
* Request fails when required entitlement is missing
27+
* Request fails when entitlement exists but is_entitled is false
28+
* Request fails when User identity is missing user_id
29+
* Request fails when User identity is missing username
30+
* Request fails when System identity is missing cn
31+
32+
## [`conversation_cache_v2.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/conversation_cache_v2.feature)
33+
34+
* V2 conversations endpoint WITHOUT no_tools (known bug - empty vector DB)
35+
* V2 conversations endpoint finds the correct conversation when it exists
36+
* V2 conversations endpoint fails when auth header is not present
37+
* V2 conversations/{conversation_id} endpoint finds conversation with full metadata
38+
* V2 conversations/{conversation_id} endpoint fails when auth header is not present
39+
* V2 conversations/{conversation_id} GET endpoint fails when conversation_id is malformed
40+
* V2 conversations/{conversation_id} GET endpoint fails when conversation does not exist
41+
* Check conversations/{conversation_id} works when llama-stack is down
42+
* Check conversations/{conversation_id} fails when cache not configured
43+
* V2 conversations DELETE endpoint removes the correct conversation
44+
* V2 conversations/{conversation_id} DELETE endpoint fails when auth header is not present
45+
* V2 conversations/{conversation_id} DELETE endpoint fails when conversation_id is malformed
46+
* V2 conversations DELETE endpoint fails when the conversation does not exist
47+
* V2 conversations DELETE endpoint works even when llama-stack is down
48+
* V2 conversations PUT endpoint successfully updates topic summary
49+
* V2 conversations PUT endpoint fails when auth header is not present
50+
* V2 conversations PUT endpoint fails when conversation_id is malformed
51+
* V2 conversations PUT endpoint fails when conversation does not exist
52+
* V2 conversations PUT endpoint fails with empty topic summary (422)
53+
54+
## [`conversations.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/conversations.feature)
55+
56+
* Check if conversations endpoint finds the correct conversation when it exists
57+
* Check if conversations endpoint fails when the auth header is not present
58+
* Check if conversations/{conversation_id} endpoint finds the correct conversation when it exists
59+
* Check if conversations/{conversation_id} endpoint fails when the auth header is not present
60+
* Check if conversations/{conversation_id} GET endpoint fails when conversation_id is malformed
61+
* Check if conversations/{conversation_id} GET endpoint fails when llama-stack is unavailable
62+
* Check if conversations DELETE endpoint removes the correct conversation
63+
* Check if conversations/{conversation_id} DELETE endpoint fails when conversation_id is malformed
64+
* Check if conversations DELETE endpoint fails when the conversation does not exist
65+
* Check if conversations/{conversation_id} DELETE endpoint fails when llama-stack is unavailable
66+
67+
## [`faiss.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/faiss.feature)
68+
69+
* check if vector store is registered
70+
* Check if rags endpoint fails when llama-stack is unavailable
71+
* Check if rags endpoints responds with error when not authenticated
72+
* Query vector db using the file_search tool
73+
74+
## [`feedback.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/feedback.feature)
75+
76+
* Check if enabling the feedback is working
77+
* Check if disabling the feedback is working
78+
* Check if toggling the feedback with incorrect attribute name fails
79+
* Check if getting feedback status returns true when feedback is enabled
80+
* Check if getting feedback status returns false when feedback is disabled
81+
* Check if feedback endpoint is not working when feedback is disabled
82+
* Check if feedback endpoint fails when required fields are not specified
83+
* Check if feedback endpoint is working when sentiment is negative
84+
* Check if feedback endpoint is working when sentiment is positive
85+
* Check if feedback submittion fails when invald sentiment is passed
86+
* Check if feedback submission fails when nonexisting conversation ID is passed
87+
* Check if feedback submission fails when conversation belongs to a different user
88+
* Check if feedback endpoint is not working when not authorized
89+
* Check if update feedback status endpoint is not working when not authorized
90+
* Check if feedback submittion fails when invalid feedback storage path is configured
91+
* Check if feedback endpoint fails when only empty string user_feedback is provided
92+
93+
## [`health.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/health.feature)
894

995
* Check if service report proper readiness state
1096
* Check if service report proper liveness state
97+
* Check if service report proper readiness state when llama stack is not available
98+
* Check if service report proper liveness state even when llama stack is not available
99+
100+
## [`info.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/info.feature)
101+
11102
* Check if the OpenAPI endpoint works as expected
12103
* Check if info endpoint is working
104+
* Check if info endpoint reports error when llama-stack connection is not working
105+
* Check if models endpoint is working
106+
* Check if models endpoint reports error when llama-stack in unreachable
107+
* Check if shields endpoint is working
108+
* Check if shields endpoint reports error when llama-stack in unreachable
109+
* Check if tools endpoint is working
110+
* Check if tools endpoint reports error when llama-stack in unreachable
111+
* Check if metrics endpoint is working
112+
* Check if MCP client auth options endpoint is working
113+
114+
## [`query.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/query.feature)
115+
116+
* Check if LLM responds properly to restrictive system prompt to sent question with different system prompt
117+
* Check if LLM responds properly to non-restrictive system prompt to sent question with different system prompt
118+
* Check if LLM ignores new system prompt in same conversation
119+
* Check if LLM responds to sent question with error when not authenticated
120+
* Check if LLM responds to sent question with error when bearer token is missing
121+
* Check if LLM responds to sent question with error when model does not exist
122+
* Check if LLM responds to sent question with error when attempting to access conversation
123+
* Check if LLM responds for query request with error for missing query
124+
* Check if LLM responds for query request for missing model and provider
125+
* Check if LLM responds for query request with error for missing model
126+
* Check if LLM responds for query request with error for missing provider
127+
* Check if LLM responds for query request with error for missing provider
128+
* Check if LLM responds properly when XML and JSON attachments are sent
129+
130+
## [`rbac.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/rbac.feature)
131+
132+
* Request without token returns 401
133+
* Request with malformed Authorization header returns 401
134+
* Admin can access query endpoint
135+
* Admin can access models endpoint
136+
* Admin can list conversations
137+
* User can access query endpoint
138+
* User can list conversations
139+
* Viewer can list conversations
140+
* Viewer can access info endpoint
141+
* Viewer cannot query - returns 403
142+
* Query-only user can query without specifying model
143+
* Query-only user cannot override model - returns 403
144+
* Query-only user cannot list conversations - returns 403
145+
* No-role user can access info endpoint (everyone role)
146+
* No-role user cannot query - returns 403
147+
* No-role user cannot list conversations - returns 403
148+
149+
## [`rest_api.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/rest_api.feature)
150+
151+
* Check if the OpenAPI endpoint works as expected
152+
153+
## [`smoketests.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/smoketests.feature)
154+
155+
* Check if the main endpoint is reachable
156+
157+
## [`streaming_query.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/streaming_query.feature)
13158

14-
## [`llm_interface.feature`](https://github.com/lightspeed-core/lightspeed-stack/blob/main/tests/e2e/features/llm_interface.feature)
159+
* Check if streaming_query response in tokens matches the full response
160+
* Check if LLM responds properly to restrictive system prompt to sent question with different system prompt
161+
* Check if LLM responds properly to non-restrictive system prompt to sent question with different system prompt
162+
* Check if LLM ignores new system prompt in same conversation
163+
* Check if LLM responds for streaming_query request with error for missing query
164+
* Check if LLM responds for streaming_query request for missing model and provider
165+
* Check if LLM responds for streaming_query request with error for missing model
166+
* Check if LLM responds for streaming_query request with error for missing provider
167+
* Check if LLM responds properly when XML and JSON attachments are sent
168+
* Check if LLM responds to sent question with error when not authenticated
15169

16-
* Check if LLM responds to sent question

0 commit comments

Comments
 (0)