Skip to content

Commit cdfcf62

Browse files
jrobertboosjrobertboos
committed
tmp
1 parent 5c73738 commit cdfcf62

1 file changed

Lines changed: 39 additions & 164 deletions

File tree

‎tests/e2e/features/skills.feature‎

Lines changed: 39 additions & 164 deletions
Original file line numberDiff line numberDiff line change
@@ -1,53 +1,14 @@
1-
@e2e_group_2
1+
@e2e_group_2 @skip
22
Feature: Agent skills tests
33

44
Background:
55
Given The service is started locally
66
And The system is in default state
7-
And I set the Authorization header to Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6Ikpva
87
And REST API service prefix is /v1
98
And the Lightspeed stack configuration directory is "tests/e2e/configuration"
109

11-
# --- Configuration & startup ---
12-
13-
@SkillsConfig
14-
Scenario: Service starts successfully with skills configured
15-
Given The e2e-test-skill skill directory is present in the container
16-
And The service uses the lightspeed-stack-skills.yaml configuration
17-
And The service is restarted
18-
When I access endpoint "readiness" using HTTP GET method
19-
Then The status code of the response is 200
20-
And The body of the response contains true
21-
22-
@SkillsConfig
23-
Scenario: Service fails to start when skill path does not exist
24-
Given The service uses the lightspeed-stack-skills-invalid-path.yaml configuration
25-
And The service is restarted
26-
When I access endpoint "readiness" using HTTP GET method
27-
Then The status code of the response is not 200
28-
29-
@SkillsConfig
30-
Scenario: Service fails to start when SKILL.md is missing from skill directory
31-
Given The e2e-missing-skillmd-skill skill directory is present in the container
32-
And The service uses the lightspeed-stack-skills-missing-skillmd.yaml configuration
33-
And The service is restarted
34-
When I access endpoint "readiness" using HTTP GET method
35-
Then The status code of the response is not 200
36-
37-
@SkillsConfig
38-
Scenario: Service fails to start when SKILL.md has invalid frontmatter
39-
Given The service uses the lightspeed-stack-skills-invalid-frontmatter.yaml configuration
40-
And The service is restarted
41-
When I access endpoint "readiness" using HTTP GET method
42-
Then The status code of the response is not 200
43-
44-
@SkillsConfig
45-
Scenario: Service fails to start when duplicate skill names are configured
46-
Given The service uses the lightspeed-stack-skills-duplicate-names.yaml configuration
47-
And The service is restarted
48-
When I access endpoint "readiness" using HTTP GET method
49-
Then The status code of the response is not 200
5010

11+
#TODO: Remove "The e2e-test-skill skill directory is present in the container"
5112

5213
# --- Skill tools registration ---
5314

@@ -61,6 +22,7 @@ Feature: Agent skills tests
6122
And The body of the response contains list_skills
6223
And The body of the response contains activate_skill
6324
And The body of the response contains load_skill_resource
25+
#TODO: list all the tools, check for number of tools (total) (More comprehensive than just basic testing is +)
6426

6527
Scenario: Skill tools are not registered when no skills are configured
6628
Given The service uses the lightspeed-stack.yaml configuration
@@ -70,31 +32,33 @@ Feature: Agent skills tests
7032
And The body of the response does not contain list_skills
7133
And The body of the response does not contain activate_skill
7234
And The body of the response does not contain load_skill_resource
35+
#TODO: list all the tools, check for number of tools (total should be just non-skill tools)
7336

7437

7538
# --- Skill discovery ---
7639

77-
@SkillsConfig @Authorized @flaky
40+
@SkillsConfig
7841
Scenario: LLM can discover skills via list_skills tool using query endpoint
7942
Given The e2e-test-skill skill directory is present in the container
8043
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
8144
And The service is restarted
8245
And I capture the current token metrics
83-
When I use "query" to ask question with authorization header
46+
When I use "query" to ask question
8447
"""
8548
{"query": "What skills are available? Use the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"}
8649
"""
8750
Then The status code of the response is 200
88-
And The body of the response contains e2e-test-skill
51+
And The body of the response contains e2e-test-skill #TODO: Make this more specific (instead of checking entire response check the skill content metadata (new step required))
52+
#TODO: Instead of ^ Check tool results from list_skills tool (make it more decisive)
8953
And The token metrics have increased
9054

91-
@SkillsConfig @Authorized @flaky
55+
@SkillsConfig
9256
Scenario: LLM can discover skills via list_skills tool using streaming_query endpoint
9357
Given The e2e-test-skill skill directory is present in the container
9458
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
9559
And The service is restarted
9660
And I capture the current token metrics
97-
When I use "streaming_query" to ask question with authorization header
61+
When I use "streaming_query" to ask question
9862
"""
9963
{"query": "What skills are available? Use the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"}
10064
"""
@@ -104,31 +68,32 @@ Feature: Agent skills tests
10468
| Fragments in LLM response |
10569
| e2e-test-skill |
10670
And The token metrics have increased
71+
#TODO: SEE ABOVE TEST
10772

10873

10974
# --- Skill activation ---
11075

111-
@SkillsConfig @Authorized @flaky
76+
@SkillsConfig
11277
Scenario: LLM can activate a skill and use its instructions via query endpoint
11378
Given The e2e-test-skill skill directory is present in the container
11479
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
11580
And The service is restarted
11681
And I capture the current token metrics
117-
When I use "query" to ask question with authorization header
82+
When I use "query" to ask question
11883
"""
11984
{"query": "I need help with e2e testing. Use the activate_skill tool to load the e2e-test-skill.", "model": "{MODEL}", "provider": "{PROVIDER}"}
12085
"""
12186
Then The status code of the response is 200
122-
And The body of the response contains skill_content
87+
And The body of the response contains skill_content #FIX: VERY GENERAL should check tool_results instead
12388
And The token metrics have increased
12489

125-
@SkillsConfig @Authorized @flaky
90+
@SkillsConfig
12691
Scenario: LLM can activate a skill and use its instructions via streaming_query endpoint
12792
Given The e2e-test-skill skill directory is present in the container
12893
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
12994
And The service is restarted
13095
And I capture the current token metrics
131-
When I use "streaming_query" to ask question with authorization header
96+
When I use "streaming_query" to ask question
13297
"""
13398
{"query": "I need help with e2e testing. Use the activate_skill tool to load the e2e-test-skill.", "model": "{MODEL}", "provider": "{PROVIDER}"}
13499
"""
@@ -142,27 +107,27 @@ Feature: Agent skills tests
142107

143108
# --- Skill resource loading ---
144109

145-
@SkillsConfig @Authorized @flaky
110+
@SkillsConfig
146111
Scenario: LLM can load a skill reference file via load_skill_resource tool using query endpoint
147112
Given The e2e-test-skill skill directory is present in the container
148113
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
149114
And The service is restarted
150115
And I capture the current token metrics
151-
When I use "query" to ask question with authorization header
116+
When I use "query" to ask question
152117
"""
153118
{"query": "Load the reference file references/guide.md from the e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"}
154119
"""
155120
Then The status code of the response is 200
156121
And The body of the response contains skill_resource
157122
And The token metrics have increased
158123

159-
@SkillsConfig @Authorized @flaky
124+
@SkillsConfig
160125
Scenario: LLM can load a skill reference file via load_skill_resource tool using streaming_query endpoint
161126
Given The e2e-test-skill skill directory is present in the container
162127
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
163128
And The service is restarted
164129
And I capture the current token metrics
165-
When I use "streaming_query" to ask question with authorization header
130+
When I use "streaming_query" to ask question
166131
"""
167132
{"query": "Load the reference file references/guide.md from the e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"}
168133
"""
@@ -173,114 +138,55 @@ Feature: Agent skills tests
173138
| skill_resource |
174139
And The token metrics have increased
175140

176-
177-
# --- Security: path traversal ---
178-
179-
@SkillsConfig @Authorized @flaky
180-
Scenario: load_skill_resource rejects path traversal attempts via query endpoint
181-
Given The e2e-test-skill skill directory is present in the container
182-
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
183-
And The service is restarted
184-
When I use "query" to ask question with authorization header
185-
"""
186-
{"query": "Load the resource ../../etc/passwd from the e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"}
187-
"""
188-
Then The status code of the response is 200
189-
And The body of the response contains outside skill directory
190-
191-
@SkillsConfig @Authorized @flaky
192-
Scenario: load_skill_resource rejects path traversal attempts via streaming_query endpoint
193-
Given The e2e-test-skill skill directory is present in the container
194-
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
195-
And The service is restarted
196-
When I use "streaming_query" to ask question with authorization header
197-
"""
198-
{"query": "Load the resource ../../etc/passwd from the e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"}
199-
"""
200-
When I wait for the response to be completed
201-
Then The status code of the response is 200
202-
And The streamed response contains following fragments
203-
| Fragments in LLM response |
204-
| outside skill directory |
205-
206-
207141
# --- Error handling: unknown skill ---
208142

209-
@SkillsConfig @Authorized @flaky
143+
@SkillsConfig
210144
Scenario: activate_skill returns error for unknown skill name via query endpoint
211145
Given The e2e-test-skill skill directory is present in the container
212146
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
213147
And The service is restarted
214-
When I use "query" to ask question with authorization header
148+
When I use "query" to ask question
215149
"""
216150
{"query": "Activate a skill called nonexistent-skill using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"}
217151
"""
218152
Then The status code of the response is 200
219-
And The body of the response contains Unknown skill
153+
And The body of the response contains Unknown skill #TODO: Make more descriptive
220154

221-
@SkillsConfig @Authorized @flaky
155+
@SkillsConfig
222156
Scenario: activate_skill returns error for unknown skill name via streaming_query endpoint
223157
Given The e2e-test-skill skill directory is present in the container
224158
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
225159
And The service is restarted
226-
When I use "streaming_query" to ask question with authorization header
160+
When I use "streaming_query" to ask question
227161
"""
228162
{"query": "Activate a skill called nonexistent-skill using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"}
229163
"""
230164
When I wait for the response to be completed
231165
Then The status code of the response is 200
232166
And The streamed response contains following fragments
233167
| Fragments in LLM response |
234-
| Unknown skill |
235-
236-
@SkillsConfig @Authorized @flaky
237-
Scenario: load_skill_resource returns error for unknown skill name via query endpoint
238-
Given The e2e-test-skill skill directory is present in the container
239-
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
240-
And The service is restarted
241-
When I use "query" to ask question with authorization header
242-
"""
243-
{"query": "Load references/guide.md from a skill called nonexistent-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"}
244-
"""
245-
Then The status code of the response is 200
246-
And The body of the response contains Unknown skill
247-
248-
@SkillsConfig @Authorized @flaky
249-
Scenario: load_skill_resource returns error for unknown skill name via streaming_query endpoint
250-
Given The e2e-test-skill skill directory is present in the container
251-
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
252-
And The service is restarted
253-
When I use "streaming_query" to ask question with authorization header
254-
"""
255-
{"query": "Load references/guide.md from a skill called nonexistent-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"}
256-
"""
257-
When I wait for the response to be completed
258-
Then The status code of the response is 200
259-
And The streamed response contains following fragments
260-
| Fragments in LLM response |
261-
| Unknown skill |
262-
168+
| Unknown skill | #TODO: Make descriptive
263169
264170
# --- Error handling: missing resource ---
265171
266-
@SkillsConfig @Authorized @flaky
172+
@SkillsConfig
267173
Scenario: load_skill_resource returns error for nonexistent resource file via query endpoint
268174
Given The e2e-test-skill skill directory is present in the container
269175
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
270176
And The service is restarted
271-
When I use "query" to ask question with authorization header
177+
When I use "query" to ask question
272178
"""
273179
{"query": "Load references/nonexistent.md from e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"}
274180
"""
275181
Then The status code of the response is 200
276182
And The body of the response contains Resource not found
277183
278-
@SkillsConfig @Authorized @flaky
184+
@SkillsConfig
279185
Scenario: load_skill_resource returns error for nonexistent resource file via streaming_query endpoint
280186
Given The e2e-test-skill skill directory is present in the container
281187
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
282188
And The service is restarted
283-
When I use "streaming_query" to ask question with authorization header
189+
When I use "streaming_query" to ask question
284190
"""
285191
{"query": "Load references/nonexistent.md from e2e-test-skill using load_skill_resource.", "model": "{MODEL}", "provider": "{PROVIDER}"}
286192
"""
@@ -293,12 +199,12 @@ Feature: Agent skills tests
293199

294200
# --- Context management: deduplication ---
295201

296-
@SkillsConfig @Authorized @flaky
202+
@SkillsConfig
297203
Scenario: Duplicate skill activation in same conversation returns already-loaded note via query endpoint
298-
Given The e2e-test-skill skill directory is present in the container
204+
Given The e2e-test-skill skill directory path is "e2e-test-skill"
299205
And The service uses the lightspeed-stack-skills-auth-noop-token.yaml configuration
300206
And The service is restarted
301-
When I use "query" to ask question with authorization header
207+
When I use "query" to ask question
302208
"""
303209
{"query": "Activate e2e-test-skill using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"}
304210
"""
@@ -309,63 +215,32 @@ Feature: Agent skills tests
309215
{"query": "Activate e2e-test-skill again using the activate_skill tool.", "model": "{MODEL}", "provider": "{PROVIDER}"}
310216
"""
311217
Then The status code of the response is 200
312-
And The body of the response contains already loaded
218+
And The body of the response contains already loaded #FIX: This looks/feels wrong (make more descriptve)
313219

314220

315221
# --- Multiple skills ---
316222

317-
@SkillsMultiConfig @Authorized @flaky
318-
Scenario: Multiple skills can be discovered via query endpoint
319-
Given The e2e-test-skill skill directory is present in the container
320-
And The e2e-second-skill skill directory is present in the container
321-
And The service uses the lightspeed-stack-skills-multi.yaml configuration
322-
And The service is restarted
323-
When I use "query" to ask question with authorization header
324-
"""
325-
{"query": "List all available skills using the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"}
326-
"""
327-
Then The status code of the response is 200
328-
And The body of the response contains e2e-test-skill
329-
And The body of the response contains e2e-second-skill
330-
331-
@SkillsMultiConfig @Authorized @flaky
332-
Scenario: Multiple skills can be discovered via streaming_query endpoint
333-
Given The e2e-test-skill skill directory is present in the container
334-
And The e2e-second-skill skill directory is present in the container
335-
And The service uses the lightspeed-stack-skills-multi.yaml configuration
336-
And The service is restarted
337-
When I use "streaming_query" to ask question with authorization header
338-
"""
339-
{"query": "List all available skills using the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"}
340-
"""
341-
When I wait for the response to be completed
342-
Then The status code of the response is 200
343-
And The streamed response contains following fragments
344-
| Fragments in LLM response |
345-
| e2e-test-skill |
346-
| e2e-second-skill |
347-
348-
@SkillsMultiConfig @Authorized @flaky
223+
@SkillsMultiConfig
349224
Scenario: Skills directory path discovers all skills in subdirectories via query endpoint
350225
Given The e2e-test-skill skill directory is present in the container
351226
And The e2e-second-skill skill directory is present in the container
352227
And The service uses the lightspeed-stack-skills-directory.yaml configuration
353228
And The service is restarted
354-
When I use "query" to ask question with authorization header
229+
When I use "query" to ask question
355230
"""
356231
{"query": "List all available skills using the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"}
357232
"""
358233
Then The status code of the response is 200
359234
And The body of the response contains e2e-test-skill
360235
And The body of the response contains e2e-second-skill
361236

362-
@SkillsMultiConfig @Authorized @flaky
237+
@SkillsMultiConfig
363238
Scenario: Skills directory path discovers all skills in subdirectories via streaming_query endpoint
364-
Given The e2e-test-skill skill directory is present in the container
239+
Given The e2e-test-skill skill directory path is "skills/e2e-test-skill"
365240
And The e2e-second-skill skill directory is present in the container
366241
And The service uses the lightspeed-stack-skills-directory.yaml configuration
367242
And The service is restarted
368-
When I use "streaming_query" to ask question with authorization header
243+
When I use "streaming_query" to ask question
369244
"""
370245
{"query": "List all available skills using the list_skills tool.", "model": "{MODEL}", "provider": "{PROVIDER}"}
371246
"""

0 commit comments

Comments
 (0)