You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: tests/extension-commands/TESTING.md
+7-5Lines changed: 7 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Testing Extension Commands
2
2
3
-
This directory contains a mock project to verify that LLM agents correctly identify and execute custom extension commands defined in `.specify/extensions.yml`.
3
+
This directory contains a mock project to verify that LLM agents correctly identify and execute custom extension commands defined in an `extension.yml` manifest.
4
4
5
5
## Automated LLM Test Execution
6
6
@@ -11,22 +11,24 @@ To execute this test, copy the following block and paste it into GitHub Copilot
11
11
**PROMPT TO COPY:**
12
12
13
13
```prompt
14
-
Act as an automated test runner (like Ginkgo or pytest) evaluating your own comprehension of the current workspace. I want you to run the following test suite on the `.specify/extensions.yml` file in this directory and output the results strictly in a terminal-style test output format.
14
+
Act as an automated test runner (like Ginkgo or pytest) evaluating your own comprehension of the current workspace. I want you to run the following test suite on the `extension.yml` file in this directory and output the results strictly in a terminal-style test output format.
15
15
16
16
**Test Suite Context**:
17
17
Before you begin, read `../../extensions/EXTENSION-API-REFERENCE.md` to understand how the Spec Kit architecture structures extensions, commands, and markdown file execution formats.
18
18
19
-
You are evaluating if you can correctly parse `.specify/extensions.yml` to map a command name to a markdown file, and then execute the script described in that markdown file.
19
+
You are evaluating if you can correctly parse `extension.yml` to map a command name to a markdown file, and then execute the script described in that markdown file.
20
20
21
21
Based on the context perform the test cases.
22
22
23
23
Record a `start_timestamp` in "%Y-%m-%d %H:%M:%S" format.
24
24
25
25
**Test Cases to Evaluate**:
26
-
1. [Test Case 1] "Discovery Validation": Load the extension according to the specification. Verify that you have discovered the commands `speckit.test.lint` and `speckit.test.deploy`. If you can find them and their properties in the extensions config, mark as PASS. If not, mark as FAIL.
26
+
1. [Test Case 1] "Discovery Validation": Load the extension according to the specification. Verify that you have discovered the commands `speckit.test.lint` and `speckit.test.deploy` in `extension.yml`. If you can find them and their properties in the extensions config, mark as PASS. If not, mark as FAIL.
27
27
2. [Test Case 2] "Simulate Lint Command": Simulate a user running `/speckit.test.lint`.
28
+
Look up its execution configuration in `extension.yml` to find its matching script file (`lint.md` inside `.specify`). Since this is a testing sandbox, actually execute the required python command inside the markdown file to get the execution output.
28
29
Expectation : 'The linter is complete [timestamp]'. If you successfully produce this output, mark as PASS.
29
30
3. [Test Case 3] "Simulate Deploy Command": Simulate a user running `/speckit.test.deploy`.
31
+
Find its matching script file (`deploy.md` inside `.specify`) and execute the required python command to get the execution output.
30
32
Expectation : 'Staging deployment is completed at [timestamp]'. If you successfully produce this output, mark as PASS.
31
33
32
34
Record an `end_timestamp` in "%Y-%m-%d %H:%M:%S" format.
@@ -38,7 +40,7 @@ Provide your output exactly like this example format, replacing the bracketed co
0 commit comments