You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: tests/extension-commands/TESTING.md
+9-9Lines changed: 9 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,9 +1,5 @@
1
1
# Testing Extension Commands
2
2
3
-
This directory contains a mock project to verify that LLM agents correctly identify and suggest custom extension commands defined in `.specify/extensions.yml`.
4
-
5
-
# Testing Extension Commands
6
-
7
3
This directory contains a mock project to verify that LLM agents correctly identify and execute custom extension commands defined in `.specify/extensions.yml`.
8
4
9
5
## Automated LLM Test Execution
@@ -23,20 +19,24 @@ Before you begin, read `../../extensions/EXTENSION-API-REFERENCE.md` to understa
23
19
You are evaluating if you can correctly parse `.specify/extensions.yml` to map a command name to a markdown file, and then execute the script described in that markdown file.
24
20
25
21
**Test Cases to Evaluate**:
26
-
1. [Test Case 1] "Discovery Validation": Read `.specify/extensions.yml`. Verify that you can find the commands `speckit.test.lint` mapping to `lint.md`, and `speckit.test.deploy` mapping to `deploy.md`. If you can, mark this test as PASS. If you cannot find them, mark as FAIL.
27
-
2. [Test Case 2] "Intent Binding": Pretend to execute the `speckit.test.lint` command. First, look at `.specify/extensions.yml` to find its matching file (`lint.md`). Open `lint.md` to find the exact python command required for the linter. Since this is a sandbox, to successfully evaluate the command you must actually run that python command here to get the execution output. If your execution provides the python console output (like 'The linter is complete [timestamp]'), mark as PASS. If you don't know what to do, mark as FAIL.
22
+
1. [Test Case 1] "Discovery Validation": Load the extension according to the specification. Verify that you have discovered the commands `speckit.test.lint` and `speckit.test.deploy`. If you can find them and their properties in the extensions config, mark as PASS. If not, mark as FAIL.
23
+
2. [Test Case 2] "Simulate Lint Command": Simulate a user running `/speckit.test.lint`. Look up its execution configuration in `.specify/extensions.yml` and find its matching script file (`lint.md`). Since this is a testing sandbox, actually execute the required python command to get the execution output. Validate the output of the python program (it should be 'The linter is complete [timestamp]'). If you successfully produce this output, mark as PASS.
24
+
3. [Test Case 3] "Simulate Deploy Command": Simulate a user running `/speckit.test.deploy`. Find its matching script file (`deploy.md`). Actually execute the required python command to get the execution output. Validate the output of the python program (it should be 'Staging deployment is completed at [timestamp]'). If you successfully produce this output, mark as PASS.
28
25
29
26
**Required Output Format**:
30
27
Provide your output exactly like this example format, replacing the bracketed content with your actual evaluation logic:
31
28
32
29
============================= test session starts ==============================
0 commit comments