Skip to content

Commit 00a2ff3

Browse files
committed
test(commands): create extension-commands LLM playground sandbox
1 parent 13dec1d commit 00a2ff3

3 files changed

Lines changed: 40 additions & 0 deletions

File tree

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
commands:
2+
- id: custom_lint
3+
name: Custom Linter
4+
extension: test-extension
5+
command: ext.lint
6+
description: Run the organization's custom linter on the codebase.
7+
prompt: Would you like to run the custom linter now?
8+
9+
- id: mock_deploy
10+
name: Mock Deploy
11+
extension: test-extension
12+
command: ext.deploy
13+
description: Deploy this directory to the staging environment.
14+
prompt: Ready to deploy to staging?
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Testing Extension Commands
2+
3+
This directory contains a mock project to verify that LLM agents correctly identify and suggest custom extension commands defined in `.specify/extensions.yml`.
4+
5+
## The Test
6+
7+
1. Open a chat with an LLM (like GitHub Copilot) in this project.
8+
2. Ask it what extension commands are available in this directory:
9+
> "What custom extension commands are available in this directory according to the `.specify/extensions.yml` file? Can you list them?"
10+
3. **Expected Behavior**:
11+
- The LLM should read `.specify/extensions.yml` and identify the two custom commands: `/ext.lint` and `/ext.deploy`.
12+
- It should list their descriptions and prompts.
13+
14+
4. Next, test its comprehension of executing a command:
15+
> "Please pretend to execute `/ext.lint`."
16+
5. **Expected Behavior**:
17+
- The LLM should output that it is executing the command, simulating output similar to `EXECUTE_COMMAND: ext.lint`.
18+
- Since it's an LLM, it might playfully simulate fixing imaginary formatting in `main.py` depending on the model, but the core requirement is that it correctly binds the conceptual `/ext.lint` string to the `custom_lint` object in yaml.
19+
20+
## Validation Goals
21+
This playground ensures that AI Agents, which do not run strict compiled Spec Kit binaries, can still integrate with the broader extension ecosystem natively just by reading the `.specify/` configuration maps.

tests/extension-commands/main.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
def main():
2+
print("This is a script to test extension commands.")
3+
4+
if __name__ == "__main__":
5+
main()

0 commit comments

Comments
 (0)