Skip to content

Commit cb16412

Browse files
authored
docs: ensure manual tests use local specify (#2020)
* docs: ensure manual tests use local specify * docs: mention venv activation before editable install * docs: clarify Windows venv activation commands
1 parent 804cd10 commit cb16412

File tree

2 files changed

+74
-7
lines changed

2 files changed

+74
-7
lines changed

CONTRIBUTING.md

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,14 +36,17 @@ On [GitHub Codespaces](https://github.com/features/codespaces) it's even simpler
3636
> If your pull request introduces a large change that materially impacts the work of the CLI or the rest of the repository (e.g., you're introducing new templates, arguments, or otherwise major changes), make sure that it was **discussed and agreed upon** by the project maintainers. Pull requests with large changes that did not have a prior conversation and agreement will be closed.
3737
3838
1. Fork and clone the repository
39-
1. Configure and install the dependencies: `uv sync`
39+
1. Configure and install the dependencies: `uv sync --extra test`
4040
1. Make sure the CLI works on your machine: `uv run specify --help`
4141
1. Create a new branch: `git checkout -b my-branch-name`
4242
1. Make your change, add tests, and make sure everything still works
4343
1. Test the CLI functionality with a sample project if relevant
4444
1. Push to your fork and submit a pull request
4545
1. Wait for your pull request to be reviewed and merged.
4646

47+
For the detailed test workflow, command-selection prompt, and PR reporting template, see [`TESTING.md`](./TESTING.md).
48+
Activate the project virtual environment (see the Setup block in [`TESTING.md`](./TESTING.md)), then install the CLI from your working tree (`uv pip install -e .` after `uv sync --extra test`) or otherwise ensure the shell uses the local `specify` binary before running the manual slash-command tests described below.
49+
4750
Here are a few things you can do that will increase the likelihood of your pull request being accepted:
4851

4952
- Follow the project's coding conventions.
@@ -62,6 +65,14 @@ When working on spec-kit:
6265
3. Test script functionality in the `scripts/` directory
6366
4. Ensure memory files (`memory/constitution.md`) are updated if major process changes are made
6467

68+
### Recommended validation flow
69+
70+
For the smoothest review experience, validate changes in this order:
71+
72+
1. **Run focused automated checks first** — use the quick verification commands in [`TESTING.md`](./TESTING.md) to catch packaging, scaffolding, and configuration regressions early.
73+
2. **Run manual workflow tests second** — if your change affects slash commands or the developer workflow, follow [`TESTING.md`](./TESTING.md) to choose the right commands, run them in an agent, and capture results for your PR.
74+
3. **Use local release packages when debugging packaged output** — if you need to inspect the exact files CI-style packaging produces, generate local release packages as described below.
75+
6576
### Testing template and command changes locally
6677

6778
Running `uv run specify init` pulls released packages, which won’t include your local changes.
@@ -85,6 +96,8 @@ To test your templates, commands, and other changes locally, follow these steps:
8596

8697
Navigate to your test project folder and open the agent to verify your implementation.
8798

99+
If you only need to validate generated file structure and content before doing manual agent testing, start with the focused automated checks in [`TESTING.md`](./TESTING.md). Keep this section for the cases where you need to inspect the exact packaged output locally.
100+
88101
## AI contributions in Spec Kit
89102

90103
> [!IMPORTANT]

TESTING.md

Lines changed: 60 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,59 @@
1-
# Manual Testing Guide
1+
# Testing Guide
2+
3+
This document is the detailed testing companion to [`CONTRIBUTING.md`](./CONTRIBUTING.md).
4+
5+
Use it for three things:
6+
7+
1. running quick automated checks before manual testing,
8+
2. manually testing affected slash commands through an AI agent, and
9+
3. capturing the results in a PR-friendly format.
210

311
Any change that affects a slash command's behavior requires manually testing that command through an AI agent and submitting results with the PR.
412

5-
## Process
13+
## Recommended order
14+
15+
1. **Sync your environment** — install the project and test dependencies.
16+
2. **Run focused automated checks** — especially for packaging, scaffolding, agent config, and generated-file changes.
17+
3. **Run manual agent tests** — for any affected slash commands.
18+
4. **Paste results into your PR** — include both command-selection reasoning and manual test results.
19+
20+
## Quick automated checks
21+
22+
Run these before manual testing when your change affects packaging, scaffolding, templates, release artifacts, or agent wiring.
23+
24+
### Environment setup
25+
26+
```bash
27+
cd <spec-kit-repo>
28+
uv sync --extra test
29+
source .venv/bin/activate # On Windows (CMD): .venv\Scripts\activate | (PowerShell): .venv\Scripts\Activate.ps1
30+
```
31+
32+
### Generated package structure and content
33+
34+
```bash
35+
uv run python -m pytest tests/test_core_pack_scaffold.py -q
36+
```
37+
38+
This validates the generated files that CI-style packaging depends on, including directory layout, file names, frontmatter/TOML validity, placeholder replacement, `.specify/` path rewrites, and parity with `create-release-packages.sh`.
39+
40+
### Agent configuration and release wiring consistency
41+
42+
```bash
43+
uv run python -m pytest tests/test_agent_config_consistency.py -q
44+
```
45+
46+
Run this when you change agent metadata, release scripts, context update scripts, or artifact naming.
47+
48+
### Optional single-agent packaging spot check
49+
50+
```bash
51+
AGENTS=copilot SCRIPTS=sh ./.github/workflows/scripts/create-release-packages.sh v1.0.0
52+
```
53+
54+
Inspect `.genreleases/sdd-copilot-package-sh/` and the matching ZIP in `.genreleases/` when you want to review the exact packaged output for one agent/script combination.
55+
56+
## Manual testing process
657

758
1. **Identify affected commands** — use the [prompt below](#determining-which-tests-to-run) to have your agent analyze your changed files and determine which commands need testing.
859
2. **Set up a test project** — scaffold from your local branch (see [Setup](#setup)).
@@ -13,19 +64,22 @@ Any change that affects a slash command's behavior requires manually testing tha
1364
## Setup
1465

1566
```bash
16-
# Install the CLI from your local branch
67+
# Install the project and test dependencies from your local branch
1768
cd <spec-kit-repo>
18-
uv venv .venv
19-
source .venv/bin/activate # On Windows: .venv\Scripts\activate
69+
uv sync --extra test
70+
source .venv/bin/activate # On Windows (CMD): .venv\Scripts\activate | (PowerShell): .venv\Scripts\Activate.ps1
2071
uv pip install -e .
72+
# Ensure the `specify` binary in this environment points at your working tree so the agent runs the branch you're testing.
2173

2274
# Initialize a test project using your local changes
23-
specify init /tmp/speckit-test --ai <agent> --offline
75+
uv run specify init /tmp/speckit-test --ai <agent> --offline
2476
cd /tmp/speckit-test
2577

2678
# Open in your agent
2779
```
2880

81+
If you are testing the packaged output rather than the live source tree, create a local release package first as described in [`CONTRIBUTING.md`](./CONTRIBUTING.md).
82+
2983
## Reporting results
3084

3185
Paste this into your PR:

0 commit comments

Comments
 (0)