|
1 | | -# Integration Test Checklist |
| 1 | +# Manual MCP QA Runbook |
2 | 2 |
|
3 | | -Manual verification steps for mcp-server-python-docs v0.1.0. |
4 | | -These tests require human execution -- they cannot be automated. |
| 3 | +Use this document for manual MCP validation during development and before a |
| 4 | +release. The goal is to verify real client behavior after the automated test |
| 5 | +suite passes. |
5 | 6 |
|
6 | | -## Prerequisites |
7 | | - |
8 | | -- [ ] All CI tests pass on main branch |
9 | | -- [ ] `mcp-server-python-docs doctor` reports all checks PASS |
10 | | -- [ ] Index is built: `mcp-server-python-docs build-index --versions 3.12,3.13` |
11 | | - |
12 | | -## Test 1: Claude Desktop Integration (SHIP-01) |
13 | | - |
14 | | -### Setup |
| 7 | +Release-specific sign-off still lives in [`.github/RELEASE.md`](RELEASE.md). |
15 | 8 |
|
16 | | -1. Open Claude Desktop settings (Developer > Edit Config or `~/Library/Application Support/Claude/claude_desktop_config.json` on macOS) |
17 | | -2. Add the following to `mcpServers`: |
18 | | - ```json |
19 | | - { |
20 | | - "mcpServers": { |
21 | | - "python-docs": { |
22 | | - "command": "uvx", |
23 | | - "args": ["mcp-server-python-docs"] |
24 | | - } |
25 | | - } |
26 | | - } |
27 | | - ``` |
28 | | -3. Restart Claude Desktop |
29 | | -4. Verify the MCP server icon appears in the chat input area |
30 | | - |
31 | | -### Test Steps |
32 | | - |
33 | | -- [ ] **T1.1**: Ask Claude: "what is asyncio.TaskGroup" |
34 | | - - **Expected**: Response references `asyncio.TaskGroup` with a URI containing `library/asyncio-task.html` |
35 | | - - **Expected**: Response includes symbol information (not just generic LLM knowledge) |
36 | | - - **Actual result**: _______________ |
| 9 | +## Prerequisites |
37 | 10 |
|
38 | | -- [ ] **T1.2**: Ask Claude: "how do I use pathlib.Path.glob" |
39 | | - - **Expected**: Response references `pathlib` documentation with relevant section content |
40 | | - - **Actual result**: _______________ |
| 11 | +- CI or local checks are green: |
| 12 | + - `uv run ruff check src/ tests/` |
| 13 | + - `uv run pyright src/` |
| 14 | + - `uv run pytest --tb=short -q` |
| 15 | +- Local index build completed: |
| 16 | + - `uv run mcp-server-python-docs build-index --versions 3.12,3.13` |
| 17 | +- Doctor passes: |
| 18 | + - `uv run mcp-server-python-docs doctor` |
| 19 | +- If `uv` is not on `PATH`, use `python -m uv ...` instead |
41 | 20 |
|
42 | | -- [ ] **T1.3**: Ask Claude: "search for json parsing in Python" |
43 | | - - **Expected**: Response includes hits from the `json` module documentation |
44 | | - - **Actual result**: _______________ |
| 21 | +## Test 1: MCP Inspector quick loop |
45 | 22 |
|
46 | | -- [ ] **T1.4**: Verify no errors in Claude Desktop developer console |
47 | | - - **Expected**: No MCP protocol errors or connection drops |
48 | | - - **Actual result**: _______________ |
| 23 | +Use Inspector for fast local iteration before checking real clients. |
49 | 24 |
|
50 | | -### Teardown |
| 25 | +### Start Inspector |
51 | 26 |
|
52 | | -- Remove the `python-docs` entry from `mcpServers` (or keep for ongoing use) |
| 27 | +```bash |
| 28 | +npx @modelcontextprotocol/inspector uv --directory . run mcp-server-python-docs |
| 29 | +``` |
53 | 30 |
|
54 | | -## Test 2: Cursor Integration (SHIP-02) |
| 31 | +### Verify |
| 32 | + |
| 33 | +- [ ] Connect successfully over stdio |
| 34 | +- [ ] Confirm the tool list includes: |
| 35 | + - `search_docs` |
| 36 | + - `get_docs` |
| 37 | + - `list_versions` |
| 38 | + - `detect_python_version` |
| 39 | +- [ ] Call `search_docs` with query `asyncio.TaskGroup`, `kind="symbol"`, `version="3.13"` |
| 40 | + - Expected: exact symbol hit with `library/asyncio-task.html` |
| 41 | +- [ ] Call `get_docs` for the returned slug and anchor |
| 42 | + - Expected: section-level documentation, not an unrelated page dump |
| 43 | +- [ ] Call `list_versions` |
| 44 | + - Expected: indexed versions appear with the configured default version |
| 45 | +- [ ] Call `detect_python_version` |
| 46 | + - Expected: returns local interpreter information without breaking the session |
| 47 | +- [ ] Observe no protocol corruption or unexplained disconnects in Inspector |
| 48 | + |
| 49 | +## Test 2: Claude Desktop integration |
55 | 50 |
|
56 | 51 | ### Setup |
57 | 52 |
|
58 | | -1. Open Cursor Settings > MCP |
59 | | -2. Add a new MCP server: |
60 | | - - **Name**: python-docs |
61 | | - - **Command**: `uvx` |
62 | | - - **Args**: `mcp-server-python-docs` |
63 | | -3. Verify the server shows as connected (green indicator) |
64 | | - |
65 | | -### Test Steps |
66 | | - |
67 | | -- [ ] **T2.1**: In a chat or Composer session, ask: "what is asyncio.TaskGroup" |
68 | | - - **Expected**: Response references `asyncio.TaskGroup` with documentation content |
69 | | - - **Expected**: The MCP tool call is visible in the chat |
70 | | - - **Actual result**: _______________ |
71 | | - |
72 | | -- [ ] **T2.2**: Ask: "show me the docs for collections.OrderedDict" |
73 | | - - **Expected**: Response includes `collections.OrderedDict` documentation |
74 | | - - **Actual result**: _______________ |
| 53 | +1. Open Claude Desktop settings |
| 54 | +2. Add this server config: |
| 55 | + |
| 56 | +```json |
| 57 | +{ |
| 58 | + "mcpServers": { |
| 59 | + "python-docs": { |
| 60 | + "command": "uvx", |
| 61 | + "args": ["mcp-server-python-docs"] |
| 62 | + } |
| 63 | + } |
| 64 | +} |
| 65 | +``` |
75 | 66 |
|
76 | | -- [ ] **T2.3**: Verify the server stays connected across multiple queries |
77 | | - - **Expected**: No disconnections or "server not responding" errors |
78 | | - - **Actual result**: _______________ |
| 67 | +3. Fully restart Claude Desktop |
| 68 | +4. Verify the MCP server appears in the chat UI |
79 | 69 |
|
80 | | -### Teardown |
| 70 | +### Checks |
81 | 71 |
|
82 | | -- Remove or disable the python-docs MCP server in Cursor settings (or keep) |
| 72 | +- [ ] Ask: `what is asyncio.TaskGroup` |
| 73 | + - Expected: response uses stdlib documentation, not only model prior knowledge |
| 74 | +- [ ] Ask: `how do I use pathlib.Path.glob` |
| 75 | + - Expected: response cites the right docs section |
| 76 | +- [ ] Ask: `search for json parsing in Python` |
| 77 | + - Expected: response surfaces `json` docs results |
| 78 | +- [ ] Check the Claude developer console |
| 79 | + - Expected: no protocol errors or repeated reconnect loops |
83 | 80 |
|
84 | | -## Test 3: Fresh Install Verification (SHIP-06 partial) |
| 81 | +## Test 3: Cursor integration |
85 | 82 |
|
86 | 83 | ### Setup |
87 | 84 |
|
88 | | -1. Create a throwaway virtualenv or use a machine without the package: |
89 | | - ```bash |
90 | | - # Option A: Fresh venv |
91 | | - uv venv /tmp/test-install && source /tmp/test-install/bin/activate |
92 | | - |
93 | | - # Option B: Use uvx (isolated by default) |
94 | | - # No setup needed -- uvx creates its own isolated env |
95 | | - ``` |
| 85 | +1. Open Cursor MCP settings |
| 86 | +2. Add a server: |
| 87 | + - Name: `python-docs` |
| 88 | + - Command: `uvx` |
| 89 | + - Args: `mcp-server-python-docs` |
| 90 | +3. Confirm the server shows as connected |
96 | 91 |
|
97 | | -### Test Steps |
| 92 | +### Checks |
98 | 93 |
|
99 | | -- [ ] **T3.1**: Install from PyPI (after package is published): |
100 | | - ```bash |
101 | | - uvx mcp-server-python-docs --version |
102 | | - ``` |
103 | | - - **Expected**: Prints `0.1.0` |
104 | | - - **Actual result**: _______________ |
| 94 | +- [ ] Ask: `what is asyncio.TaskGroup` |
| 95 | + - Expected: MCP tool usage is visible and the answer references the right docs |
| 96 | +- [ ] Ask: `show me the docs for collections.OrderedDict` |
| 97 | + - Expected: response includes the relevant documentation section |
| 98 | +- [ ] Ask a second or third follow-up query |
| 99 | + - Expected: the server stays connected across multiple calls |
105 | 100 |
|
106 | | -- [ ] **T3.2**: Build the index: |
107 | | - ```bash |
108 | | - uvx mcp-server-python-docs build-index --versions 3.12,3.13 |
109 | | - ``` |
110 | | - - **Expected**: Downloads objects.inv files, builds index, prints success message |
111 | | - - **Actual result**: _______________ |
| 101 | +## Test 4: Fresh install verification |
112 | 102 |
|
113 | | -- [ ] **T3.3**: Run doctor: |
114 | | - ```bash |
115 | | - uvx mcp-server-python-docs doctor |
116 | | - ``` |
117 | | - - **Expected**: All checks PASS |
118 | | - - **Actual result**: _______________ |
| 103 | +Use this when validating the published package or a clean local environment. |
119 | 104 |
|
120 | | -- [ ] **T3.4**: Verify the full README install flow works end-to-end |
121 | | - - **Expected**: Following README instructions from scratch produces a working server |
122 | | - - **Actual result**: _______________ |
123 | | - |
124 | | -### Teardown |
125 | | - |
126 | | -```bash |
127 | | -# Clean up throwaway venv if used |
128 | | -rm -rf /tmp/test-install |
129 | | -``` |
| 105 | +On Windows, close the MCP client before rebuilding if the live index file is |
| 106 | +locked. |
130 | 107 |
|
131 | | -## Sign-Off |
| 108 | +### Checks |
132 | 109 |
|
133 | | -| Test | Pass/Fail | Tester | Date | |
134 | | -|------|-----------|--------|------| |
135 | | -| T1: Claude Desktop | | | | |
136 | | -| T2: Cursor | | | | |
137 | | -| T3: Fresh Install | | | | |
| 110 | +- [ ] `uvx mcp-server-python-docs --version` |
| 111 | + - Expected: prints the current package version |
| 112 | +- [ ] `uvx mcp-server-python-docs build-index --versions 3.12,3.13` |
| 113 | + - Expected: index build completes successfully |
| 114 | +- [ ] `uvx mcp-server-python-docs doctor` |
| 115 | + - Expected: all required checks pass |
| 116 | +- [ ] Follow the README from scratch |
| 117 | + - Expected: a new user can get to a working client configuration without using `.planning/` |
138 | 118 |
|
139 | | -**Release approved**: [ ] Yes / [ ] No -- needs fixes |
| 119 | +## Evidence log |
140 | 120 |
|
141 | | -**Notes**: |
| 121 | +| Test | Pass/Fail | Tester | Date | Notes | |
| 122 | +|------|-----------|--------|------|-------| |
| 123 | +| Inspector quick loop | | | | | |
| 124 | +| Claude Desktop | | | | | |
| 125 | +| Cursor | | | | | |
| 126 | +| Fresh install | | | | | |
0 commit comments