Skip to content

Commit e797064

Browse files
eleanorjboydeleanorjboyd
committed
Add project support for pytest execution
1 parent 8f413a2 commit e797064

File tree

11 files changed

+3100
-722
lines changed

11 files changed

+3100
-722
lines changed

‎docs/test-plan-project-based-execution.md‎

Lines changed: 511 additions & 0 deletions
Large diffs are not rendered by default.

‎plan-project-based-exec.md‎

Lines changed: 217 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,217 @@
1+
# Plan: Project-Based Pytest Execution
2+
3+
## Overview
4+
5+
This plan describes the implementation of **project-based test execution for pytest**, enabling multi-project workspace support where each Python project within a workspace can execute tests using its own Python environment. This builds on top of the project-based discovery work from PR #25760.
6+
7+
## Problem to Solve
8+
9+
In a multi-project workspace (e.g., a monorepo with multiple Python services), users currently cannot:
10+
- Run tests with the correct Python interpreter for each project
11+
- Have separate test trees per project in the Test Explorer
12+
- Properly handle nested projects (parent/child)
13+
14+
## Architecture
15+
16+
### Key Components to Add
17+
18+
| Component | File | Purpose |
19+
| ----------------------- | -------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
20+
| **TestProjectRegistry** | [testProjectRegistry.ts](../src/client/testing/testController/common/testProjectRegistry.ts) | Registry that discovers and manages Python projects per workspace |
21+
| **ProjectAdapter** | [projectAdapter.ts](../src/client/testing/testController/common/projectAdapter.ts) | Interface representing a single Python project with its test infrastructure |
22+
| **projectUtils** | [projectUtils.ts](../src/client/testing/testController/common/projectUtils.ts) | Utility functions for project ID generation and adapter creation |
23+
24+
### How It Works
25+
26+
```
27+
┌─────────────────────────────────────────────────────────────────┐
28+
│ VS Code Workspace │
29+
│ ┌─────────────────────────────────────────────────────────────┐│
30+
│ │ TestController ││
31+
│ │ ┌───────────────────────────────────────────────────────┐ ││
32+
│ │ │ TestProjectRegistry │ ││
33+
│ │ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ ││
34+
│ │ │ │ ProjectA │ │ ProjectB │ │ ProjectC │ │ ││
35+
│ │ │ │ (Py 3.11) │ │ (Py 3.12) │ │ (Py 3.10) │ │ ││
36+
│ │ │ │ ┌─────────┐ │ │ ┌─────────┐ │ │ ┌─────────┐ │ │ ││
37+
│ │ │ │ │Discovery│ │ │ │Discovery│ │ │ │Discovery│ │ │ ││
38+
│ │ │ │ │Adapter │ │ │ │Adapter │ │ │ │Adapter │ │ │ ││
39+
│ │ │ │ ├─────────┤ │ │ ├─────────┤ │ │ ├─────────┤ │ │ ││
40+
│ │ │ │ │Execution│ │ │ │Execution│ │ │ │Execution│ │ │ ││
41+
│ │ │ │ │Adapter │ │ │ │Adapter │ │ │ │Adapter │ │ │ ││
42+
│ │ │ │ └─────────┘ │ │ └─────────┘ │ │ └─────────┘ │ │ ││
43+
│ │ │ └─────────────┘ └─────────────┘ └─────────────┘ │ ││
44+
│ │ └───────────────────────────────────────────────────────┘ ││
45+
│ └─────────────────────────────────────────────────────────────┘│
46+
└─────────────────────────────────────────────────────────────────┘
47+
```
48+
49+
### Execution Flow
50+
51+
1. **User runs tests** → `TestRunRequest` with selected `TestItem`s arrives
52+
2. **Controller** checks if project-based testing is enabled
53+
3. **Group tests by project** → Tests are sorted by which `ProjectAdapter` they belong to (via URI matching)
54+
4. **Execute per project** → Each project's `executionAdapter.runTests()` is called with:
55+
- The project's Python environment
56+
- `PROJECT_ROOT_PATH` environment variable set to project root
57+
5. **Results collected** → Each project's `resultResolver` maps results back to test items
58+
59+
### Required Changes by File
60+
61+
#### Controller ([controller.ts](../src/client/testing/testController/controller.ts))
62+
- Add `TestProjectRegistry` integration
63+
- New methods: `discoverForProject()`, `executeTestsForProjects()`, `groupTestItemsByProject()`
64+
- Debug mode should handle multi-project scenarios by launching multiple debug sessions
65+
66+
#### Pytest Execution Adapter ([pytestExecutionAdapter.ts](../src/client/testing/testController/pytest/pytestExecutionAdapter.ts))
67+
- Add `project?: ProjectAdapter` parameter to `runTests()`
68+
- Set `PROJECT_ROOT_PATH` environment variable when project is provided
69+
- Use project's Python environment instead of workspace environment
70+
- Debug launches should use `pythonPath` from project when available
71+
72+
#### Debug Launcher ([debugLauncher.ts](../src/client/testing/common/debugLauncher.ts))
73+
- Add optional `pythonPath` to `LaunchOptions` for project-specific interpreter
74+
- Add optional `debugSessionName` to `LaunchOptions` for session identification
75+
- Debug sessions should use explicit Python path when provided
76+
- Use unique session markers to track individual debug sessions (avoids `activeDebugSession` race conditions)
77+
- Properly dispose event handlers when debugging completes
78+
79+
#### Python Side ([vscode_pytest/__init__.py](../python_files/vscode_pytest/__init__.py))
80+
- `get_test_root_path()` should return `PROJECT_ROOT_PATH` env var if set (otherwise cwd)
81+
- Session node should use project root for test tree structure
82+
83+
## Feature Behavior
84+
85+
### Single Project Workspace
86+
No change from existing behavior—tests run using the workspace's interpreter.
87+
88+
### Multi-Project Workspace
89+
- Each project has its own root node in Test Explorer
90+
- Running tests uses the correct interpreter for each project
91+
- Results are scoped to the correct project
92+
93+
### Nested Projects
94+
```
95+
workspace/
96+
└── parent-project/
97+
├── tests/
98+
└── child-project/
99+
└── tests/
100+
```
101+
- Parent project discovery ignores child project via `--ignore` flags
102+
- Execution receives specific test IDs, so no cross-contamination
103+
104+
### Debug Mode
105+
- **Single project**: Debug should proceed normally with project interpreter
106+
- **Multiple projects**: Multiple debug sessions should be launched in parallel—one per project, each using its own interpreter
107+
- **Session naming**: Each debug session includes the project name (e.g., "Debug Tests: alice (Python 3.11)")
108+
- **Session isolation**: Each debug session is tracked independently using unique markers, so stopping one session doesn't affect others
109+
110+
### Cancellation Handling
111+
112+
Cancellation is handled at multiple levels to ensure proper cleanup across all parallel project executions:
113+
114+
```
115+
┌─────────────────────────────────────────────────────────────────┐
116+
│ User Clicks "Stop" │
117+
│ │ │
118+
│ ▼ │
119+
│ CancellationToken fires │
120+
│ │ │
121+
│ ┌─────────────────┼─────────────────┐ │
122+
│ ▼ ▼ ▼ │
123+
│ ┌───────────┐ ┌───────────┐ ┌───────────┐ │
124+
│ │ Project A │ │ Project B │ │ Project C │ │
125+
│ │ Execution │ │ Execution │ │ Execution │ │
126+
│ └─────┬─────┘ └─────┬─────┘ └─────┬─────┘ │
127+
│ │ │ │ │
128+
│ ▼ ▼ ▼ │
129+
│ Kill subprocess Kill subprocess Kill subprocess │
130+
│ Close pipes Close pipes Close pipes │
131+
│ Resolve deferred Resolve deferred Resolve deferred │
132+
└─────────────────────────────────────────────────────────────────┘
133+
```
134+
135+
#### Cancellation Levels
136+
137+
1. **Project execution level** ([projectTestExecution.ts](src/client/testing/testController/common/projectTestExecution.ts))
138+
- Early exit if cancelled before starting
139+
- Checks cancellation before starting each project's execution
140+
- Projects not yet started are skipped gracefully
141+
142+
2. **Execution adapter level** ([pytestExecutionAdapter.ts](src/client/testing/testController/pytest/pytestExecutionAdapter.ts))
143+
- `runInstance.token.onCancellationRequested` kills the subprocess
144+
- Named pipe server is closed via the callback
145+
- Deferred promises resolve to unblock waiting code
146+
147+
3. **Debug launcher level** ([debugLauncher.ts](src/client/testing/common/debugLauncher.ts))
148+
- Token cancellation resolves the deferred and invokes cleanup callback
149+
- Session termination events are filtered to only react to the correct session
150+
- Event handlers are disposed when debugging completes
151+
152+
#### Multi-Session Debug Independence
153+
154+
When debugging multiple projects simultaneously, each `launchDebugger()` call must track its own debug session independently. The implementation uses a unique marker system:
155+
156+
```typescript
157+
// Each debug session gets a unique marker in its configuration
158+
const sessionMarker = `test-${Date.now()}-${random}`;
159+
launchArgs[TEST_SESSION_MARKER_KEY] = sessionMarker;
160+
161+
// When sessions start/terminate, we match by marker (not activeDebugSession)
162+
onDidStartDebugSession((session) => {
163+
if (session.configuration[TEST_SESSION_MARKER_KEY] === sessionMarker) {
164+
ourSession = session; // Found our specific session
165+
}
166+
});
167+
```
168+
169+
This avoids race conditions where the global `activeDebugSession` could be overwritten by another concurrent session start.
170+
171+
### Legacy Fallback
172+
When Python Environments API is unavailable, the system falls back to single-workspace adapter mode.
173+
174+
## Files to Change
175+
176+
| Category | Files |
177+
| ----------------------- | ------------------------------------------------------------------------------------------------------------ |
178+
| **Core Implementation** | `controller.ts`, `testProjectRegistry.ts`, `projectAdapter.ts`, `projectUtils.ts`, `projectTestExecution.ts` |
179+
| **Adapters** | `pytestExecutionAdapter.ts`, `pytestDiscoveryAdapter.ts`, `resultResolver.ts` |
180+
| **Types** | `types.ts` (common), `types.ts` (testController) |
181+
| **Debug** | `debugLauncher.ts` |
182+
| **Python** | `vscode_pytest/__init__.py` |
183+
| **Tests** | `controller.unit.test.ts`, `testProjectRegistry.unit.test.ts`, `projectUtils.unit.test.ts` |
184+
185+
## Testing
186+
187+
### Unit Tests to Add
188+
- `testProjectRegistry.unit.test.ts` - Registry lifecycle, project discovery, nested projects
189+
- `controller.unit.test.ts` - Controller integration, debug scenarios, test grouping
190+
- `projectUtils.unit.test.ts` - Utility functions
191+
192+
### Test Scenarios to Cover
193+
| Scenario | Coverage |
194+
| ----------------------------- | --------------------------------------------- |
195+
| Single project workspace | Unit tests + legacy flows |
196+
| Multi-project workspace | New controller unit tests |
197+
| Nested projects | Discovery tests + ignore behavior |
198+
| Debug mode (single project) | Existing debug tests |
199+
| Debug mode (multi-project) | Session isolation, independent cancellation |
200+
| Legacy fallback | Existing controller tests |
201+
| Test cancellation | Cancellation at all levels (see above) |
202+
203+
## Out of Scope
204+
- **Unittest support**: Project-based unittest execution will be handled in a separate PR
205+
- **End-to-end tests**: Manual testing will be required for full validation
206+
- **Multi-project coverage aggregation**: Deferred to future work
207+
208+
## Expected User Experience
209+
210+
### Debugging Across Multiple Projects
211+
When debugging tests spanning multiple projects:
212+
- Multiple debug sessions should be launched simultaneously—one per project
213+
- Each debug session should use the project's configured Python interpreter
214+
- All projects' tests should run in debug mode in parallel
215+
- Users should be able to switch between debug sessions in VS Code's debug panel
216+
- **Stopping one debug session should NOT affect other running sessions**
217+
- Each debug session is named with its project (e.g., "Debug Tests: alice (Python 3.11)")

0 commit comments

Comments
 (0)