You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: note that GPU pytest runs must be serialised
Captures the operational gotcha discovered while preparing v0.2.0:
the default 24-worker pytest-xdist parallelism opens 24 CUDA
contexts on the single visible GPU and triggers spurious
CUBLAS_STATUS_ALLOC_FAILED / OutOfMemoryError failures even on
96 GB cards. The same suite passes cleanly under -n 0.
Updated:
- README.md (Contributing block)
- CLAUDE.md (Build and Test Commands)
- AGENTS.md (testing block)
- .cursor/rules/tests.mdc (testing rule)
Copy file name to clipboardExpand all lines: .cursor/rules/tests.mdc
+3Lines changed: 3 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -31,3 +31,6 @@ Users run:
31
31
- `pytest -m "not gpu"` -- CPU-only environments
32
32
- `pytest -m "not benchmark"` -- fast CI (correctness only)
33
33
- `pytest -m "benchmark and not slow"` -- fast perf checks
34
+
- `pytest -m "gpu and not slow" -n 0` -- GPU agreement (serial)
35
+
36
+
**GPU runs must use `-n 0`** (single worker). The 24-worker pytest-xdist default opens 24 CUDA contexts on the single visible GPU and triggers spurious `CUBLAS_STATUS_ALLOC_FAILED` / `OutOfMemoryError` failures. The same suite passes cleanly when serialised.
Copy file name to clipboardExpand all lines: AGENTS.md
+7Lines changed: 7 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -46,8 +46,15 @@ pytest -m "benchmark and not slow" # fast benchmarks only
46
46
pytest -m "not gpu"# CPU-only run
47
47
pytest -m "benchmark and gpu"# all GPU-exercising benchmarks
48
48
pytest -m "not (slow or benchmark)"# minimum fast CI subset
49
+
pytest -m "gpu and not slow" -n 0 # GPU agreement (serial -- see note)
49
50
```
50
51
52
+
> **Always pass `-n 0` to GPU-marked runs.** The default 24-worker
53
+
> pytest-xdist parallelism opens 24 simultaneous CUDA contexts on the
54
+
> single visible GPU and triggers spurious `CUBLAS_STATUS_ALLOC_FAILED`
55
+
> / `OutOfMemoryError` failures even on 96 GB cards. The same suite
56
+
> passes cleanly under `-n 0` (or `-n 1`).
57
+
51
58
Register any new markers in `pyproject.toml``[tool.pytest.ini_options].markers`.
52
59
53
60
After modifying any production code under `src/mdpp/`, you MUST complete the following loop before considering the task done. Repeat until no CRITICAL issues remain:
0 commit comments