Skip to content

Commit a3e0411

Browse files
docs: note that GPU pytest runs must be serialised
Captures the operational gotcha discovered while preparing v0.2.0: the default 24-worker pytest-xdist parallelism opens 24 CUDA contexts on the single visible GPU and triggers spurious CUBLAS_STATUS_ALLOC_FAILED / OutOfMemoryError failures even on 96 GB cards. The same suite passes cleanly under -n 0. Updated: - README.md (Contributing block) - CLAUDE.md (Build and Test Commands) - AGENTS.md (testing block) - .cursor/rules/tests.mdc (testing rule)
1 parent e9301ae commit a3e0411

4 files changed

Lines changed: 24 additions & 0 deletions

File tree

.cursor/rules/tests.mdc

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,3 +31,6 @@ Users run:
3131
- `pytest -m "not gpu"` -- CPU-only environments
3232
- `pytest -m "not benchmark"` -- fast CI (correctness only)
3333
- `pytest -m "benchmark and not slow"` -- fast perf checks
34+
- `pytest -m "gpu and not slow" -n 0` -- GPU agreement (serial)
35+
36+
**GPU runs must use `-n 0`** (single worker). The 24-worker pytest-xdist default opens 24 CUDA contexts on the single visible GPU and triggers spurious `CUBLAS_STATUS_ALLOC_FAILED` / `OutOfMemoryError` failures. The same suite passes cleanly when serialised.

AGENTS.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,8 +46,15 @@ pytest -m "benchmark and not slow" # fast benchmarks only
4646
pytest -m "not gpu" # CPU-only run
4747
pytest -m "benchmark and gpu" # all GPU-exercising benchmarks
4848
pytest -m "not (slow or benchmark)" # minimum fast CI subset
49+
pytest -m "gpu and not slow" -n 0 # GPU agreement (serial -- see note)
4950
```
5051

52+
> **Always pass `-n 0` to GPU-marked runs.** The default 24-worker
53+
> pytest-xdist parallelism opens 24 simultaneous CUDA contexts on the
54+
> single visible GPU and triggers spurious `CUBLAS_STATUS_ALLOC_FAILED`
55+
> / `OutOfMemoryError` failures even on 96 GB cards. The same suite
56+
> passes cleanly under `-n 0` (or `-n 1`).
57+
5158
Register any new markers in `pyproject.toml` `[tool.pytest.ini_options].markers`.
5259

5360
After modifying any production code under `src/mdpp/`, you MUST complete the following loop before considering the task done. Repeat until no CRITICAL issues remain:

CLAUDE.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,13 @@ pytest -m benchmark
108108
# Skip slow + benchmark tests
109109
pytest -m "not (slow or benchmark)"
110110

111+
# GPU backend agreement (cupy / torch / jax vs mdtraj / numba).
112+
# Always pass `-n 0` -- the default 24-worker pytest-xdist parallelism
113+
# opens 24 CUDA contexts on a single GPU and produces spurious
114+
# CUBLAS_STATUS_ALLOC_FAILED / OutOfMemoryError failures even on
115+
# 96 GB cards. Serial runs are clean.
116+
pytest -m "gpu and not slow" -n 0
117+
111118
# Lint and format
112119
ruff check src/ tests/ --fix
113120
ruff format src/ tests/

README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -242,13 +242,20 @@ pytest # full suite
242242
pytest -m "not benchmark" # skip perf benchmarks
243243
pytest -m "not gpu" # skip GPU backend tests (CPU-only)
244244
pytest -m "benchmark and not slow" # fast performance checks
245+
pytest -m "gpu and not slow" -n 0 # GPU backend agreement (serial, see note)
245246

246247
# Full pre-commit suite
247248
pre-commit run --all-files
248249
```
249250

250251
Custom pytest markers: `benchmark` (timing), `slow` (>10s), `gpu` (cupy/torch/jax).
251252

253+
> **GPU test note:** Always run GPU-marked tests with `-n 0` (one worker).
254+
> The default 24-worker pytest-xdist parallelism opens 24 simultaneous CUDA
255+
> contexts on the single visible GPU and triggers spurious
256+
> `CUBLAS_STATUS_ALLOC_FAILED` / `OutOfMemoryError` failures even on a
257+
> 96 GB card. Serial runs are clean.
258+
252259
## License
253260

254261
[MIT](LICENSE) — Zhaoyang Li

0 commit comments

Comments
 (0)