Make it possible to run Pytest in parallel (#1291)

This PR makes it possible to run parallel Pytest execution via the
`pytest-xdist` plugin.

* Set a fixed random seed for testing, in `src/openfermion/conftest.py`.
This is necessary for `pytest-xdist` worker processes to be able to
perform identical test collections. Otherwise, different workers might
generate different values and you end up with "Inconsistent collection"
errors during execution.

* Change some tests that were writing to fixed-path files and folders.
Those needed to be changed to use temporary directories so that parallel
Pytest processes don't collide trying to write to the same things.

* Add the pytest plugin `pytest-randomly`, which executes tests in
random order, as a way to help make sure that the changes above don't
have some hidden assumption about order of execution.

* Update the requirements files by running the
`re-pip-compile-in-docker.sh` script (necessary because of the addition
of the `pytest-randomly` package).
 
* Update `CONTRIBUTING.md` to mention the use of options for parallel
testing.

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Author: mhucka

.github/workflows/ci.yaml

@@ -219,7 +219,7 @@ jobs:
       - name: Run pytest
         run: |
           echo '::add-matcher::.github/problem-matchers/pytest.json'
-          check/pytest -m "not slow"
+          check/pytest -n auto -m "not slow"
 
   pytest-extra:
     if: needs.changes.outputs.python == 'true'
@@ -254,7 +254,7 @@ jobs:
       - name: Run pytest
         run: |
           echo '::add-matcher::.github/problem-matchers/pytest.json'
-          check/pytest -m "not slow" src/openfermion/resource_estimates
+          check/pytest -n auto -m "not slow" src/openfermion/resource_estimates
 
   pytest-compat:
     if: needs.changes.outputs.python == 'true'
@@ -283,7 +283,7 @@ jobs:
       - name: Run pytest
         run: |
           echo '::add-matcher::.github/problem-matchers/pytest.json'
-          check/pytest -m "not slow"
+          check/pytest -n auto -m "not slow"
 
   coverage:
     if: needs.changes.outputs.python == 'true'

CONTRIBUTING.md

@@ -210,16 +210,6 @@ pre-commit run --all-files
 
 After that, the hooks will run automatically when triggered by the corresponding git operations.
 
-### Type annotation conventions
-
-Code should have [type annotations](https://www.python.org/dev/peps/pep-0484/). We use
-[mypy](http://mypy-lang.org/) to check that type annotations are correct, and the following script
-to run it:
-
-```shell
-check/mypy
-```
-
 ### Python setup
 
 1.  Create a Python virtual environment. To use Python's built-in `venv` package, run:
@@ -239,6 +229,20 @@ check/mypy
 Please refer to the section _Developer install_ of the [installation instructions](docs/install.md)
 for information about how to set up a local copy of the software for development.
 
+### Type annotation conventions
+
+Code should have [type annotations](https://www.python.org/dev/peps/pep-0484/). We use
+[mypy](http://mypy-lang.org/) to check that type annotations are correct, and the following script
+to run it:
+
+```shell
+check/mypy
+```
+
+If your computer has multiple processor cores, you can add the option `-j 0` to the command above to
+make Mypy run in parallel for a substantial speed increase.
+
+
 ### Linting and formatting
 
 Code should meet common style standards for Python and be free of error-prone constructs. We use
@@ -282,7 +286,9 @@ We use [pytest](https://docs.pytest.org) to run our tests and
 *   While developing, periodically check that changes do not break anything. For fast checks, use
     `pytest -m "not slow" PATH`, where `PATH` is a directory or pytest file to test.
 
-*   After finishing a task, run `check/pytest` to test all of the OpenFermion code.
+*   After finishing a task, run `check/pytest` to test all of the OpenFermion code. If your system
+    has multiple processor cores, you can add the option `-n auto` to make it run in parallel for a
+    substantial speed increase. (Beware, though, that this is resource-intensive.)
 
 We don't require 100% coverage, but coverage should be very high, and any uncovered code must be
 annotated with `# pragma: no cover`. To ignore coverage of a single line, place `# pragma: no cover`
@@ -294,9 +300,9 @@ cover` comment on its own line. Note, however, that these annotations should be
 After a task is finished, run each of the following to make sure everything passes all the tests:
 
 *   `check/format-incremental`
-*   `check/pylint`
+*   `check/pylint -j 0`
 *   `check/mypy`
-*   `check/pytest`
+*   `check/pytest -n auto`
 *   `check/pytest-and-incremental-coverage`
 
 ### Pull requests and code reviews

check/pytest-and-incremental-coverage

@@ -68,6 +68,7 @@ fi
 
 # Run tests while producing coverage files.
 check/pytest . \
+    -n auto \
     --actually-quiet \
     --cov \
     --cov-report=annotate

dev_tools/requirements/deps/pytest.txt

@@ -1,6 +1,7 @@
 pytest
 pytest-asyncio
 pytest-cov
+pytest-randomly
 pytest-retry
 pytest-xdist
 

dev_tools/requirements/envs/dev.env.txt

@@ -19,15 +19,15 @@ backports-asyncio-runner==1.2.0 ; python_full_version < "3.11"
     #   pytest-asyncio
 black==26.3.1
     # via -r deps/format.txt
-build==1.4.2
+build==1.4.4
     # via pip-tools
-certifi==2026.2.25
+certifi==2026.4.22
     # via requests
 charset-normalizer==3.4.7
     # via requests
 cirq-core==1.5.0
     # via -r deps/runtime.txt
-click==8.3.1
+click==8.3.3
     # via
     #   black
     #   pip-tools
@@ -57,7 +57,7 @@ h5py==3.16.0
     # via
     #   -r deps/runtime.txt
     #   pyscf
-idna==3.11
+idna==3.13
     # via requests
 iniconfig==2.3.0
     # via pytest
@@ -77,9 +77,9 @@ jupyter-core==5.9.1
     # via nbformat
 kiwisolver==1.5.0
     # via matplotlib
-librt==0.8.1
+librt==0.9.0
     # via mypy
-matplotlib==3.10.8
+matplotlib==3.10.9
     # via
     #   ase
     #   cirq-core
@@ -91,7 +91,7 @@ ml-dtypes==0.5.4
     #   jaxlib
 mpmath==1.3.0
     # via sympy
-mypy==1.20.0
+mypy==1.20.2
     # via -r deps/mypy.txt
 mypy-extensions==1.1.0
     # via
@@ -121,7 +121,7 @@ numpy==2.2.6
     #   types-networkx
 opt-einsum==3.4.0
     # via jax
-packaging==26.0
+packaging==26.2
     # via
     #   black
     #   build
@@ -133,15 +133,15 @@ pandas==2.3.3
     # via cirq-core
 pandas-stubs==2.3.3.260113
     # via -r deps/mypy.txt
-pathspec==1.0.4
+pathspec==1.1.1
     # via
     #   black
     #   mypy
 pillow==12.2.0
     # via matplotlib
 pip-tools==7.5.3
     # via -r deps/pip-tools.txt
-platformdirs==4.9.4
+platformdirs==4.9.6
     # via
     #   black
     #   jupyter-core
@@ -162,19 +162,22 @@ pyproject-hooks==1.2.0
     # via
     #   build
     #   pip-tools
-pyscf==2.12.1
+pyscf==2.13.0
     # via -r deps/resource_estimates_runtime.txt
-pytest==9.0.2
+pytest==9.0.3
     # via
     #   -r deps/pytest.txt
     #   pytest-asyncio
     #   pytest-cov
+    #   pytest-randomly
     #   pytest-retry
     #   pytest-xdist
 pytest-asyncio==1.3.0
     # via -r deps/pytest.txt
 pytest-cov==7.1.0
     # via -r deps/pytest.txt
+pytest-randomly==4.1.0
+    # via -r deps/pytest.txt
 pytest-retry==1.7.0
     # via -r deps/pytest.txt
 pytest-xdist==3.8.0
@@ -230,13 +233,13 @@ traitlets==5.14.3
     # via
     #   jupyter-core
     #   nbformat
-types-networkx==3.6.1.20260402
+types-networkx==3.6.1.20260408
     # via -r deps/mypy.txt
-types-pytz==2026.1.1.20260402
+types-pytz==2026.1.1.20260408
     # via pandas-stubs
-types-requests==2.33.0.20260402
+types-requests==2.33.0.20260408
     # via -r deps/mypy.txt
-types-setuptools==82.0.0.20260402
+types-setuptools==82.0.0.20260408
     # via -r deps/mypy.txt
 typing-extensions==4.15.0
     # via
@@ -247,13 +250,13 @@ typing-extensions==4.15.0
     #   mypy
     #   pytest-asyncio
     #   referencing
-tzdata==2025.3
+tzdata==2026.2
     # via pandas
 urllib3==2.6.3
     # via
     #   requests
     #   types-requests
-wheel==0.46.3
+wheel==0.47.0
     # via pip-tools
 
 # The following packages are considered to be unsafe in a requirements file:

dev_tools/requirements/envs/format.env.txt

@@ -16,7 +16,7 @@ black==26.3.1
     # via
     #   -c envs/dev.env.txt
     #   -r deps/format.txt
-certifi==2026.2.25
+certifi==2026.4.22
     # via
     #   -c envs/dev.env.txt
     #   requests
@@ -28,7 +28,7 @@ cirq-core==1.5.0
     # via
     #   -c envs/dev.env.txt
     #   -r deps/runtime.txt
-click==8.3.1
+click==8.3.3
     # via
     #   -c envs/dev.env.txt
     #   black
@@ -57,15 +57,15 @@ h5py==3.16.0
     # via
     #   -c envs/dev.env.txt
     #   -r deps/runtime.txt
-idna==3.11
+idna==3.13
     # via
     #   -c envs/dev.env.txt
     #   requests
 kiwisolver==1.5.0
     # via
     #   -c envs/dev.env.txt
     #   matplotlib
-matplotlib==3.10.8
+matplotlib==3.10.9
     # via
     #   -c envs/dev.env.txt
     #   cirq-core
@@ -92,7 +92,7 @@ numpy==2.2.6
     #   matplotlib
     #   pandas
     #   scipy
-packaging==26.0
+packaging==26.2
     # via
     #   -c envs/dev.env.txt
     #   black
@@ -102,15 +102,15 @@ pandas==2.3.3
     # via
     #   -c envs/dev.env.txt
     #   cirq-core
-pathspec==1.0.4
+pathspec==1.1.1
     # via
     #   -c envs/dev.env.txt
     #   black
 pillow==12.2.0
     # via
     #   -c envs/dev.env.txt
     #   matplotlib
-platformdirs==4.9.4
+platformdirs==4.9.6
     # via
     #   -c envs/dev.env.txt
     #   black
@@ -170,7 +170,7 @@ typing-extensions==4.15.0
     #   -c envs/dev.env.txt
     #   black
     #   cirq-core
-tzdata==2025.3
+tzdata==2026.2
     # via
     #   -c envs/dev.env.txt
     #   pandas