Add Python 3.15 to the test matrix (#2108)

* Add Python 3.15 to the test matrix

* Update cibuildwheel

* Remove matplotlib dependency

* Remove cffi as a test dependency

* Use LOCAL_CTK=0 for Python 3.15

* Pre-build numpy for Python 3.15; exclude 3.15 from releases

numpy 2.4.x has no pre-built wheel for Python 3.15 yet, and its
[tool.cibuildwheel] config is incompatible with cibuildwheel v4.0.
Build numpy from patched sdist via cibuildwheel with -Dallow-noblas=true
on all platforms (linux-64, linux-aarch64, win-64).

Also exclude python315 artifacts from release publishing since 3.15
is pre-release and not ready for PyPI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Drop CIBW_REPAIR_WHEEL_COMMAND_WINDOWS override; version-gate matplotlib/cffi

Remove the empty CIBW_REPAIR_WHEEL_COMMAND_WINDOWS that was overriding the
pyproject.toml delvewheel config. The pyproject.toml already specifies
`delvewheel repair --namespace-pkg cuda` which is the correct behavior.

Keep matplotlib and cffi as test dependencies but gate them to
python_version < 3.15 (no wheels available yet for 3.15).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Tighten allow-prereleases to Python 3.15 only

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Skip musllinux when building numpy wheel

Main's CIBW_BUILD is cp315-* (not cp315-manylinux* like 12.9.x), so
cibuildwheel builds both manylinux and musllinux numpy wheels. The
musllinux wheel is incompatible with the glibc-based test runners.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix Windows numpy build; allow 3.15 test failures

- Skip musllinux and win32 when building numpy (CIBW_BUILD=cp315-*
  matches both win32 and win_amd64; numpy has no pyproject.toml
  constraint after we strip its cibuildwheel config)
- Mark 3.15/3.15t test jobs as continue-on-error so they don't block
  CI (3.15t segfaults on CPython beta, rlcompleter test changed behavior)
- Skip test_opt_out_env_var_disables_patch_even_when_interactive on
  Python 3.15 (upstream fixed the rlcompleter bug this test relied on)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Skip nvidia-cutlass on Python 3.15 (scipy unavailable)

nvidia-cutlass depends on scipy, which has no pre-built wheel for
Python 3.15 yet. Version-gate the dependency so pathfinder tests
can still install the rest of the test dependencies on 3.15.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add TODO comments for all Python 3.15 workarounds

Mark every temporary 3.15 hack (version-gated deps, numpy pre-build,
continue-on-error, allow-prereleases, cpython-prerelease, release
exclusion) with a TODO so they are easy to find and remove once 3.15
is officially supported.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Skip cutlass/cute header tests on Python 3.15

nvidia-cutlass is not installed on 3.15 (scipy has no 3.15 wheel),
so the pathfinder header discovery tests for cutlass and cute will
always fail under all_must_work strictness. Skip those parametrized
cases on 3.15+.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix ruff SIM118: use `in dict` instead of `in dict.keys()`

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* disable py315t pipeline for now

---------

Co-authored-by: Leo Fang <leof@nvidia.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Author: 3 people

.github/workflows/build-wheel.yml

@@ -34,6 +34,8 @@ jobs:
           - "3.13"
           - "3.14"
           - "3.14t"
+          - "3.15"
+          - "3.15t"
     name: py${{ matrix.python-version }}
     runs-on: ${{ (inputs.host-platform == 'linux-64' && 'linux-amd64-cpu8') ||
                  (inputs.host-platform == 'linux-aarch64' && 'linux-arm64-cpu8') ||
@@ -165,12 +167,17 @@ jobs:
           cuda-version: ${{ inputs.cuda-version }}
 
       - name: Build cuda.bindings wheel
-        uses: pypa/cibuildwheel@8d2b08b68458a16aeb24b64e68a09ab1c8e82084  # v3.4.1
+        uses: pypa/cibuildwheel@54327ab9d35de03b359ac25c97de9417d94639c0  # v4.0.0rc1
         with:
           package-dir: ./cuda_bindings/
           output-dir: ${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }}
         env:
           CIBW_BUILD: ${{ env.CIBW_BUILD }}
+          # TODO: remove cpython-prerelease once 3.15 is officially supported
+          # Allow CPython pre-release builds (currently 3.15 / 3.15t). This is a
+          # no-op for stable Python versions because CIBW_BUILD still filters
+          # the target version.
+          CIBW_ENABLE: cpython-prerelease
           # CIBW mounts the host filesystem under /host
           CIBW_ENVIRONMENT_LINUX: >
             CUDA_PATH=/host/${{ env.CUDA_PATH }}
@@ -229,12 +236,17 @@ jobs:
           if-no-files-found: error
 
       - name: Build cuda.core wheel
-        uses: pypa/cibuildwheel@8d2b08b68458a16aeb24b64e68a09ab1c8e82084  # v3.4.1
+        uses: pypa/cibuildwheel@54327ab9d35de03b359ac25c97de9417d94639c0  # v4.0.0rc1
         with:
           package-dir: ./cuda_core/
           output-dir: ${{ env.CUDA_CORE_ARTIFACTS_DIR }}
         env:
           CIBW_BUILD: ${{ env.CIBW_BUILD }}
+          # TODO: remove cpython-prerelease once 3.15 is officially supported
+          # Allow CPython pre-release builds (currently 3.15 / 3.15t). This is a
+          # no-op for stable Python versions because CIBW_BUILD still filters
+          # the target version.
+          CIBW_ENABLE: cpython-prerelease
           # CIBW mounts the host filesystem under /host
           CIBW_ENVIRONMENT_LINUX: >
             CUDA_PATH=/host/${{ env.CUDA_PATH }}
@@ -330,6 +342,8 @@ jobs:
         uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6.2.0
         with:
           python-version: ${{ matrix.python-version }}
+          # TODO: remove allow-prereleases once 3.15 is officially supported
+          allow-prereleases: ${{ startsWith(matrix.python-version, '3.15') }}
 
       - name: verify free-threaded build
         if: endsWith(matrix.python-version, 't')
@@ -349,6 +363,71 @@ jobs:
         run: |
           pip install cuda_pathfinder/*.whl
 
+      - name: Hide GNU link.exe so Meson finds MSVC link.exe
+        if: ${{ startsWith(inputs.host-platform, 'win') }}
+        run: |
+          if [ -f "/c/Program Files/Git/usr/bin/link.exe" ]; then
+            mv "/c/Program Files/Git/usr/bin/link.exe" "/c/Program Files/Git/usr/bin/link.exe.bak"
+          fi
+
+      # TODO: remove the numpy pre-build steps once 3.15 is officially supported
+      # (numpy will publish pre-built 3.15 wheels at that point)
+      - name: Download and patch numpy sdist (pre-release Python)
+        if: ${{ startsWith(matrix.python-version, '3.15') }}
+        run: |
+          pip download --no-binary numpy --no-deps "numpy>=1.21.1" -d numpy-sdist/
+          cd numpy-sdist && tar xf numpy-*.tar.gz && rm numpy-*.tar.gz
+          # WAR: numpy 2.4.x ships [tool.cibuildwheel] config that is
+          # incompatible with cibuildwheel v4.0 (cpython-freethreading enable
+          # group, OpenBLAS before-build scripts, etc.). Strip the cibuildwheel
+          # sections but preserve [tool.meson-python] (vendored meson path).
+          python -c "
+          import glob
+          for f in glob.glob('numpy-*/pyproject.toml'):
+              lines, skip = open(f).readlines(), False
+              out = []
+              for line in lines:
+                  hdr = line.strip()
+                  if hdr.startswith('[tool.cibuildwheel') or hdr.startswith('[[tool.cibuildwheel'):
+                      skip = True
+                      continue
+                  if skip and hdr.startswith('[') and 'cibuildwheel' not in hdr:
+                      skip = False
+                  if not skip:
+                      out.append(line)
+              open(f, 'w').writelines(out)
+          "
+          echo "NUMPY_SRC_DIR=$(pwd)/$(ls -d numpy-*/)" >> $GITHUB_ENV
+
+      - name: Build numpy wheel (pre-release Python)
+        if: ${{ startsWith(matrix.python-version, '3.15') }}
+        uses: pypa/cibuildwheel@54327ab9d35de03b359ac25c97de9417d94639c0  # v4.0.0rc1
+        env:
+          CIBW_BUILD: ${{ env.CIBW_BUILD }}
+          CIBW_SKIP: "*-musllinux* *-win32"
+          CIBW_ARCHS_LINUX: "native"
+          CIBW_BUILD_VERBOSITY: 1
+          CIBW_CONFIG_SETTINGS: "setup-args=-Dallow-noblas=true"
+          CIBW_CONFIG_SETTINGS_WINDOWS: "setup-args=--vsenv setup-args=-Dallow-noblas=true"
+          CIBW_BEFORE_BUILD_WINDOWS: "pip install delvewheel"
+          CIBW_REPAIR_WHEEL_COMMAND_WINDOWS: "delvewheel repair -w {dest_dir} {wheel}"
+          CIBW_ENABLE: "cpython-prerelease"
+        with:
+          package-dir: ${{ env.NUMPY_SRC_DIR }}
+          output-dir: numpy-wheel/
+
+      - name: Upload numpy wheel
+        if: ${{ startsWith(matrix.python-version, '3.15') }}
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a  # v7.0.1
+        with:
+          name: numpy-python${{ env.PYTHON_VERSION_FORMATTED }}-${{ inputs.host-platform }}
+          path: numpy-wheel/*.whl
+          if-no-files-found: error
+
+      - name: Install numpy wheel
+        if: ${{ startsWith(matrix.python-version, '3.15') }}
+        run: pip install numpy-wheel/*.whl
+
       - name: Build cuda.bindings Cython tests
         run: |
           pip install ${{ env.CUDA_BINDINGS_ARTIFACTS_DIR }}/*.whl --group ./cuda_bindings/pyproject.toml:test
@@ -414,12 +493,17 @@ jobs:
           rmdir $OLD_BASENAME
 
       - name: Build cuda.core wheel
-        uses: pypa/cibuildwheel@8d2b08b68458a16aeb24b64e68a09ab1c8e82084  # v3.4.1
+        uses: pypa/cibuildwheel@54327ab9d35de03b359ac25c97de9417d94639c0  # v4.0.0rc1
         with:
           package-dir: ./cuda_core/
           output-dir: ${{ env.CUDA_CORE_ARTIFACTS_DIR }}
         env:
           CIBW_BUILD: ${{ env.CIBW_BUILD }}
+          # TODO: remove cpython-prerelease once 3.15 is officially supported
+          # Allow CPython pre-release builds (currently 3.15 / 3.15t). This is a
+          # no-op for stable Python versions because CIBW_BUILD still filters
+          # the target version.
+          CIBW_ENABLE: cpython-prerelease
           # CIBW mounts the host filesystem under /host
           CIBW_ENVIRONMENT_LINUX: >
             CUDA_PATH=/host/${{ env.CUDA_PATH }}

.github/workflows/test-wheel-linux.yml

@@ -102,6 +102,8 @@ jobs:
       fail-fast: false
       matrix: ${{ fromJSON(needs.compute-matrix.outputs.MATRIX) }}
     runs-on: "${{ matrix.FLAVOR || 'linux' }}-${{ matrix.ARCH }}-gpu-${{ matrix.GPU }}-${{ matrix.DRIVER }}-${{ matrix.GPU_COUNT }}"
+    # TODO: remove continue-on-error once 3.15 is officially supported
+    continue-on-error: ${{ startsWith(matrix.PY_VER, '3.15') }}
     # The build stage could fail but we want the CI to keep moving.
     if: ${{ github.repository_owner == 'nvidia' && !cancelled() }}
     # Our self-hosted runners require a container
@@ -258,6 +260,8 @@ jobs:
         uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6.2.0
         with:
           python-version: ${{ matrix.PY_VER }}
+          # TODO: remove allow-prereleases once 3.15 is officially supported
+          allow-prereleases: ${{ startsWith(matrix.PY_VER, '3.15') }}
         env:
           # we use self-hosted runners on which setup-python behaves weirdly (Python include can't be found)...
           AGENT_TOOLSDIRECTORY: "/opt/hostedtoolcache"
@@ -270,6 +274,18 @@ jobs:
           host-platform: ${{ inputs.host-platform }}
           cuda-version: ${{ matrix.CUDA_VER }}
 
+      # TODO: remove the numpy wheel steps once 3.15 is officially supported
+      - name: Download numpy wheel (pre-release Python)
+        if: ${{ startsWith(matrix.PY_VER, '3.15') }}
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c  # v8.0.1
+        with:
+          name: numpy-python${{ env.PYTHON_VERSION_FORMATTED }}-${{ inputs.host-platform }}
+          path: numpy-wheel
+
+      - name: Install numpy wheel (pre-release Python)
+        if: ${{ startsWith(matrix.PY_VER, '3.15') }}
+        run: pip install numpy-wheel/*.whl
+
       - name: Set up latest cuda_sanitizer_api
         if: ${{ env.SETUP_SANITIZER == '1' }}
         uses: ./.github/actions/fetch_ctk

.github/workflows/test-wheel-windows.yml

@@ -95,6 +95,8 @@ jobs:
       fail-fast: false
       matrix: ${{ fromJSON(needs.compute-matrix.outputs.MATRIX) }}
     if: ${{ github.repository_owner == 'nvidia' && !cancelled() }}
+    # TODO: remove continue-on-error once 3.15 is officially supported
+    continue-on-error: ${{ startsWith(matrix.PY_VER, '3.15') }}
     runs-on: "windows-${{ matrix.ARCH }}-gpu-${{ matrix.GPU }}-${{ matrix.DRIVER }}-${{ matrix.GPU_COUNT }}"
     steps:
       - name: Checkout ${{ github.event.repository.name }}
@@ -262,6 +264,19 @@ jobs:
           host-platform: ${{ inputs.host-platform }}
           cuda-version: ${{ matrix.CUDA_VER }}
 
+      # TODO: remove the numpy wheel steps once 3.15 is officially supported
+      - name: Download numpy wheel (pre-release Python)
+        if: ${{ startsWith(matrix.PY_VER, '3.15') }}
+        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c  # v8.0.1
+        with:
+          name: numpy-python${{ env.PYTHON_VERSION_FORMATTED }}-${{ inputs.host-platform }}
+          path: numpy-wheel
+
+      - name: Install numpy wheel (pre-release Python)
+        if: ${{ startsWith(matrix.PY_VER, '3.15') }}
+        shell: bash --noprofile --norc -xeuo pipefail {0}
+        run: pip install numpy-wheel/*.whl
+
       - name: Set up test repetition on nightly runs
         shell: bash --noprofile --norc -xeuo pipefail {0}
         run: echo "PYTEST_ADDOPTS=\"--count=${{ inputs.nruns }}\"" >> "$GITHUB_ENV"

ci/test-matrix.yml

@@ -36,6 +36,9 @@ linux:
     - { ARCH: 'amd64', PY_VER: '3.14t', CUDA_VER: '12.9.1', LOCAL_CTK: '1', GPU: 't4',         GPU_COUNT: '1', DRIVER: 'latest' }
     - { ARCH: 'amd64', PY_VER: '3.14t', CUDA_VER: '13.0.2', LOCAL_CTK: '1', GPU: 'l4',         GPU_COUNT: '1', DRIVER: 'latest' }
     - { ARCH: 'amd64', PY_VER: '3.14t', CUDA_VER: '13.3.0', LOCAL_CTK: '1', GPU: 'l4',         GPU_COUNT: '1', DRIVER: 'latest' }
+    - { ARCH: 'amd64', PY_VER: '3.15',  CUDA_VER: '13.3.0', LOCAL_CTK: '0', GPU: 'l4',         GPU_COUNT: '1', DRIVER: 'latest' }
+    # Blocked by NVIDIA/cuda-python#2171
+    # - { ARCH: 'amd64', PY_VER: '3.15t', CUDA_VER: '13.3.0', LOCAL_CTK: '0', GPU: 'l4',         GPU_COUNT: '1', DRIVER: 'latest' }
     # linux-aarch64
     - { ARCH: 'arm64', PY_VER: '3.10',  CUDA_VER: '12.9.1', LOCAL_CTK: '1', GPU: 'a100',       GPU_COUNT: '1', DRIVER: 'latest' }
     - { ARCH: 'arm64', PY_VER: '3.10',  CUDA_VER: '13.0.2', LOCAL_CTK: '0', GPU: 'l4',         GPU_COUNT: '1', DRIVER: 'latest' }

ci/tools/download-wheels

@@ -56,6 +56,13 @@ do
         continue
     fi
 
+    # TODO: remove the Python 3.15 exclusion once 3.15 is officially supported
+    # exclude pre-release Python (3.15) wheels from releasing
+    if [[ "${p}" == *python315* ]]; then
+        echo "Skipping pre-release Python artifact: $p"
+        continue
+    fi
+
     # If we're not downloading "all", only process matching component
     if [[ "$COMPONENT" != "all" && "$p" != ${COMPONENT}* ]]; then
         continue

cuda_bindings/pyproject.toml

@@ -46,7 +46,8 @@ all = [
 test = [
     "cython>=3.2,<3.3",
     "setuptools>=77.0.0",
-    "matplotlib>=3.5.0",  # Required by isoFDModelling_test.py
+    # TODO: remove the Python 3.15 guard once 3.15 is officially supported
+    "matplotlib>=3.5.0; python_version < '3.15'",
     "numpy>=1.21.1",
     "pytest>=6.2.4",
     "pytest-benchmark>=3.4.1",

cuda_core/pyproject.toml

@@ -58,7 +58,13 @@ cu12 = ["cuda-bindings[all]==12.*", "cuda-toolkit==12.*"]
 cu13 = ["cuda-bindings[all]==13.*", "cuda-toolkit==13.*"]
 
 [dependency-groups]
-test = ["cython>=3.2,<3.3", "setuptools", "pytest>=6.2.4", "pytest-benchmark", "pytest-randomly", "pytest-repeat", "pytest-rerunfailures", "pytest-timeout", "cloudpickle", "psutil", "cffi"]
+test = [
+    "cython>=3.2,<3.3", "setuptools", "pytest>=6.2.4", "pytest-benchmark",
+    "pytest-randomly", "pytest-repeat", "pytest-rerunfailures", "pytest-timeout",
+    "cloudpickle", "psutil",
+    # TODO: remove the Python 3.15 guard once 3.15 is officially supported
+    "cffi; python_version < '3.15'",
+]
 ml-dtypes = ["ml-dtypes>=0.5.4,<0.6.0"]
 test-cu12 = [ {include-group = "ml-dtypes" }, {include-group = "test" }, "cupy-cuda12x; python_version < '3.14'", "cuda-toolkit[cudart]==12.*"]  # runtime headers needed by CuPy
 test-cu13 = [ {include-group = "ml-dtypes" }, {include-group = "test" }, "cupy-cuda13x; python_version < '3.14'", "cuda-toolkit[cudart]==13.*"]  # runtime headers needed by CuPy

cuda_core/tests/test_rlcompleter_patch.py

@@ -96,6 +96,7 @@ def test_patched_completion_succeeds_on_non_ipc_resource():
     assert "allocation_handle: True" in result.stdout, result.stdout
 
 
+@pytest.mark.skipif(sys.version_info >= (3, 15), reason="Python 3.15 fixed the rlcompleter bug upstream")
 def test_opt_out_env_var_disables_patch_even_when_interactive():
     """`CUDA_CORE_DONT_FIX_TAB_COMPLETION=1` must short-circuit before the
     interactive check, so the bug reproduces again even under PYTHONINSPECT."""

cuda_pathfinder/pyproject.toml

@@ -46,7 +46,8 @@ cu13 = [
     "nvidia-nvshmem-cu13; sys_platform != 'win32'",
 ]
 host = [
-    "nvidia-cutlass",
+    # TODO: remove the Python 3.15 guard once 3.15 is officially supported
+    "nvidia-cutlass; python_version < '3.15'",
     "nvpl-fft; platform_system == 'Linux' and platform_machine == 'aarch64'",
 ]
 

cuda_pathfinder/tests/test_find_nvidia_headers.py

@@ -16,6 +16,7 @@
 import importlib.metadata
 import os
 import re
+import sys
 from pathlib import Path
 
 import pytest
@@ -112,7 +113,18 @@ def _fake_cudart_canary_abs_path(ctk_root: Path) -> str:
     return str(ctk_root / "lib64" / "libcudart.so.13")
 
 
-@pytest.mark.parametrize("libname", SUPPORTED_HEADERS_NON_CTK.keys())
+# TODO: remove the Python 3.15 guard once 3.15 is officially supported
+_CUTLASS_SKIP = pytest.mark.skipif(
+    sys.version_info >= (3, 15),
+    reason="nvidia-cutlass not available on Python 3.15 (scipy missing)",
+)
+_NON_CTK_HEADER_PARAMS = [
+    pytest.param(name, marks=_CUTLASS_SKIP) if name in ("cutlass", "cute") else name
+    for name in SUPPORTED_HEADERS_NON_CTK
+]
+
+
+@pytest.mark.parametrize("libname", _NON_CTK_HEADER_PARAMS)
 def test_locate_non_ctk_headers(info_summary_append, libname):
     hdr_dir = find_nvidia_header_directory(libname)
     located_hdr_dir = locate_nvidia_header_directory(libname)