cuda.core.system: Add MIG-related APIs

[mdboom]

These APIs are required by dask-cuda.

[github-actions]

Doc Preview CI
🚀 View preview at https://nvidia.github.io/cuda-python/pr-preview/pr-1916/
https://nvidia.github.io/cuda-python/pr-preview/pr-1916/cuda-core/
https://nvidia.github.io/cuda-python/pr-preview/pr-1916/cuda-bindings/
https://nvidia.github.io/cuda-python/pr-preview/pr-1916/cuda-pathfinder/
Preview will be ready when the GitHub Pages deployment is complete.

[rwgk]

Strong suggestion to move this comment into the docstring, mainly so that it appears in the online documentation.

[mdboom]

Got it. Updated.

[rwgk]

Minor nit: It'd be nice to be consistent; or simply remove the comment (preferred).

Lower-case here:

        In the upstream NVML C++ API, the UUID includes a ``gpu-`` or ``mig-``

Upper-case here:

        # NVML UUIDs have a `GPU-` or `MIG-` prefix.  We remove that here.

[rwgk]

Generated with Cursor GPT-5.4 Extra High Fast

I did not check these findings manually. (Recently such findings have become generally highly reliable.)

---

1. High: cuda_core/cuda/core/system/_mig.pxi:126 uses self._handle inside MigInfo.parent, but MigInfo only stores self._device. Accessing device.mig.parent will raise AttributeError instead of returning the parent device.

2. High: the get_device_count -> device_count rename was only half-applied. cuda_core/cuda/core/system/_mig.pxi:176 still calls self.get_device_count(), and cuda_core/tests/system/test_system_device.py:745 still calls mig.get_device_count(). On any system that exercises the MIG path, both mig.get_all_devices() and the test will fail with AttributeError.

3. Medium: the MIG mode logic is inverted for normal callers. system.Device.get_all_devices() returns top-level NVML devices from cuda_core/cuda/core/system/_device.pyx:296, but cuda_core/cuda/core/system/_mig.pxi:46, cuda_core/cuda/core/system/_mig.pxi:67, and cuda_core/cuda/core/system/_mig.pxi:92 gate mode/pending_mode/setter on is_mig_device. That means device.mig.mode reports False and the setter raises on the parent GPU devices that actually own MIG mode, so the new API does not expose enabled MIG mode through the normal enumeration path.

[mdboom]

Generated with Cursor GPT-5.4 Extra High Fast

I did not check these findings manually. (Recently such findings have become generally highly reliable.)

1. High: cuda_core/cuda/core/system/_mig.pxi:126 uses self._handle inside MigInfo.parent, but MigInfo only stores self._device. Accessing device.mig.parent will raise AttributeError instead of returning the parent device.
2. High: the get_device_count -> device_count rename was only half-applied. cuda_core/cuda/core/system/_mig.pxi:176 still calls self.get_device_count(), and cuda_core/tests/system/test_system_device.py:745 still calls mig.get_device_count(). On any system that exercises the MIG path, both mig.get_all_devices() and the test will fail with AttributeError.
3. Medium: the MIG mode logic is inverted for normal callers. system.Device.get_all_devices() returns top-level NVML devices from cuda_core/cuda/core/system/_device.pyx:296, but cuda_core/cuda/core/system/_mig.pxi:46, cuda_core/cuda/core/system/_mig.pxi:67, and cuda_core/cuda/core/system/_mig.pxi:92 gate mode/pending_mode/setter on is_mig_device. That means device.mig.mode reports False and the setter raises on the parent GPU devices that actually own MIG mode, so the new API does not expose enabled MIG mode through the normal enumeration path.

1 and 2 are good catches. 3 is word salad, but it probably makes sense to remove the gating until we understand how the API works.

[rwgk]

Using different kinds of agents seems super useful, but "fighting" surely doesn't make much sense.

Maybe we should have a team discussion how we could make better use of agents. Technically, I think it'd be most efficient to have multiple agents/reviewers actively work together on a PR (e.g. these are the fixes my agent worked out in a couple minutes: commit 94677f6). That's a big shift away from traditional reviews though.

  Findings
  • High: cuda_core/cuda/core/system/_mig.pxi:115 still calls nvml.device_get_handle_from_mig_device_handle(...), but the bindings only
    expose device_get_device_handle_from_mig_device_handle in cuda_bindings/cuda/bindings/nvml.pxd:412. So device.mig.parent still fails
     at runtime. This partially fixes my first finding, but it does not close it.
  • High: cuda_core/tests/system/test_system_device.py:745 still calls mig.get_device_count(), while MigInfo now only exposes the
    device_count property in cuda_core/cuda/core/system/_mig.pxi:88. On any machine that actually enters that MIG branch, the test still
     fails. My second finding is still open in the test.
  • Medium: cuda_core/tests/system/test_system_device.py:741 still guards the important MIG assertions behind if mig.is_mig_device,
    while system.Device.get_all_devices() in cuda_core/cuda/core/system/_device.pyx:299 enumerates top-level devices. That means the
    test still does not cover the normal device.mig.mode / pending_mode / device_count path for parent GPUs. The runtime logic looks
    improved, but the testing half of my third finding is still unresolved.

  Assumptions
  • Reviewed current HEAD 58b4b21bac.
  • Did not run local tests.

  Change context
  • The branch does address part of the earlier feedback: mode and pending_mode no longer early-return on non-MIG handles, parent now
    uses self._device, and get_all_devices() now uses self.device_count.
  • But no, I would not say all three earlier findings are addressed yet. One is still a runtime bug, one is still a stale test failure,
     and the last is only partially fixed because the test still misses the normal caller path.