Skip to content

Nightly faliures #101

@TomAugspurger

Description

@TomAugspurger

CuML

Some timeouts, e.g in packages/cuml/python/cuml/tests/dask/test_dask_base.py::test_get_combined_model

Details
Error: test_get_combined_model[True-data_size0-LinearRegression-float32]

Failed: Timeout (>120.0s) from pytest-timeout.
packages/cuml/python/cuml/tests/dask/test_dask_base.py::test_get_combined_model[True-data_size0-LinearRegression-float32] +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
~ Stack of AsyncProcess Dask Worker process (from Nanny) watch process join (138015837894336) ~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/process.py", line 250, in _watch_process
    process.join()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/popen_fork.py", line 43, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/popen_fork.py", line 27, in poll
    pid, sts = os.waitpid(self.pid, flag)
~ Stack of AsyncProcess Dask Worker process (from Nanny) watch message queue (138015175337664) ~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/process.py", line 230, in _watch_message_queue
    msg = q.get()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/queue.py", line 171, in get
    self.not_empty.wait()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 355, in wait
    waiter.acquire()
~~~~~~~~~~~~~~~~~ Stack of QueueFeederThread (138015183730368) ~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/queues.py", line 251, in _feed
    nwait()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 355, in wait
    waiter.acquire()
~~~~~~~~~~~~~~~~~~~~~~ Stack of Profile (138015854679744) ~~~~~~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/profile.py", line 370, in _watch
    sleep(interval)
~~~~~~~~~~~~~~~~~~~~~~ Stack of IO loop (138015846287040) ~~~~~~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 466, in wrapper
    target()
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 574, in run_loop
    asyncio_run(amain(), loop_factory=get_loop_factory())
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 678, in run_until_complete
    self.run_forever()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 645, in run_forever
    self._run_once()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 1961, in _run_once
    event_list = self._selector.select(timeout)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/selectors.py", line 468, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
~~~~~~~~~~~~~~~~~~ Stack of Dask-Offload_0 (138014556595904) ~~~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 90, in _worker
    work_item = work_queue.get(block=True)
~~~~~~~~~~~~~~~ Stack of Thread-1 (run_server) (138025835161280) ~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pytest_rerunfailures.py", line 474, in run_server
    conn, _ = self.sock.accept()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/socket.py", line 295, in accept
    fd, addr = self._accept()
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
FAILED                                                                   [  4%]
Error: test_get_combined_model[True-data_size0-LinearRegression-float64]

Failed: Timeout (>120.0s) from pytest-timeout.
packages/cuml/python/cuml/tests/dask/test_dask_base.py::test_get_combined_model[True-data_size0-LinearRegression-float64] +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
~ Stack of AsyncProcess Dask Worker process (from Nanny) watch process join (138015837894336) ~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/process.py", line 250, in _watch_process
    process.join()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 149, in join
    res = self._popen.wait(timeout)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/popen_fork.py", line 43, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/popen_fork.py", line 27, in poll
    pid, sts = os.waitpid(self.pid, flag)
~ Stack of AsyncProcess Dask Worker process (from Nanny) watch message queue (138015175337664) ~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/process.py", line 230, in _watch_message_queue
    msg = q.get()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/queue.py", line 171, in get
    self.not_empty.wait()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 355, in wait
    waiter.acquire()
~~~~~~~~~~~~~~~~~ Stack of QueueFeederThread (138015183730368) ~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/queues.py", line 251, in _feed
    nwait()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 355, in wait
    waiter.acquire()
~~~~~~~~~~~~~~~~~~~~~~ Stack of Profile (138015854679744) ~~~~~~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/profile.py", line 370, in _watch
    sleep(interval)
~~~~~~~~~~~~~~~~~~~~~~ Stack of IO loop (138015846287040) ~~~~~~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 466, in wrapper
    target()
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 574, in run_loop
    asyncio_run(amain(), loop_factory=get_loop_factory())
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195, in run
    return runner.run(main)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 678, in run_until_complete
    self.run_forever()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 645, in run_forever
    self._run_once()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 1961, in _run_once
    event_list = self._selector.select(timeout)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/selectors.py", line 468, in select
    fd_event_list = self._selector.poll(timeout, max_ev)
~~~~~~~~~~~~~~~~~~ Stack of Dask-Offload_0 (138014556595904) ~~~~~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 90, in _worker
    work_item = work_queue.get(block=True)
~~~~~~~~~~~~~~~ Stack of Thread-1 (run_server) (138025835161280) ~~~~~~~~~~~~~~~
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032, in _bootstrap
    self._bootstrap_inner()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
    self.run()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012, in run
    self._target(*self._args, **self._kwargs)
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pytest_rerunfailures.py", line 474, in run_server
    conn, _ = self.sock.accept()
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/socket.py", line 295, in accept
    fd, addr = self._accept()
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
Warning: coroutine 'Client._run' was never awaited
Warning: coroutine 'Client._run_on_scheduler' was never awaited
FAILED                                                                   [  4%]

And a segfault

Details
+++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
Warning: Exception ignored in: <function Comms.__del__ at 0x7d889f12cfe0>

Traceback (most recent call last):
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/raft_dask/common/comms.py", line 124, in __del__
    self.destroy()
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/raft_dask/common/comms.py", line 215, in destroy
    self.client.run(
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/client.py", line 3230, in run
    return self.sync(
           ^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 379, in sync
    return sync(
           ^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 455, in sync
    raise error
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 429, in f
    result = yield future
             ^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/tornado/gen.py", line 783, in run
    value = future.result()
            ^^^^^^^^^^^^^^^
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/client.py", line 3106, in _run
    responses = await self.scheduler.broadcast(
                      ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'broadcast'

Enable tracemalloc to get traceback where the object was allocated.
See https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings for more info.
FAILED                                                                   [  5%]
Fatal Python error: Segmentation fault

Thread 0x00007d8650fdb6c0 (most recent call first):
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/popen_fork.py", line 27 in poll
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/popen_fork.py", line 43 in wait
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/process.py", line 149 in join
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/process.py", line 250 in _watch_process
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007d8629fff6c0 (most recent call first):
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 355 in wait
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/queue.py", line 171 in get
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/process.py", line 230 in _watch_message_queue
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007d86297fe6c0 (most recent call first):
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 355 in wait
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/multiprocessing/queues.py", line 251 in _feed
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007d8651fdd6c0 (most recent call first):
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/profile.py", line 370 in _watch
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007d86517dc6c0 (most recent call first):
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/selectors.py", line 468 in select
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 1961 in _run_once
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 645 in run_forever
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/base_events.py", line 678 in run_until_complete
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 118 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/asyncio/runners.py", line 195 in run
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 574 in run_loop
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/distributed/utils.py", line 466 in wrapper
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007d86049ea6c0 (most recent call first):
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/concurrent/futures/thread.py", line 90 in _worker
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x00007d88a4dfe6c0 (most recent call first):
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/socket.py", line 295 in accept
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pytest_rerunfailures.py", line 474 in run_server
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1012 in run
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/github/home/.local/share/uv/python/cpython-3.12.13-linux-x86_64-gnu/lib/python3.12/threading.py", line 1032 in _bootstrap

Current thread 0x00007d8a34558740 (most recent call first):
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/cuml/internals/outputs.py", line 433 in inner
  File "/__w/dask-upstream-testing/dask-upstream-testing/packages/cuml/python/cuml/tests/dask/test_dask_base.py", line 124 in test_regressor_sg_train_mg_predict
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/python.py", line 166 in pytest_pyfunc_call
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/python.py", line 1720 in runtest
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 179 in pytest_runtest_call
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 245 in <lambda>
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 353 in from_call
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 244 in call_and_report
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 137 in runtestprotocol
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 118 in pytest_runtest_protocol
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/main.py", line 396 in pytest_runtestloop
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/main.py", line 372 in _main
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/main.py", line 318 in wrap_session
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/main.py", line 365 in pytest_cmdline_main
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 199 in main
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/lib/python3.12/site-packages/_pytest/config/__init__.py", line 223 in console_main
  File "/__w/dask-upstream-testing/dask-upstream-testing/.venv/bin/pytest", line 10 in <module>

Extension modules: markupsafe._speedups, sklearn.__check_build._check_build, numpy._core._multiarray_umath, numpy.linalg._umath_linalg, cython.cimports.libc.math, scipy._lib._ccallback_c, charset_normalizer.md, charset_normalizer.cd, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._generator, numpy.random._mt19937, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.linalg._propack._spropack, scipy.sparse.linalg._propack._dpropack, scipy.sparse.linalg._propack._cpropack, scipy.sparse.linalg._propack._zpropack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, psutil._psutil_linux, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.spatial.transform._rotation, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._cobyla, scipy.optimize._slsqp, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy.optimize._cython_nnls, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.integrate._lsoda, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.interpolate._bspl, scipy.special.cython_special, scipy.stats._stats, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._biasedurn, scipy.stats._stats_pythran, scipy.stats._levy_stable.levyst, scipy.stats._ansari_swilk_statistics, scipy.stats._mvn, scipy.stats._rcont.rcont, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, _ni_label, scipy.ndimage._ni_label, pyarrow.lib, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, _cyutility, sklearn._cyutility, sklearn.utils._isfinite, sklearn.utils.sparsefuncs_fast, sklearn.utils.murmurhash, sklearn.utils._openmp_helpers, yaml._yaml, numba.core.typeconv._typeconv, numba._helperlib, numba._dynfunc, numba._dispatcher, numba.core.runtime._nrt_python, numba.np.ufunc._internal, numba.experimental.jitclass._box, cupy_backends.cuda._softlink, cupy_backends.cuda.api._runtime_enum, cupy_backends.cuda.api.runtime, cupy._util, cupy.cuda.device, cupy.cuda.memory_hook, cupy_backends.cuda.stream, cupy.cuda.graph, cupy.cuda.stream, cupy_backends.cuda.api._driver_enum, cupy_backends.cuda.api.driver, cupy.cuda.memory, cupy._core.internal, cupy._core._carray, cupy._core._dtype, cupy._core._scalar, cupy.cuda.texture, cupy.cuda.function, cupy_backends.cuda.libs.nvrtc, cupy.cuda.pinned_memory, cupy.cuda.common, cupy.cuda.cub, cupy_backends.cuda.libs.nvtx, cupy.cuda.thrust, cupy._core._accelerator, cupy._core._memory_range, cupy._core._fusion_thread_local, cupy._core._kernel, cupy._core._routines_manipulation, cupy._core._routines_binary, cupy._core._optimize_config, cupy._core._cub_reduction, cupy._core._reduction, cupy._core._routines_math, cupy._core._routines_indexing, cupy._core._routines_linalg, cupy._core._routines_logic, cupy._core._routines_sorting, cupy._core._routines_statistics, cupy._core.dlpack, cupy._core.flags, cupy._core.core, cupy._core._fusion_variable, cupy._core._fusion_trace, cupy._core._fusion_kernel, cupy._core.new_fusion, cupy._core.fusion, cupy._core.raw, cupy.fft._cache, cupy.fft._callback, cupy.random._generator_api, cupy.random._bit_generator, cupy.lib._polynomial, numba.cuda.types.abstract.itertools, numba.cuda.types.functions.itertools, numba.cuda.cext._typeconv, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.bindings.cyruntime, cuda.bindings.runtime, cuda.core.system._system, cuda.bindings._internal.utils, cuda.bindings._internal.nvml, cuda.bindings.cynvml, cuda.bindings.nvml, cuda.core.system._nvml_context, cuda.core.cu12.system._nvml_context, cuda.core.cu12.system._system, cuda.core.system._device, cuda.core.cu12.system._device, cuda.core.system._system_events, cuda.core.cu12.system._system_events, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings._internal.nvvm, cuda.bindings.cynvvm, cuda.bindings._internal.nvjitlink, cuda.bindings.cynvjitlink, cuda.bindings.nvrtc, cuda.bindings.nvvm, cuda.bindings.nvjitlink, cuda.core._utils.cuda_utils, cuda.core._layout, cuda.core._resource_handles, cuda.core._dlpack, cuda.core._context, cuda.core._event, cuda.core._stream, cuda.core._memory._ipc, cuda.core._memory._memory_pool, cuda.core._memory._device_memory_resource, cuda.core._memory._pinned_memory_resource, cuda.core._device, cuda.core._memory._buffer, cuda.core._memory._graph_memory_resource, cuda.core._memory._managed_memory_resource, cuda.core._utils.version, cuda.core._memoryview, cuda.core._graphics, cuda.core._launch_config, cuda.core._tensor_map, cuda.core._kernel_arg_handler, cuda.core._module, cuda.core._launcher, cuda.core._linker, cuda.core._program, cuda.core.graph._utils, cuda.core.graph._graph_builder, cuda.core.graph._subclasses, cuda.core.graph._adjacency_set_proxy, cuda.core.graph._graph_node, cuda.core.graph._graph_def, numba.cuda.cext._helperlib, numba.cuda.cext.mviewbuf, numba.cuda.cext._dispatcher, rmm.pylibrmm.stream, rmm.pylibrmm.utils, rmm.pylibrmm.helper, rmm.pylibrmm.memory_resource._memory_resource, rmm.pylibrmm.memory_resource.experimental, rmm.pylibrmm.cuda_stream_pool, rmm.pylibrmm.device_buffer, rmm.librmm._logger, rmm.pylibrmm.logger, nvtx._lib.lib, nvtx._lib.profiler, pylibcudf.libcudf.types, pylibcudf.types, pylibcudf.libcudf.aggregation, pylibcudf.aggregation, pylibcudf.gpumemoryview, pylibcudf.utils, pylibcudf._interop_helpers, pylibcudf.table, pylibcudf.filling, pylibcudf.traits, pylibcudf.column, pylibcudf.scalar, pylibcudf.libcudf.binaryop, pylibcudf.binaryop, pylibcudf.column_factories, pylibcudf.concatenate, pylibcudf.context, pylibcudf.contiguous_split, pylibcudf.libcudf.copying, pylibcudf.copying, pylibcudf.libcudf.datetime, pylibcudf.datetime, pylibcudf.experimental._join_streams, pylibcudf.libcudf.expressions, pylibcudf.expressions, pylibcudf.groupby, pylibcudf.hashing, pylibcudf.interop, pylibcudf.io.datasource, pylibcudf.libcudf.io.json, pylibcudf.libcudf.io.types, pylibcudf.io.types, pylibcudf.io.avro, pylibcudf.io.csv, pylibcudf.io.parquet, pylibcudf.io.text, pylibcudf.libcudf.io.hybrid_scan, pylibcudf.io.experimental.hybrid_scan, pylibcudf.io.json, pylibcudf.io.orc, pylibcudf.io.parquet_metadata, pylibcudf.io.timezone, pylibcudf.libcudf.join, pylibcudf.join, pylibcudf.json, pylibcudf.libcudf.labeling, pylibcudf.labeling, pylibcudf.libcudf.lists.combine, pylibcudf.libcudf.lists.contains, pylibcudf.lists, pylibcudf.merge, pylibcudf.null_mask, pylibcudf.nvtext.byte_pair_encode, pylibcudf.nvtext.deduplicate, pylibcudf.nvtext.edit_distance, pylibcudf.nvtext.generate_ngrams, pylibcudf.nvtext.jaccard, pylibcudf.nvtext.minhash, pylibcudf.nvtext.ngrams_tokenize, pylibcudf.nvtext.normalize, pylibcudf.nvtext.replace, pylibcudf.libcudf.nvtext.stemmer, pylibcudf.nvtext.stemmer, pylibcudf.nvtext.tokenize, pylibcudf.nvtext.wordpiece_tokenize, pylibcudf.libcudf.partitioning, pylibcudf.partitioning, pylibcudf.prefetch, pylibcudf.quantiles, pylibcudf.libcudf.reduce, pylibcudf.reduce, pylibcudf.libcudf.replace, pylibcudf.replace, pylibcudf.reshape, pylibcudf.rolling, pylibcudf.libcudf.round, pylibcudf.round, pylibcudf.search, pylibcudf.sorting, pylibcudf.libcudf.stream_compaction, pylibcudf.stream_compaction, pylibcudf.strings.attributes, pylibcudf.libcudf.strings.char_types, pylibcudf.strings.capitalize, pylibcudf.strings.case, pylibcudf.strings.char_types, pylibcudf.libcudf.strings.combine, pylibcudf.strings.combine, pylibcudf.strings.regex_program, pylibcudf.strings.contains, pylibcudf.strings.convert.convert_booleans, pylibcudf.strings.convert.convert_datetime, pylibcudf.strings.convert.convert_durations, pylibcudf.strings.convert.convert_fixed_point, pylibcudf.strings.convert.convert_floats, pylibcudf.strings.convert.convert_integers, pylibcudf.strings.convert.convert_ipv4, pylibcudf.strings.convert.convert_lists, pylibcudf.strings.convert.convert_urls, pylibcudf.strings.extract, pylibcudf.strings.find, pylibcudf.strings.find_multiple, pylibcudf.strings.findall, pylibcudf.strings.padding, pylibcudf.libcudf.strings.regex_flags, pylibcudf.strings.regex_flags, pylibcudf.strings.repeat, pylibcudf.strings.replace, pylibcudf.strings.replace_re, pylibcudf.strings.reverse, pylibcudf.libcudf.strings.side_type, pylibcudf.strings.side_type, pylibcudf.strings.slice, pylibcudf.strings.split.partition, pylibcudf.strings.split.split, pylibcudf.strings.strip, pylibcudf.libcudf.strings.translate, pylibcudf.strings.translate, pylibcudf.strings.wrap, pylibcudf.transform, pylibcudf.transpose, pylibcudf.libcudf.unary, pylibcudf.unary, pylibcudf.utilities, numba.types.itertools, cudf._lib.strings_udf, pyarrow._feather, sklearn.preprocessing._csr_polynomial_expansion, sklearn.preprocessing._target_encoder_fast, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, sklearn.datasets._svmlight_format_fast, sklearn.utils._random, sklearn.utils._vector_sentinel, sklearn.feature_extraction._hashing_fast, pylibraft.common.cuda, pylibraft.common.handle, cuml.internals.logger, cuml.internals.internals, cuml.cluster.agglomerative, cuml.cluster.dbscan, cuml.cluster.hdbscan.hdbscan, cuml.cluster.kmeans, cuml.cluster.spectral_clustering, cuml.datasets.arima, cuml.datasets.regression, cuml.decomposition.pca, cuml.decomposition.tsvd, cuml.internals.treelite, cuml.fil.fil, cuml.metrics.cluster.adjusted_rand_index, cuml.metrics.cluster.completeness_score, cuml.metrics.cluster.entropy, cuml.metrics.cluster.homogeneity_score, cuml.metrics.cluster.mutual_info_score, cuml.metrics.pairwise_distances, cuml.metrics.cluster.silhouette_score, cuml.metrics.cluster.v_measure, cuml.metrics.kl_divergence, cuml.metrics.trustworthiness, cuml.ensemble.randomforest_common, cuml.explainer.base, cupy_backends.cuda.libs.cublas, cuml.solvers.cd, cuml.solvers.qn, cuml.solvers.sgd, cuml.linear_model.linear_regression, cuml.linear_model.ridge, cuml.explainer.kernel_shap, cuml.explainer.permutation_shap, sklearn.metrics.cluster._expected_mutual_info_fast, sklearn.metrics._dist_metrics, sklearn.metrics._pairwise_distances_reduction._datasets_pair, sklearn.utils._cython_blas, sklearn.metrics._pairwise_distances_reduction._base, sklearn.metrics._pairwise_distances_reduction._middle_term_computer, sklearn.utils._heap, sklearn.utils._sorting, sklearn.metrics._pairwise_distances_reduction._argkmin, sklearn.metrics._pairwise_distances_reduction._argkmin_classmode, sklearn.metrics._pairwise_distances_reduction._radius_neighbors, sklearn.metrics._pairwise_distances_reduction._radius_neighbors_classmode, sklearn.metrics._pairwise_fast, cuml.explainer.tree_shap, cuml.manifold.spectral_embedding, cuml.manifold.t_sne, cuml.manifold.umap.umap, cuml.neighbors.nearest_neighbors, cuml.neighbors.kneighbors_classifier, cuml.neighbors.kneighbors_regressor, cuml.svm.linear, cuml.svm.svm_base, cuml.tsa.arima, cuml.tsa.holtwinters, cuml.tsa.stationarity, cuml.tsa.auto_arima, scipy.fftpack.convolve, zstandard.backend_c, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, pyarrow._acero, pyarrow._csv, pyarrow._json, pyarrow._substrait, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet, pyarrow._parquet_encryption, pyarrow._dataset_parquet_encryption, pyarrow._dataset_parquet, pyarrow._orc, tornado.speedups, msgpack._cmsgpack, raft_dask.common.comms_utils, raft_dask.common.nccl, ucxx._lib.arr, ucxx._lib.libucxx, sklearn.utils._seq_dataset, sklearn.linear_model._cd_fast, _loss, sklearn._loss._loss, sklearn.utils.arrayfuncs, sklearn.linear_model._sag_fast, sklearn.svm._liblinear, sklearn.svm._libsvm, sklearn.svm._libsvm_sparse, sklearn.utils._weight_vector, sklearn.linear_model._sgd_fast, sklearn.utils._fast_dict, sklearn.cluster._hierarchical_fast, sklearn.cluster._k_means_common, sklearn.cluster._k_means_elkan, sklearn.cluster._k_means_lloyd, sklearn.cluster._k_means_minibatch, sklearn.cluster._dbscan_inner, sklearn.neighbors._partition_nodes, sklearn.neighbors._ball_tree, sklearn.neighbors._kd_tree, sklearn.decomposition._online_lda_fast, sklearn.decomposition._cdnmf_fast, sklearn.cluster._hdbscan._tree, sklearn.cluster._hdbscan._linkage, sklearn.cluster._hdbscan._reachability, sklearn._isotonic, sklearn.tree._utils, sklearn.tree._tree, sklearn.tree._partitioner, sklearn.tree._splitter, sklearn.tree._criterion, sklearn.neighbors._quad_tree, sklearn.manifold._barnes_hut_tsne, sklearn.manifold._utils, sklearn.ensemble._gradient_boosting, sklearn.ensemble._hist_gradient_boosting.common, sklearn.ensemble._hist_gradient_boosting._gradient_boosting, sklearn.ensemble._hist_gradient_boosting._binning, sklearn.ensemble._hist_gradient_boosting._bitset, sklearn.ensemble._hist_gradient_boosting.histogram, sklearn.ensemble._hist_gradient_boosting._predictor, sklearn.ensemble._hist_gradient_boosting.splitting, cupy_backends.cuda.libs.curand, cupy_backends.cuda.libs.cusparse, cuml.common.opg_data_utils_mg, cuml.linear_model.base_mg, cuml.linear_model.linear_regression_mg (total: 554)
./scripts/test.sh: line 110: 31831 Segmentation fault      (core dumped) pytest -v --timeout=120 --quick_run packages/cuml/python/cuml/tests/dask

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions