__________________________ test_malloc_trim_threshold __________________________
c = <Client: No scheduler connected>
s = <Scheduler 'tcp://127.0.0.1:38743', workers: 0, cores: 0, tasks: 0>
a = <Nanny: None, threads: 2>
@pytest.mark.slow
@pytest.mark.skipif(not LINUX, reason="Requires GNU libc")
@gen_cluster(
client=True,
Worker=Nanny,
nthreads=[("", 2)],
worker_kwargs={"memory_limit": "2GiB"},
)
async def test_malloc_trim_threshold(c, s, a):
"""Test that the nanny sets the MALLOC_TRIM_THRESHOLD_ environment variable before
starting the worker process.
This test relies on these settings to work:
distributed.nanny.pre-spawn-environ.MALLOC_TRIM_THRESHOLD_: 65536
distributed.worker.multiprocessing-method: spawn
We're deliberately not setting them explicitly in @gen_cluster above, as we want
this test to trip if somebody changes distributed.yaml.
Note
----
This test may start failing in a future Python version if CPython switches to
using mimalloc by default. If it does, a thorough benchmarking exercise is needed.
"""
pytest.importorskip("numpy")
da = pytest.importorskip("dask.array")
arr = da.random.random(2**29 // 8, chunks="512 kiB") # 0.5 GiB
arr = c.persist(arr)
await wait(arr)
# Wait for heartbeat
await async_poll_for(lambda: s.memory.process > 2**29, timeout=5)
del arr
# This is the delicate bit, as it relies on
# 1. PyMem_Free() to be quick to invoke glibc free() when memory becomes available
# 2. glibc free() to be quick to invoke the kernel's sbrk() when the same happens
#
# At the moment of writing, the readings are:
# - 132 MiB after starting a new worker
# - 670~725[1] MiB after arr has been fully computed
# - 156~210[1] MiB at the end of this test, with MALLOC_TRIM_THRESHOLD_=65536
# - 620~670[1] MiB at the end of this test, without MALLOC_TRIM_THRESHOLD_
# [1] depends on distributed.scheduler.worker-saturation
> await async_poll_for(lambda: s.memory.process < 300 * 2**20, timeout=5)
packages/distributed/distributed/tests/test_nanny.py:818:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
predicate = <function test_malloc_trim_threshold.<locals>.<lambda> at 0x769972435300>
timeout = 5, fail_func = None, period = 0.05
async def async_poll_for(predicate, timeout, fail_func=None, period=0.05):
deadline = time() + timeout
while not predicate():
await asyncio.sleep(period)
if time() > deadline:
if fail_func is not None:
fail_func()
> pytest.fail(f"condition not reached until {timeout} seconds")
E Failed: condition not reached until 5 seconds
packages/distributed/distributed/utils_test.py:1257: Failed
----------------------------- Captured stderr call -----------------------------
2026-04-01 08:22:41,142 - distributed.scheduler - INFO - State start
2026-04-01 08:22:41,148 - distributed.scheduler - INFO - Scheduler at: tcp://127.0.0.1:38743
2026-04-01 08:22:41,149 - distributed.scheduler - INFO - dashboard at: http://127.0.0.1:38657/status
2026-04-01 08:22:41,150 - distributed.scheduler - INFO - Registering Worker plugin shuffle
2026-04-01 08:22:41,158 - distributed.nanny - INFO - Start Nanny at: 'tcp://127.0.0.1:38899'
2026-04-01 08:22:42,050 - distributed.worker - INFO - Start worker at: tcp://127.0.0.1:34927
2026-04-01 08:22:42,050 - distributed.worker - INFO - Listening to: tcp://127.0.0.1:34927
2026-04-01 08:22:42,050 - distributed.worker - INFO - Worker name: 0
2026-04-01 08:22:42,051 - distributed.worker - INFO - dashboard at: 127.0.0.1:39669
2026-04-01 08:22:42,051 - distributed.worker - INFO - Waiting to connect to: tcp://127.0.0.1:38743
2026-04-01 08:22:42,051 - distributed.worker - INFO - -------------------------------------------------
2026-04-01 08:22:42,051 - distributed.worker - INFO - Threads: 2
2026-04-01 08:22:42,051 - distributed.worker - INFO - Memory: 2.00 GiB
2026-04-01 08:22:42,051 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-gsz2u6xf
2026-04-01 08:22:42,051 - distributed.worker - INFO - -------------------------------------------------
2026-04-01 08:22:42,097 - distributed.scheduler - INFO - Register worker addr: tcp://127.0.0.1:34927 name: 0
2026-04-01 08:22:42,100 - distributed.worker - INFO - Starting Worker plugin shuffle
2026-04-01 08:22:42,100 - distributed.worker - INFO - Registered to: tcp://127.0.0.1:38743
2026-04-01 08:22:42,101 - distributed.worker - INFO - -------------------------------------------------
2026-04-01 08:22:42,099 - distributed.scheduler - INFO - Starting worker compute stream, tcp://127.0.0.1:34927
2026-04-01 08:22:42,101 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:46860
2026-04-01 08:22:42,102 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:38743
2026-04-01 08:22:42,143 - distributed.scheduler - INFO - Receive client connection: Client-f3f978ae-2da3-11f1-96c5-89308faff95a
2026-04-01 08:22:42,145 - distributed.core - INFO - Starting established connection to tcp://127.0.0.1:46870
2026-04-01 08:22:52,417 - distributed.scheduler - INFO - Remove client Client-f3f978ae-2da3-11f1-96c5-89308faff95a
2026-04-01 08:22:52,419 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:46870; closing.
2026-04-01 08:22:52,420 - distributed.scheduler - INFO - Remove client Client-f3f978ae-2da3-11f1-96c5-89308faff95a
2026-04-01 08:22:52,423 - distributed.scheduler - INFO - Close client connection: Client-f3f978ae-2da3-11f1-96c5-89308faff95a
2026-04-01 08:22:52,425 - distributed.nanny - INFO - Closing Nanny at 'tcp://127.0.0.1:38899'. Reason: nanny-close
2026-04-01 08:22:52,426 - distributed.nanny - INFO - Nanny asking worker to close. Reason: nanny-close
2026-04-01 08:22:52,429 - distributed.worker - INFO - Stopping worker at tcp://127.0.0.1:34927. Reason: nanny-close
2026-04-01 08:22:52,430 - distributed.worker - INFO - Removing Worker plugin shuffle
2026-04-01 08:22:52,433 - distributed.core - INFO - Received 'close-stream' from tcp://127.0.0.1:46860; closing.
2026-04-01 08:22:52,433 - distributed.core - INFO - Connection to tcp://127.0.0.1:38743 has been closed.
2026-04-01 08:22:52,434 - distributed.scheduler - INFO - Remove worker addr: tcp://127.0.0.1:34927 name: 0 (stimulus_id='handle-worker-cleanup-1775031772.4345043')
2026-04-01 08:22:52,436 - distributed.scheduler - INFO - Lost all workers
2026-04-01 08:22:52,441 - distributed.nanny - INFO - Worker closed
2026-04-01 08:22:53,012 - distributed.nanny - INFO - Nanny at 'tcp://127.0.0.1:38899' closed.
2026-04-01 08:22:53,014 - distributed.scheduler - INFO - Closing scheduler. Reason: unknown
2026-04-01 08:22:53,017 - distributed.scheduler - INFO - Scheduler closing all comms
I didn't see it in distributed's own CI, passing or failing (it might be skipped?)
This test is failing on
main:I didn't see it in distributed's own CI, passing or failing (it might be skipped?)