Documentation build failing in Windows

[rwkozar]

What happened?

When using pixi run doc to create documentation in Windows, time-coding.rst is aborting with a failure because the file tries to delete test-datetimes1.nc while windows thinks it is open. We tried using gc.collect() to force garbage collection and del _, and a using a sleep timer to ensure that the processes were complete, but none of those worked.

Switching from xr.open_dataset() to xr.load_dataset() prevents the problem, but may not be appropriate in this situation.

What did you expect to happen?

Ideally the docs should build in any environment; however, expecting users Windows users to use WSL or another method when developing may be acceptable, if covered in the contributing guide.

Minimal Complete Verifiable Example

# /// script
# requires-python = ">=3.11"
# dependencies = [
#   "xarray[complete]@git+https://github.com/pydata/xarray.git@main",
# ]
# ///
#
# This script automatically imports the development branch of xarray to check for issues.
# Please delete this header if you have _not_ tested this script with `uv run`!

import xarray as xr
xr.show_versions()
# your reproducer code ...

Steps to reproduce

No response

MVCE confirmation

• Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
• Complete example — the example is self-contained, including all data and the text of any traceback.
• Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
• New issue — a search of GitHub Issues suggests this is not a duplicate.
• Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

sphinx.errors.ExtensionError: Cell raised uncaught exception:
---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
File ~\python\xarray\.pixi\envs\doc\Lib\shutil.py:701, in _rmtree_unsafe(path, dir_fd, onexc)
    700 try:
--> 701     os.unlink(fullname)
    702 except FileNotFoundError:

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\rwkoz\\AppData\\Local\\Temp\\tmptpg6hhgw\\test-datetimes1.nc'

During handling of the above exception, another exception occurred:

PermissionError                           Traceback (most recent call last)
Cell In[49], line 2
      1 # Cleanup
----> 2 tempdir.cleanup()

File ~\python\xarray\.pixi\envs\doc\Lib\tempfile.py:975, in TemporaryDirectory.cleanup(self)
    973 def cleanup(self):
    974     if self._finalizer.detach() or _os.path.exists(self.name):
--> 975         self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)

File ~\python\xarray\.pixi\envs\doc\Lib\tempfile.py:955, in TemporaryDirectory._rmtree(cls, name, ignore_errors, repeated)
    952         if not ignore_errors:
    953             raise
--> 955 _shutil.rmtree(name, onexc=onexc)

File ~\python\xarray\.pixi\envs\doc\Lib\shutil.py:852, in rmtree(path, ignore_errors, onerror, onexc, dir_fd)
    849                 exc_info = type(exc), exc, exc.__traceback__
    850             return onerror(func, path, exc_info)
--> 852 _rmtree_impl(path, dir_fd, onexc)

File ~\python\xarray\.pixi\envs\doc\Lib\shutil.py:705, in _rmtree_unsafe(path, dir_fd, onexc)
    703             continue
    704         except OSError as err:
--> 705             onexc(os.unlink, fullname, err)
    706 try:
    707     os.rmdir(path)

File ~\python\xarray\.pixi\envs\doc\Lib\tempfile.py:930, in TemporaryDirectory._rmtree.<locals>.onexc(func, path, exc)
    927 _resetperms(path)
    929 try:
--> 930     _os.unlink(path)
    931 except IsADirectoryError:
    932     cls._rmtree(path, ignore_errors=ignore_errors)

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\rwkoz\\AppData\\Local\\Temp\\tmptpg6hhgw\\test-datetimes1.nc'

Extension error:
Cell raised uncaught exception:
---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
File ~\python\xarray\.pixi\envs\doc\Lib\shutil.py:701, in _rmtree_unsafe(path, dir_fd, onexc)
    700 try:
--> 701     os.unlink(fullname)
    702 except FileNotFoundError:

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\rwkoz\\AppData\\Local\\Temp\\tmptpg6hhgw\\test-datetimes1.nc'

During handling of the above exception, another exception occurred:

PermissionError                           Traceback (most recent call last)
Cell In[49], line 2
      1 # Cleanup
----> 2 tempdir.cleanup()

File ~\python\xarray\.pixi\envs\doc\Lib\tempfile.py:975, in TemporaryDirectory.cleanup(self)
    973 def cleanup(self):
    974     if self._finalizer.detach() or _os.path.exists(self.name):
--> 975         self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)

File ~\python\xarray\.pixi\envs\doc\Lib\tempfile.py:955, in TemporaryDirectory._rmtree(cls, name, ignore_errors, repeated)
    952         if not ignore_errors:
    953             raise
--> 955 _shutil.rmtree(name, onexc=onexc)

File ~\python\xarray\.pixi\envs\doc\Lib\shutil.py:852, in rmtree(path, ignore_errors, onerror, onexc, dir_fd)
    849                 exc_info = type(exc), exc, exc.__traceback__
    850             return onerror(func, path, exc_info)
--> 852 _rmtree_impl(path, dir_fd, onexc)

File ~\python\xarray\.pixi\envs\doc\Lib\shutil.py:705, in _rmtree_unsafe(path, dir_fd, onexc)
    703             continue
    704         except OSError as err:
--> 705             onexc(os.unlink, fullname, err)
    706 try:
    707     os.rmdir(path)

File ~\python\xarray\.pixi\envs\doc\Lib\tempfile.py:930, in TemporaryDirectory._rmtree.<locals>.onexc(func, path, exc)
    927 _resetperms(path)
    929 try:
--> 930     _os.unlink(path)
    931 except IsADirectoryError:
    932     cls._rmtree(path, ignore_errors=ignore_errors)

PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\rwkoz\\AppData\\Local\\Temp\\tmptpg6hhgw\\test-datetimes1.nc'

Anything else we need to know?

No response

Environment

Details

[Aimwee]

Hi @rwkozar,

I think the issue revolves around the datasets opened with xr.open_dataset() in the time-coding.rst file never explicitly closing. Since the pages run like a notebook, IPython holds on to the returned dataset objects in the output history even though they're not assigned to a variable in the code. Since open_dataset doesn't read everything into memory right away, the file stays open in the background. Windows doesn't allow users to delete files that are still open, leading to cleanup fails in the end. gc.collect() and del didn't help since no variable in the doc were pointing to the object that needed to be deleted.

If you don't mind, I would like to give a shot at this issue. I believe that assigning each dataset to a variable when it's opened and add a .close() immediately after it's used, instead of switching to load_dataset(). I will create a PR and link it here once I'm done.

[Aimwee]

I have created a PR for this issue, #11459

This PR assigns each opened dataset in time-coding.rst to a variable and closes them immediately before tempdir.cleanup() runs. I was able to test this locally on macOS, but since I can't access a Windows environment, I am not able to reproduce the error directly. If you could verify that the changes worked and resolved the issue that would be great!

[VeckoTheGecko]

See #11459 (comment) also for more context around this.

Thanks for attempting to fix this @Aimwee .

IMO, some general notes for this issue:

• we need a solution that doesn't require large docs refactorings, or promoting "bad Xarray usage" (like using load_dataset everywhere instead of open_dataset). gc.collect() would be nice, but this didn't work. A global conf.py windows-only fix would be better.
• we need to make sure this bug reproduces on other windows machines, and also in a python script (ie outside the docs build)
• I think it's best if whoever fixes this also has a machine to reproduce the problem on. Otherwise I can imagine the review process in GitHub and the debugging process get quite longwinded

[VeckoTheGecko]

Cc @pydata/xarray

I think this issue is impotant, so that we enable Windows users to contribute to docs

[VeckoTheGecko]

we need to make sure this bug reproduces on other windows machines, and also in a python script (ie outside the docs build)

@rwkozar would you mind working on the "reproduces in a python script" part of this? That's be really helpful . Basically copying all the jupyter-execute code cells into a python script and making sure the same error shows

[kmuehlbauer]

We have some precaution in the unit tests:

xarray/xarray/tests/test_backends.py

Lines 1703 to 1717 in fa2c0ca

	@contextlib.contextmanager
	def create_tmp_file(
	suffix: str = ".nc", allow_cleanup_failure: bool = False
	) -> Iterator[str]:
	temp_dir = tempfile.mkdtemp()
	path = os.path.join(temp_dir, f"temp-{next(_counter)}{suffix}")
	try:
	yield path
	finally:
	try:
	shutil.rmtree(temp_dir)
	except OSError:
	if not allow_cleanup_failure:
	raise

On windows allow_cleanup_failure would be set to true.

Maybe we can just re-use this scheme here? It might not work because of the context manager. But then, maybe it's easier to use ignore_cleanup_errors=True in tempfile.TemporaryDirectory()?

[rwkozar]

we need to make sure this bug reproduces on other windows machines, and also in a python script (ie outside the docs build)

@rwkozar would you mind working on the "reproduces in a python script" part of this? That's be really helpful . Basically copying all the jupyter-execute code cells into a python script and making sure the same error shows

@VeckoTheGecko I can try that over the weekend.

[kmuehlbauer]

OK, to make this more explicit, see tempfile.TemporaryDirectory.

We could just invoke it like this, adding ignore_cleanup_errors=True in the call to tempfile.TemporaryDirectory:

.. jupyter-execute::
    :hide-code:

    # Ensure the file is located in a unique temporary directory
    # so that it doesn't conflict with parallel builds of the
    # documentation.

    import tempfile
    import os.path

    tempdir = tempfile.TemporaryDirectory(ignore_cleanup_errors=True)
    datetimes1_filename = os.path.join(tempdir.name, datetimes1_filename)

This is exactly for these kind of purposes IIUC the docstring:

cleanup()

The directory can be explicitly cleaned up by calling the cleanup() method. If ignore_cleanup_errors is
true, any unhandled exceptions during explicit or implicit cleanup (such as a
PermissionError removing open files on Windows) will be ignored, and the remaining removable items deleted on a “best-effort” basis.
Otherwise, errors will be raised in whatever context cleanup occurs (the cleanup() call, exiting the
context manager, when the object is garbage-collected or during interpreter shutdown).

[VeckoTheGecko]

OK, to make this more explicit, see tempfile.TemporaryDirectory.

We could just invoke it like this, adding ignore_cleanup_errors=True in the call to tempfile.TemporaryDirectory:

Ah, that's an easy fix! Thanks for the note @kmuehlbauer - I was on mobile and didn't get a chance to look at source.

I think we should make that update to docs.

By the way, whoever picks this up might want to wait until #11456 lands to avoid merge conflicts :)