Skip to content

fix: numpy default NaT handling #3863

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
d-v-b merged 4 commits into from
Apr 3, 2026
Merged

fix: numpy default NaT handling #3863

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
47ab487
fix: numpy default NaT handling
ilan-gold Apr 1, 2026
31526c7
fix: timedelta64 handling
ilan-gold Apr 1, 2026
f384b48
fix: use isnat
ilan-gold Apr 1, 2026
67481de
Merge branch 'main' into ig/numpy_nat
d-v-b Apr 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion src/zarr/core/dtype/npy/time.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -545,6 +545,8 @@ def cast_scalar(self, data: object) -> np.timedelta64:
raise a TypeError.
"""
if self._check_scalar(data):
if isinstance(data, np.timedelta64) and np.isnan(data):
return np.timedelta64("NaT", self.unit)
Copy link
Copy Markdown
Contributor Author

@ilan-gold ilan-gold Apr 1, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really sure what to do here since the following strikes me as a bug that won't be fixed (i.e., the below is the codepath of the next line return self._cast_scalar_unchecked(data)):

# /// script
# requires-python = "==3.12"
# dependencies = [
#   "numpy==2.0",
# ]
# ///
import numpy as np

# AssertionError
assert np.isnan(
    np.dtype("timedelta64[10s]").type(
        np.timedelta64("NaT", "s"), "10s"
    )
)

Copy link
Copy Markdown
Contributor

@d-v-b d-v-b Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

either this is a bug in numpy that we should upstream, or there's some equivalent isNaT routine? Alternatively, we check the int64 value of the timedelta, i think all the NaT values land on the same value.

Copy link
Copy Markdown
Contributor Author

@ilan-gold ilan-gold Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think all the NaT values land on the same value

I observed this as well

we should upstream

My "bug that won't be fixed" point was that this appears to be numpy<2.4 specific (i.e., the above script fails on all numpy versions I tried less than 2.4)

Copy link
Copy Markdown
Contributor Author

@ilan-gold ilan-gold Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

# /// script
# requires-python = "==3.12"
# dependencies = [
#   "numpy==2.0", # or 2.3
# ]
# ///
import numpy as np

# AssertionError
assert np.isnat(
    np.dtype("timedelta64[10s]").type(
        np.timedelta64("NaT", "s"), "10s"
    )
)

also fails.

The. output of

np.dtype("timedelta64[10s]").type(
    np.timedelta64("NaT", "s"), "10s"
)

is this magic integer you mentioned

Copy link
Copy Markdown
Contributor

@d-v-b d-v-b Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gotcha, I didn't see the numpy version pin. looks like numpy does have an isnat function so we should probably just use that instead

Copy link
Copy Markdown
Contributor

@d-v-b d-v-b Apr 1, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wait what do we expect here? this isn't actually a NaT value, it's just a huge timedelta:

>>> np.dtype("timedelta64[10s]").type(np.timedelta64("NaT", "s"), "10s")
np.timedelta64(922337203685477579,'10s')

Copy link
Copy Markdown
Contributor Author

@ilan-gold ilan-gold Apr 1, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before this change, the call would have been unit-less:

assert np.isnat(np.dtype("timedelta64[10s]").type(np.timedelta64("NaT"), "10s"))

which works. Bringing in the units is what is causing the non-NaT value in pre-numpy 2.4 versions but now:

# /// script
# requires-python = "==3.12"
# dependencies = [
#   "numpy>=2.4",
# ]
# ///
import numpy as np

assert np.isnat(np.dtype("timedelta64[10s]").type(np.timedelta64("NaT", "ms"), "10s"))
assert np.isnat(np.dtype("timedelta64[10s]").type(np.timedelta64("NaT", "ns"), "10s"))

works so our old code path works fine although now that I'm writing this, is no longer hit when NaT.

return self._cast_scalar_unchecked(data)
msg = (
f"Cannot convert object {data!r} with type {type(data)} to a scalar compatible with the "
Expand All @@ -559,7 +561,7 @@ def default_scalar(self) -> np.timedelta64:
This method provides a default value for the timedelta64 scalar, which is
a 'Not-a-Time' (NaT) value.
"""
return np.timedelta64("NaT")
return np.timedelta64("NaT", self.unit)

def from_json_scalar(self, data: JSON, *, zarr_format: ZarrFormat) -> np.timedelta64:
"""
Expand Down
8 changes: 7 additions & 1 deletion tests/test_dtype/test_npy/test_time.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ class TestTimeDelta64(_TestTimeBase):

cast_value_params = (
(TimeDelta64(unit="ns", scale_factor=1), "1", np.timedelta64(1, "ns")),
(TimeDelta64(unit="ns", scale_factor=1), "NaT", np.timedelta64("NaT")),
(TimeDelta64(unit="ns", scale_factor=1), "NaT", np.timedelta64("NaT", "ns")),
)
invalid_scalar_params = (
(TimeDelta64(unit="Y", scale_factor=1), 1.3),
Expand Down Expand Up @@ -148,6 +148,12 @@ def test_time_scale_factor_too_low() -> None:
TimeDelta64(scale_factor=scale_factor)


def test_default_is_NaT() -> None:
np.testing.assert_equal(
TimeDelta64(unit="ns", scale_factor=1).default_scalar(), np.timedelta64("NaT", "ns")
)


def test_time_scale_factor_too_high() -> None:
"""
Test that an invalid unit raises a ValueError.
Expand Down
Loading