Oh, oops, thanks 😅

this test failure seems significant: https://github.com/zarr-developers/zarr-python/actions/runs/16172926021/job/45650861381?pr=3216#step:8:420

this test failure seems significant

Yes looks like this approach doesn't work for complex number types

what if we view the array as raw bytes (should be cheap) and compare the raw bytes?

>>> import numpy as np
>>> np.array([0.0]) == np.array([-0.0])
array([ True])
>>> np.array([0.0]).view('V') == np.array([-0.0]).view('V')
array([False])

I wonder if that would somehow break with floating point subnormal-s and the like. Will have to experiment 🤔

Took me a bit, but finally got around to it. Subnormals are fine, and behave as expected; the only difference between Python's float equality and bitwise float equality is that signed zeroes compare as un-equal when comparing their bits, and that nan numbers can sometimes compare as equal when comparing their bits; the former is exactly what we want, and the latter won't occur since the code path is triggered only for signed zero fill values.

>>> import numpy as np
>>> np.array(1e-323).view('V') == np.array(0.0).view('V'), 1e-323 == 0.0
(array(False), False)
>>> np.array(1e-324).view('V') == np.array(0.0).view('V'), 1e-324 == 0.0
(array(True), True)
>>> np.array(-1e-323).view('V') == np.array(-0.0).view('V'), -1e-323 == -0.0
(array(False), False)
>>> np.array(-1e-324).view('V') == np.array(-0.0).view('V'), -1e-324 == -0.0
(array(True), True)
>>> np.array(-0.0).view('V') == np.array(0.0).view('V'), 0.0 == -0.0
(array(False), True)
>>> np.inf * 0.0
nan
>>> np.array(np.nan).view('V') == np.array(np.nan).view('V'), np.nan == np.nan
(array(True), False)
>>> np.array(np.inf * 0.0).view('V') == np.array(np.nan).view('V'), np.inf * 0.0 == np.nan
(array(False), False)

nan numbers can sometimes compare as equal when comparing their bits

This is actually potentially super useful, because the zarr v3 spec distinguishes between different types of nans, even though numpy does not. In order to ensure that arrays round-trip correctly through zarr python, we need to generate exactly the specific nan defined in the metadata. I did a quick check and numpy will preserve the underlying byte representation of different nans, so this should be possible.

np.array([b'\x00\x00\x00\x00\x00\x00\xFF\xFF'], dtype='|V8').view('float').view('V')
array([b'\x00\x00\x00\x00\x00\x00\xFF\xFF'], dtype='|V8')

Oh, that's curious! Probably not something I can quite incorporate in the code here... unless we make all floating point arrays use bitwise comparison for empty chunks.. 🤔

That .view("V") trick fails on GPU with a ZeroDivisionError, presumably in cupy:_core/core.pyx:81 when v_is == dtype.itemsize == 0, as is the case for the "V" dtype...
Use structured np.void dtypes to achieve the same idea doesn't work because np.void isn't hashable, but it works when using a sized "V" dtype, like "V16". Let's see if that's enough to get all tests green 😂 YES IT IS! 🎉 That GPU test was stubborn! 😂😂

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.54%. Comparing base (f087c56) to head (58f45b7).
⚠️ Report is 87 commits behind head on main.

@@           Coverage Diff           @@
##             main    #3216   +/-   ##
=======================================
  Coverage   94.54%   94.54%           
=======================================
  Files          78       78           
  Lines        9419     9423    +4     
=======================================
+ Hits         8905     8909    +4     
  Misses        514      514