np.dtype() fails when using Finch-backed arrays with cubed.asarray(...)

[amalia-k510]

Hello,

I'm encountering an issue when using pydata/sparse. The problem is coming from Finch returning Julia-native dtypes like Julia: Float64, which are not recognized by NumPy's np.dtype(...).

The error I am getting is TypeError: Cannot interpret 'Julia: Float64' as a data type. It happens in the normalize_chunks(...) function, where this block is called:

if dtype and not isinstance(dtype, np.dtype):
    dtype = np.dtype(dtype)  # Fails here

To reproduce the error:

import os
os.environ["SPARSE_BACKEND"] = "Finch"

import cubed
import numpy as np
import sparse

os.environ["CUBED_BACKEND_ARRAY_API_MODULE"] = "sparse"

sa_finch = sparse.asarray(np.eye(4), format="coo")
print(type(sa_finch))
wrapped = cubed.asarray(sa_finch)  # fails here

This seems to be caused by a mismatch between assumptions in NumPy-oriented code and behavior in newer array API–compatible backends. Finch, for example, returns dtypes like Julia: Float64, which aren’t compatible with np.dtype(...). In normalize_chunks(...), there’s currently an implicit expectation that all dtype values will work with np.dtype(...), which may not hold for backends like Finch or other non-NumPy implementations.

Tagging @ilan-gold as discussed.

[ilan-gold]

@amalia-k510 great issue, could you provide the a line to those lines using github? Other than that, great stuff. Genuinely curious if we're doing something wrong here, so would be great to understand. Thanks!

[amalia-k510]

Here is the exact line in the cubed code where np.dtype(...) is called link

This is where the Finch-backend Julia: Float64 string causes the TypeError.

[tomwhite]

@amalia-k510 thanks for the error report!

This is a known issue, and is related to #770. I also have a branch (just updated) which is intended to allow any dtype object to be used: https://github.com/cubed-dev/cubed/tree/array-api-strict. You might want to try that out with Finch, however I suspect that it may have the same problem because we still need a way of turning any array-api dtype (such as Finch's) to a numpy dtype in order to write the array to Zarr. Currently that is done here.

Is there another way of converting Finch dtypes to numpy dtypes that doesn't rely on np.dtype(dtype)?

There's a related discussion at pytorch/pytorch#165612, which is proposing that array-api libraries optionally accept numpy dtypes.