1.7.2: ndarray_get_buffer() thinks `->array` is a python object

[dhalbert]

ulab 1.7.2 still defines the routine ndarray_get_buffer(). This code calls mp_get_buffer(self->array, ...). However, my understanding from reading create_zeros_ones_full() is that self->array is the raw array of values. It is not a Python object. mp_get_buffer() expects its first arg to be a Python object, so bad things happen.

https://github.com/v923z/micropython-ulab/blob/743d86487c83e42024ed508ed50499ad0a527d5d/code/ndarray.c#L1989-L1994

mp_int_t ndarray_get_buffer(mp_obj_t self_in, mp_buffer_info_t *bufinfo, mp_uint_t flags) {
    ndarray_obj_t *self = MP_OBJ_TO_PTR(self_in);
    // buffer_p.get_buffer() returns zero for success, while mp_get_buffer returns true for success
    return !mp_get_buffer(self->array, bufinfo, flags);
}

I had some code similar to this in CircuitPython:

a = ulab.zeros(10, dtype=ulab.uint16)
some_bleio_characteristic.value = a

_bleio.Characteristic.value expects the value coming in to obey buffer protocol, and calls mp_get_buffer() on the ndarray. That in turn calls ndarray_get_buffer(), which tries to call mp_get_buffer() on the array itself, and things go wrong.

ndarray_get_buffer() seems to have disappeared in 2.x ulab, so I'm not sure how to fix 1.7.2. Is there something new to backport to 1.7.x? Thanks.

[v923z]

ulab 1.7.2 still defines the routine ndarray_get_buffer(). This code calls mp_get_buffer(self->array, ...). However, my understanding from reading create_zeros_ones_full() is that self->array is the raw array of values. It is not a Python object.

That's right. mp_get_buffer() is leftover of the original code, when the ndarray was derived from micropython's own array object, which I bid farewell to with version 1.0. I somehow overlooked that. Sorry for that, and thanks for pointing out the problem!

_bleio.Characteristic.value expects the value coming in to obey buffer protocol, and calls mp_get_buffer() on the ndarray. That in turn calls ndarray_get_buffer(), which tries to call mp_get_buffer() on the array itself, and things go wrong.

ndarray_get_buffer() seems to have disappeared in 2.x ulab, so I'm not sure how to fix 1.7.2. Is there something new to backport to 1.7.x? Thanks.

I think @jepler has already sorted this out in #329. The long-term question is, however, whether ndarray_get_buffer is needed at all. I believe, the explicit frombuffer function should be able to handle everything that ndarray_get_buffer can do, but perhaps, you still want to be able to just toss a buffer into the constructor.

[dhalbert]

I believe, the explicit frombuffer function should be able to handle everything that ndarray_get_buffer can do, but perhaps, you still want to be able to just toss a buffer into the constructor.

I saw frombuffer but I thought that was doing the reverse of what I wanted, which is to, say, copy an ndarray to a bytearray. I was looking for tobuffer. Maybe I'm confused. And what I need is all internal, not a Python API.

[v923z]

I believe, the explicit frombuffer function should be able to handle everything that ndarray_get_buffer can do, but perhaps, you still want to be able to just toss a buffer into the constructor.

I saw frombuffer but I thought that was doing the reverse of what I wanted, which is to, say, copy an ndarray to a bytearray. I was looking for tobuffer. Maybe I'm confused. And what I need is all internal, not a Python API.

I don't know, whether tobytes would do:

micropython-ulab/code/ndarray.c

Lines 1549 to 1558 in f4604d2

	mp_obj_t ndarray_tobytes(mp_obj_t self_in) {
	// As opposed to numpy, this function returns a bytearray object with the data pointer (i.e., not a copy)
	ndarray_obj_t *self = MP_OBJ_TO_PTR(self_in);
	// Piping into a bytearray makes sense for dense arrays only,
	// so bail out, if that is not the case
	if(!ndarray_is_dense(self)) {
	mp_raise_ValueError(translate("tobytes can be invoked for dense arrays only"));
	}
	return mp_obj_new_bytearray_by_ref(self->itemsize * self->len, self->array);
	}

mp_obj_new_bytearray_by_ref should produce a buffer that you can use, but I am not 100% sure.

One more note on this subject: as the comment states, the tobytes function is not numpy-compatible, because it doesn't produce a copy. I opted for this solution to save RAM, but I am not against changing this. However, I would still like to have a function/method that just lays of the pointer of the ndarray bare, so that you get something that you can directly pipe into a DAC, I2S device, etc. I haven't yet found the corresponding numpy function.

Perhaps, the solution is not in numpy at all, and we are barking up the wrong tree. I am really close to finishing a complete new module, independent of numpy, but compatible with the ndarray object that would make extensions extremely flexible. When all dust settles down, we could delegate such issues to that module. My vision is that one should be able to extend ulab without having to add a single line to the ulab core.

[dhalbert]

Fixed by #329 for my original problem. IO will close, but @v923z, if you want to create a new issue for the tail of the discussion, sounds like a good idea.