Fix ~6x regression in numpy columnar LIST/ARRAY conversion

The NumpyArray facade read the buffer pointer via numpy's `ctypes.data`
attribute chain and allocated via `numpy.empty(count, dtype_string)`. For a
top-level column that runs once per 2048-row chunk (amortized), but the
LIST/ARRAY per-element converter allocates a fresh array per row, so at 200k
rows it became ~600k ctypes-object allocations: df()/fetchnumpy() of a LIST
column ran ~6x slower than the pybind11 baseline (829ms vs 136ms).

Read the buffer pointer directly from numpy's PyArrayObject C struct (a plain
field read, as pybind11's array.data() did), gated by a PyObject_TypeCheck
against numpy.ndarray so non-ndarray wrappers are never reinterpreted. Cache the
numpy.empty callable and per-dtype np.dtype objects, and skip the no-op
resize-to-current-length on the per-element path.

Output is byte-identical (lists, nested, nulls, empty, masked, large-N); the row
and arrow paths and the int/double/struct columnar paths are unaffected. LIST
df()/fetchnumpy() now match-or-beat the pybind11 baseline (69ms).

Author: evertlammerts

src/include/duckdb_python/numpy/numpy_array.hpp

@@ -11,8 +11,58 @@
 #include "duckdb_python/nb/casters.hpp"
 #include "duckdb.hpp"
 
+#include <unordered_map>
+
 namespace duckdb {
 
+namespace numpy_internal {
+
+//! Mirror of the leading fields of numpy's `PyArrayObject` (stable ABI across numpy 1.x and 2.x).
+//! Only the buffer pointer is needed. Reading `data` is a plain struct field access -- no Python
+//! call, no allocation, no GIL -- exactly what pybind11's `py::array::data()` did internally via
+//! its own equivalent proxy struct. Obtaining the pointer this way (instead of via a `ctypes.data`
+//! attribute chain) is what keeps the numpy columnar path fast for LIST/ARRAY columns, whose
+//! per-element converter allocates a fresh array per row.
+struct NumpyArrayProxy {
+	PyObject_HEAD
+	char *data;
+};
+
+//! Borrowed handle to the `numpy.ndarray` type, fetched once under the GIL and intentionally leaked
+//! for process lifetime (numpy is never unloaded). Used to gate the data-pointer read: the façade
+//! may also wrap non-ndarray objects (e.g. a pandas Index) whose buffer pointer is never read; for
+//! those the read must be skipped so a foreign object is never reinterpreted as a numpy array.
+inline PyTypeObject *NumpyNdarrayType() {
+	static PyTypeObject *cached = []() -> PyTypeObject * {
+		nb::object ndarray = nb::module_::import_("numpy").attr("ndarray");
+		return reinterpret_cast<PyTypeObject *>(ndarray.release().ptr());
+	}();
+	return cached;
+}
+
+//! Allocate an uninitialized 1-D numpy array of `count` elements with the given numpy dtype string.
+//! The bound `numpy.empty` and the `np.dtype` objects (a handful of distinct dtype strings) are
+//! cached to avoid a module import, an attribute lookup, and a dtype-string parse on every call --
+//! this is hot, since a LIST/ARRAY column allocates one array per row (pybind11 constructed the
+//! array at the C level via `py::array(py::dtype, count)`, which paid none of that). Cached handles
+//! are leaked for process lifetime (shutdown-safe: no Python destructor runs after finalization).
+//! Only ever called on the single-threaded, GIL-held result-materialization path.
+inline nb::object NumpyEmpty(idx_t count, const string &dtype) {
+	static PyObject *empty_fn = []() -> PyObject * {
+		nb::object fn = nb::module_::import_("numpy").attr("empty");
+		return fn.release().ptr();
+	}();
+	static auto &dtype_cache = *new std::unordered_map<string, PyObject *>();
+	PyObject *&descr = dtype_cache[dtype];
+	if (!descr) {
+		nb::object d = nb::module_::import_("numpy").attr("dtype")(dtype);
+		descr = d.release().ptr();
+	}
+	return nb::borrow<nb::object>(empty_fn)(count, nb::handle(descr));
+}
+
+} // namespace numpy_internal
+
 //! Thin façade over the numpy array representation.
 //!
 //! This class is the SINGLE place in the codebase that owns the underlying numpy-array
@@ -21,15 +71,15 @@ namespace duckdb {
 //!
 //! Performance note: `Data()`/`MutableData()` are on the HOT path — the numpy scan calls
 //! `Data()` once per column per 2048-row chunk (see numpy_scan.cpp), and DuckDB drives that
-//! scan from multiple threads WITHOUT holding the GIL. Fetching the buffer address via
-//! `arr.ctypes.data` is ~1-5µs, allocates a numpy `_ctypes` object, and *requires the GIL*,
-//! so doing it per chunk would be both a scaling regression and a correctness hazard under a
-//! parallel scan. We therefore compute the pointer ONCE, eagerly, in the constructor (always
-//! invoked single-threaded with the GIL held at bind/result time) and cache it; `Data()` then
-//! becomes a plain pointer read with no Python call and no GIL — matching pybind11's
-//! `nb::array.data()`. The cache is invalidated (and recomputed) by `Resize()`, the only
-//! operation that reallocates the buffer. `ctypes.data` is also dtype-agnostic (works for the
-//! `object` dtype that DLPack/`nb::ndarray` cannot represent).
+//! scan from multiple threads WITHOUT holding the GIL. It is also on the LIST/ARRAY result path,
+//! where a fresh array (and thus a fresh buffer pointer) is materialized per row. The pointer is
+//! read directly from the numpy array's C struct (see `numpy_internal::NumpyArrayProxy`): a plain
+//! field access, no Python call, no allocation, no GIL — exactly what pybind11's
+//! `py::array::data()` did. We compute it ONCE, eagerly, in the constructor (always invoked
+//! single-threaded with the GIL held at bind/result time) and cache it; the cache is invalidated
+//! (and recomputed) by `Resize()`, the only operation that reallocates the buffer. Reading the
+//! struct field is dtype-agnostic (works for the `object` dtype that DLPack/`nb::ndarray` cannot
+//! represent).
 //!
 //! Ownership is move-only-when-asked: the ctor takes by value and moves, GetArray() hands
 //! back a reference, and no method copies the array buffer. The raw `cached_data_` member uses
@@ -54,8 +104,9 @@ class NumpyArray {
 	//! dtype string (e.g. "int64", "float32", "object", "datetime64[us]"). Uninitialized —
 	//! callers fill it immediately, matching the previous `nb::array(nb::dtype(d), count)`.
 	static NumpyArray Allocate(const string &dtype, idx_t count) {
-		auto numpy = nb::module_::import_("numpy");
-		return NumpyArray(numpy.attr("empty")(count, dtype));
+		NumpyArray result(numpy_internal::NumpyEmpty(count, dtype));
+		result.length_ = count;
+		return result;
 	}
 
 	//! Produce a numpy array from an arbitrary Python object (np.asarray semantics: no copy
@@ -77,9 +128,16 @@ class NumpyArray {
 
 	//! Resize the underlying numpy buffer in place. This REALLOCATES the buffer, so the cached
 	//! pointer is invalidated and recomputed (GIL is held -- this only runs on the single-threaded
-	//! result-materialization path).
+	//! result-materialization path). Resizing to the current length is a genuine no-op in numpy;
+	//! we skip the Python `resize` call entirely in that case (buffer and cached pointer unchanged).
+	//! The LIST/ARRAY per-element path allocates each array at its exact final size, so its
+	//! `ToArray()` shrink-to-count is always such a no-op -- hot, hence worth skipping.
 	void Resize(idx_t count) {
+		if (length_ != DConstants::INVALID_INDEX && count == length_) {
+			return;
+		}
 		array.attr("resize")(count, nb::arg("refcheck") = false);
+		length_ = count;
 		cached_data_ = nullptr;
 		EnsurePointer();
 	}
@@ -95,21 +153,27 @@ class NumpyArray {
 
 private:
 	//! Compute and cache the buffer start address of the underlying numpy array, if not already
-	//! cached and an array is held. `ctypes.data` is dtype-agnostic (works for the `object` dtype
-	//! too). Only ever called with the GIL held (construction / Resize).
+	//! cached and a numpy ndarray is held. The pointer is read directly from the array's C struct
+	//! (dtype-agnostic, works for the `object` dtype too), matching pybind11's `py::array::data()`.
+	//! Only ever called with the GIL held (construction / Resize).
 	void EnsurePointer() {
-		// Only numpy ndarrays expose `ctypes`; some NumpyArray wrappers hold other objects (e.g. a pandas Index)
-		// whose buffer pointer is never read. Guard the eager compute so constructing such a wrapper doesn't raise
-		// (the original lazy code only touched `ctypes` if Data()/MutableData() was actually called).
-		if (!cached_data_ && array.ptr() != nullptr && nb::hasattr(array, "ctypes")) {
-			cached_data_ = reinterpret_cast<void *>(nb::cast<uintptr_t>(array.attr("ctypes").attr("data")));
+		// Some NumpyArray wrappers hold non-ndarray objects (e.g. a pandas Index) whose buffer pointer is never read.
+		// Gate the read on an actual numpy ndarray so we never reinterpret a foreign object's memory as an array.
+		if (!cached_data_ && array.ptr() != nullptr &&
+		    PyObject_TypeCheck(array.ptr(), numpy_internal::NumpyNdarrayType())) {
+			cached_data_ = reinterpret_cast<numpy_internal::NumpyArrayProxy *>(array.ptr())->data;
 		}
 	}
 
 	//! The owned numpy array (formerly `nb::array`).
 	nb::object array;
 	//! Cached buffer start address; see the class-level performance note.
 	void *cached_data_ = nullptr;
+	//! Known current element count, tracked so `Resize()` can skip a no-op. Set by `Allocate()` and
+	//! updated by `Resize()`; `INVALID_INDEX` means "unknown" (arrays wrapped from arbitrary objects),
+	//! in which case `Resize()` never skips. The array is only ever resized through `Resize()`, so
+	//! this never goes stale.
+	idx_t length_ = DConstants::INVALID_INDEX;
 };
 
 } // namespace duckdb