feat: add nw.list by CangyuanLi · Pull Request #3694 · narwhals-dev/narwhals

CangyuanLi · 2026-06-19T07:28:23Z

Description

Adds a nw.list function that constructs a list datatype from columns. It mimics duckdb's list_pack semantics. Scalars are collected into a list and lists are collected into a list of lists.

If you have comments or want to explain your changes, please do so in this section

I basically tried to mimic nw.struct wherever possible. For example, since nw.struct is not implemented for dask, neither is nw.list.

I also added a scalars_only flag which is false by default. This allows us to directly use concat_list for the polars backend without calling .implode().over(). It does nothing for all other backends.

What type of PR is this? (check all applicable)

Related issues

Related issue [Enh]: add nw.concat_list #3691
Closes [Enh]: add nw.concat_list #3691

Checklist

Code follows style guide (ruff)
Tests added
Documented the changes

MarcoGorelli

thanks @CangyuanLi !

my main question is - what's scalars_only?

MarcoGorelli · 2026-06-19T12:54:17Z

+        >>> data = {
+        ...     "a": [1, 2, 3],
+        ...     "b": [4, None, 6],
+        ...     "d": [[10, 20], None, [40, 50, 60]],
+        ...     "e": [[100], [200, 300], [None]],
+        ... }
+        >>> df = nw.from_native(pl.DataFrame(data))
+        >>> df.select(
+        ...     nw.list("a", "b").alias("list"), nw.list("d", "e").alias("list_of_lists")
+        ... )


can we keep it simple and just have:

a

b

and use with_columns to make a a_b column?

CangyuanLi · 2026-06-19T14:31:35Z

thanks @CangyuanLi !

my main question is - what's scalars_only?

it is only relevant for thr polars backend and uses concat_list directly instead of the .implode().over() trick. I was worried about the potential performance impact but maybe its leaking implementation details too much?

CangyuanLi · 2026-06-19T21:26:04Z

The stable api tests are failing

=================================== short test summary info ===================================
FAILED tests/stable_api_test.py::test_stable_api_docstrings - AttributeError: module 'narwhals.stable.v2' has no attribute 'list'. Did you mean: 'List'?
FAILED tests/stable_api_test.py::test_stable_api_completeness - AssertionError: assert not {'list'}

would you want me to edit the tests to explicitly xfail list, or add it to the stable api?

CangyuanLi · 2026-06-20T02:27:28Z

thanks @CangyuanLi !
my main question is - what's scalars_only?

it is only relevant for thr polars backend and uses concat_list directly instead of the .implode().over() trick. I was worried about the potential performance impact but maybe its leaking implementation details too much?

The performance impact is relatively minimal for concatenating two columns. It's only really much faster for very small dataframes.

However, it does appear to make a pretty big difference as the number of columns to concatenate increases.

Timings are the median timing of 50 runs, with the CIs representing the 5th and 95th percentile. At 50 columns and 1 million rows, the median time using scalars_only=False 26.26 seconds. Using scalars_only=True, the median time is 8.9694 seconds, a roughly 2.93x speedup.

MarcoGorelli

thanks! a few comments

MarcoGorelli · 2026-06-25T10:25:42Z

-            narwhals_to_native_dtype(dtype, self._version).__class__
-            if isinstance(dtype, type) and issubclass(dtype, DType)
-            else narwhals_to_native_dtype(dtype, self._version)
+            (
+                narwhals_to_native_dtype(dtype, self._version).__class__
+                if isinstance(dtype, type) and issubclass(dtype, DType)
+                else narwhals_to_native_dtype(dtype, self._version)
+            )


unrelated change?

I think this is a change that prek made (probably the ruff-format portion). I can revert it and commit with skip checks.

ruff-format is passing on main so it wouldn't have been that - you can revert this part anyway, the checks need to pass for the pr to be mergeable

MarcoGorelli

thanks for updating!

some comments

can you also make sure some tests include null values?

MarcoGorelli · 2026-06-27T08:33:25Z

-    def corr(self, a: ArrowExpr, b: ArrowExpr, *, method: CorrelationMethod) -> ArrowExpr:
+    def corr(
+        self, a: ArrowExpr, b: ArrowExpr, *, method: CorrelationMethod
+    ) -> ArrowExpr:


can we revert this (and any other cosmetic-only changes)?

so strange, ruff-format is passing on my end so I'm not sure what's going on

but will work on reverting all of the cosmetic changes, sorry for the extra noise!

MarcoGorelli · 2026-06-27T08:34:44Z

-            narwhals_to_native_dtype(dtype, self._version).__class__
-            if isinstance(dtype, type) and issubclass(dtype, DType)
-            else narwhals_to_native_dtype(dtype, self._version)
+            (
+                narwhals_to_native_dtype(dtype, self._version).__class__
+                if isinstance(dtype, type) and issubclass(dtype, DType)
+                else narwhals_to_native_dtype(dtype, self._version)
+            )


ruff-format is passing on main so it wouldn't have been that - you can revert this part anyway, the checks need to pass for the pr to be mergeable

MarcoGorelli · 2026-06-27T08:35:28Z

-            init_series, *series = tuple(chain.from_iterable(expr(df) for expr in exprs))
+            init_series, *series = tuple(
+                chain.from_iterable(expr(df) for expr in exprs)
+            )


MarcoGorelli · 2026-06-27T08:35:34Z

+            init_series, *series = tuple(
+                chain.from_iterable(expr(df) for expr in exprs)
+            )


MarcoGorelli · 2026-06-27T08:35:42Z

-            init_series, *series = align(*chain.from_iterable(expr(df) for expr in exprs))
+            init_series, *series = align(
+                *chain.from_iterable(expr(df) for expr in exprs)
+            )


MarcoGorelli · 2026-06-27T08:36:44Z

+def build_list_array(arrays: list[pa.Array]) -> pa.ChunkedArray:
+    # This works by using concat_arrays to vertically stack the arrays.
+    # Then we use take to grab the data by index and to horizontally
+    # pack the arrays.
+    n = len(arrays[0])
+    num_cols = len(arrays)
+    flat = pa.concat_arrays(arrays)
+    indices = [j * n + i for i in range(n) for j in range(num_cols)]
+    interleaved = pc.take(flat, pa.array(indices))
+    offsets = pa.array(range(0, n * num_cols + 1, num_cols), type=pa.int32())
+    return pa.chunked_array([pa.ListArray.from_arrays(offsets, interleaved)])


nice, clever! out of interest, where does this solution come from?

I found concat_arrays, realized it didn't do what I wanted, then I found ListArray.from_arrays, and found it confusing, and then I asked Claude to mash them together.

CangyuanLi · 2026-06-27T16:46:01Z

i think I fixed all the formatting errors, as well as a typing error. for the stable api tests I followed "from_dict" excluded list from the tests. I also added the nulls in the test data. With the default backend for pandas, something like [None, 5] gets converted to a float, so I only tested pyarrow-backed pandas.

Sorry again for the formatting noise and thanks for your patience!

MarcoGorelli · 2026-06-28T14:21:01Z

thanks for updating! there may be some coverage issue, check if all lines are covered (or use # pragma: no cover if necessary), and if you fetch and merge upstream/main then the unrelated ci issues should be fixed

CangyuanLi · 2026-06-29T06:17:50Z

thanks for updating! there may be some coverage issue, check if all lines are covered (or use # pragma: no cover if necessary), and if you fetch and merge upstream/main then the unrelated ci issues should be fixed

the coverage issue should be fixed! I used no cover for the branch where polars is > 2.0 since it is not out yet

…t exist

CangyuanLi · 2026-06-30T06:16:28Z

Another try to fix CI errors--old polars versions now use pl.count() instead of pl.len(). I also removed what looked to be an extraneous type ignore

CangyuanLi added 2 commits June 19, 2026 00:01

feat: add top-level nw.list

5a180cc

tests: add tests for list constructor

6c9aef5

CangyuanLi mentioned this pull request Jun 19, 2026

[Enh]: add nw.concat_list #3691

Open

1 task

MarcoGorelli requested changes Jun 19, 2026

View reviewed changes

CangyuanLi and others added 2 commits June 19, 2026 14:17

docs: simply list example

790fe7e

Merge branch 'main' into list-constructor

29469b1

CangyuanLi requested a review from MarcoGorelli June 19, 2026 21:22

fix: revert uv.lock to pre-5a180cc6 state

01718ff

MarcoGorelli requested changes Jun 25, 2026

View reviewed changes

CangyuanLi and others added 3 commits June 26, 2026 17:18

Merge branch 'main' into list-constructor

be54bb8

address marco's comments

a50c498

fix: cleanup straggling scalar_only

4f1d95c

MarcoGorelli requested changes Jun 27, 2026

View reviewed changes

CangyuanLi and others added 6 commits June 27, 2026 08:02

Merge branch 'main' into list-constructor

5652287

fix: cleanup straggling scalar_only

492de99

fix: add missing generic args

af7d893

fix: add missing generic args

eb89dd8

tests: update to include nulls in the test data

fa8e713

tests: exclude list from stable api test

8db4ea4

CangyuanLi and others added 2 commits June 28, 2026 23:16

Merge branch 'main' into list-constructor

ab30c8a

fix: skip coverage for forward-compat polars branch

dafe54f

CangyuanLi and others added 3 commits June 29, 2026 23:12

fix(polars): use pl.count for old polars versions where pl.len doesn'…

84d5787

…t exist

fix(typing): remove extra type ingore

7a877ab

Merge branch 'main' into list-constructor

b9f0aa4

Merge branch 'main' into list-constructor

7c500b6

Uh oh!

Conversation

CangyuanLi commented Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

If you have comments or want to explain your changes, please do so in this section

What type of PR is this? (check all applicable)

Related issues

Checklist

Uh oh!

MarcoGorelli left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CangyuanLi Jun 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CangyuanLi commented Jun 19, 2026

Uh oh!

CangyuanLi commented Jun 19, 2026

Uh oh!

CangyuanLi commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MarcoGorelli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

MarcoGorelli left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CangyuanLi commented Jun 27, 2026

Uh oh!

MarcoGorelli commented Jun 28, 2026

Uh oh!

CangyuanLi commented Jun 29, 2026

Uh oh!

CangyuanLi commented Jun 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

CangyuanLi commented Jun 19, 2026 •

edited

Loading

CangyuanLi Jun 19, 2026 •

edited

Loading

CangyuanLi commented Jun 20, 2026 •

edited

Loading