why I got a list instead of datasets.arrow_dataset.Column after the map？

[Septemberlemon]

this makes me very confused

[aasimansari1]

This is expected behavior — map() in HuggingFace datasets returns a Dataset, not an individual column object.

After calling .map(), accessing a column via dataset["column_name"] returns a Python list, not a datasets.arrow_dataset.Column. This changed in datasets v2.x.

How to work with the result correctly

# After map, column access returns a list
result = dataset.map(lambda x: {"new_col": x["text"].upper()})
print(type(result["new_col"]))  # <class list>

# To get a proper column object, use .data
import pyarrow as pa
col = result.data["new_col"]  # pyarrow.ChunkedArray

# Or convert to pandas
df = result.to_pandas()
print(df["new_col"])  # pandas Series

If you need the Arrow column object

# Access via the underlying Arrow table
arrow_col = result._data["new_col"]  # ChunkedArray

Why this happens

HuggingFace datasets stores data in Arrow format internally. The [] accessor is a convenience that converts to a Python list for ease of use. The datasets.arrow_dataset.Column type was an older API that is no longer exposed directly.

For most use cases, the list access is what you want. Use .to_pandas() or .to_arrow() if you need more column-level operations.