Skip to content

Fix: preserve features after IterableDataset.add_column (GH#5752)#8297

Open
uttam12331 wants to merge 1 commit into
huggingface:mainfrom
uttam12331:fix/gh5752-add-column-preserves-features
Open

Fix: preserve features after IterableDataset.add_column (GH#5752)#8297
uttam12331 wants to merge 1 commit into
huggingface:mainfrom
uttam12331:fix/gh5752-add-column-preserves-features

Conversation

@uttam12331

Copy link
Copy Markdown

Problem

After calling .add_column() on a streaming IterableDataset, accessing .features returns None:

from datasets import load_dataset

ds = load_dataset("...", streaming=True)
print(ds.features)  # {'text': Value('string'), ...}

ds2 = ds.add_column("new_column", ["val"] * len(ds))
print(ds2.features)  # None  <-- bug
print(ds2.features.keys())  # AttributeError: 'NoneType' object has no attribute 'keys'

Root Cause

add_column calls self.map(...) without passing the features argument. Inside _map, the new dataset's info is constructed as:

info = self.info.copy()
info.features = features  # None if not passed

This unconditionally overwrites the copied features with None.

Fix

Infer the feature type of the new column from the provided data, merge it with the existing features, and pass the combined Features object to map():

column_features = _infer_features_from_batch({name: list(column) if not isinstance(column, list) else column})
if self.features is not None:
    features = self.features.copy()
    features.update(column_features)
else:
    features = column_features
return self.map(..., features=features)

Tests

  • Added test_iterable_dataset_add_column_preserves_features which explicitly checks that .features is not None and contains both original and new columns after add_column.
  • Existing test_iterable_dataset_add_column continues to pass.

Closes #5752

add_column delegates to map() but does not pass features, causing
map() to set info.features = None and losing the original schema.

Fix by inferring the new column's feature type from the provided data,
merging it with the existing features, and passing the result to map()
so the returned dataset's .features are correctly set.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming dataset looses .feature method after .add_column

1 participant