fix: CSVToDocument row mode no longer crashes the batch on a row with extra fields

[otiscuilei]

Anyone using CSVToDocument with conversion_mode="row" on real-world CSV data hits this: a single data row that contains more fields than the header aborts the entire run() call. This happens with genuinely ragged files, an extra trailing delimiter, or the very common case of an unquoted comma inside a value.

The converter parses rows with csv.DictReader using the default restkey=None. Surplus fields on an over-long row are collected under the Python key None, and _build_document_from_row copies every row key into Document.meta. When the Document is constructed, Document._create_id() runs json.dumps(self.meta, sort_keys=True, ...), and sorting a None key against str keys raises TypeError: '<' not supported between instances of 'NoneType' and 'str'. The converter wraps this in a RuntimeError and re-raises, so the whole batch fails: every other source is skipped and any Documents already built from earlier valid sources are discarded. (File mode is unaffected because it never parses rows.)

The fix passes an explicit string restkey="extra_columns" to csv.DictReader, so surplus values land under a normal string metadata key instead of None. Document id generation then works and the surplus data is preserved rather than crashing the run.

Before:

from haystack.components.converters.csv import CSVToDocument
from haystack.dataclasses import ByteStream

conv = CSVToDocument(conversion_mode="row")
# header has 2 columns, data row has 3 fields (e.g. an unquoted comma)
src = ByteStream(data=b"text,note\r\nhello,city,state\r\n", meta={})
conv.run(sources=[src], content_column="text")
# RuntimeError: CSVToDocument(row): failed to process row 0 ...:
#   '<' not supported between instances of 'NoneType' and 'str'

After:

out = conv.run(sources=[src], content_column="text")
doc = out["documents"][0]
doc.content            # "hello"
doc.meta["note"]       # "city"
doc.meta["extra_columns"]  # ["state"]  -> surplus value preserved, no crash

A regression test (test_row_mode_ragged_row_does_not_crash) feeds a ragged row alongside a valid source and asserts both Documents are produced (the earlier valid source is no longer lost) and that the surplus value is stored under a non-None meta key. It fails on the pre-fix code with the TypeError/RuntimeError and passes with the fix.

A release note is included at releasenotes/notes/csv-to-document-row-mode-extra-fields-d4e5f60718293041.yaml.

[vercel]

@Otis0408 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/converters
csv.py
Project Total

This report was generated by python-coverage-comment-action

[sjrl]

Thanks!