Skip to content

Infer CSV headers from sampled column types#10361

Draft
Jo2234 wants to merge 1 commit into
apache:mainfrom
Jo2234:feat/infer-csv-header
Draft

Infer CSV headers from sampled column types#10361
Jo2234 wants to merge 1 commit into
apache:mainfrom
Jo2234:feat/infer-csv-header

Conversation

@Jo2234

@Jo2234 Jo2234 commented Jul 17, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

Format currently requires callers to set with_header explicitly, even though schema inference already samples CSV records and applies Arrow's type inference rules. This makes the default easy to get wrong for the common case where a CSV begins with textual column names.

What changes are included in this PR?

  • Add an opt-in Format::infer_format API that samples records and updates the format's header setting.
  • Classify the first record as a header only when at least one first-row value is text while the sampled values in that column resolve to a non-text, non-null Arrow type. This follows the type-consistency signal used by DuckDB's CSV sniffer while deliberately leaving all-string inputs ambiguous.
  • Reuse the existing CSV parser configuration and type inference, including delimiter, quote, escape, terminator, comment, null, and truncated-row settings.
  • Keep ambiguous inputs conservative: empty, single-record, and all-string samples remain headerless.
  • Document the API with a runnable example.

The supplied reader is consumed for inference, like infer_schema; callers can rewind or create a new reader before parsing.

Are these changes tested?

Yes. Added behavioral coverage for:

  • headers with typed data columns;
  • data without a header;
  • empty, single-record, and all-string ambiguity;
  • configured CSV delimiters; and
  • bounded sampling via max_records.

Validation run locally:

  • CARGO_BUILD_JOBS=2 cargo test -p arrow-csv --all-features — 77 unit tests and 13 doctests passed.
  • CARGO_BUILD_JOBS=2 cargo test -p arrow --features csv — 157 unit/integration tests and 11 doctests passed.
  • CARGO_BUILD_JOBS=2 cargo clippy -p arrow-csv --all-targets --all-features -- -D warnings — passed.
  • CARGO_BUILD_JOBS=2 RUSTDOCFLAGS='-D warnings' cargo doc -p arrow-csv --no-deps — passed.
  • cargo fmt --all -- --check — passed.
  • git diff --check — passed.

Are there any user-facing changes?

Yes. This adds a non-breaking public API. Existing defaults and parsing behavior are unchanged unless a caller explicitly invokes infer_format.

AI assistance disclosure

AI assistance was used for implementation and test drafting. The resulting change was reviewed against the existing CSV parser and type-inference behavior, and the validation commands and results above were run on this branch.

@github-actions github-actions Bot added the arrow Changes to the arrow crate label Jul 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

arrow Changes to the arrow crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Option to Automatically Infer if CSV file has a header

1 participant