Infer CSV headers from sampled column types#10361
Draft
Jo2234 wants to merge 1 commit into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
Formatcurrently requires callers to setwith_headerexplicitly, even though schema inference already samples CSV records and applies Arrow's type inference rules. This makes the default easy to get wrong for the common case where a CSV begins with textual column names.What changes are included in this PR?
Format::infer_formatAPI that samples records and updates the format's header setting.The supplied reader is consumed for inference, like
infer_schema; callers can rewind or create a new reader before parsing.Are these changes tested?
Yes. Added behavioral coverage for:
max_records.Validation run locally:
CARGO_BUILD_JOBS=2 cargo test -p arrow-csv --all-features— 77 unit tests and 13 doctests passed.CARGO_BUILD_JOBS=2 cargo test -p arrow --features csv— 157 unit/integration tests and 11 doctests passed.CARGO_BUILD_JOBS=2 cargo clippy -p arrow-csv --all-targets --all-features -- -D warnings— passed.CARGO_BUILD_JOBS=2 RUSTDOCFLAGS='-D warnings' cargo doc -p arrow-csv --no-deps— passed.cargo fmt --all -- --check— passed.git diff --check— passed.Are there any user-facing changes?
Yes. This adds a non-breaking public API. Existing defaults and parsing behavior are unchanged unless a caller explicitly invokes
infer_format.AI assistance disclosure
AI assistance was used for implementation and test drafting. The resulting change was reviewed against the existing CSV parser and type-inference behavior, and the validation commands and results above were run on this branch.