Update arrow-rs to v58 (major)

[renovate]

This PR contains the following updates:

Package	Type	Update	Change
arrow-cast	dependencies	major	57 → 58
parquet	dependencies	major	57 → 58

---

Warning

Some dependencies could not be looked up. Check the Dependency Dashboard for more information.

---

Release Notes

apache/arrow-rs (arrow-cast)

v58.1.0

Compare Source

Full Changelog

Implemented enhancements:

• Reuse compression dict lz4_block #​9566
• [Variant] Add variant_to_arrow Struct type support #​9529
• [Variant] Add unshred_variant support for Binary and LargeBinary types #​9526
• [Variant] Add shred_variant support for LargeUtf8 and LargeBinary types #​9525
• [Variant] variant_get tests clean up #​9517
• parquet_variant: Support LargeUtf8 typed value in unshred_variant #​9513
• parquet-variant: Support string view typed value in unshred_variant #​9512
• Deprecate ArrowTimestampType::make_value in favor of from_naive_datetime #​9490 [arrow]
• Followup for support ['fieldName'] in VariantPath #​9478
• Speedup DELTA_BINARY_PACKED decoding when bitwidth is 0 #​9476 [parquet]
• Support CSV files encoded with charsets other than UTF-8 #​9465 [arrow]
• Expose Avro writer schema when building the reader #​9460 [arrow]
• Python: avoid importing pyarrow classes ever time #​9438
• Add append_nulls to MapBuilder #​9431 [arrow]
• Add append_non_nulls to StructBuilder #​9429 [arrow]
• Add append_value_n to GenericByteBuilder #​9425 [arrow]
• Optimize from_bitwise_binary_op #​9378 [arrow]
• Configurable Arrow representation of UTC timestamps for Avro reader #​9279 [arrow]

Fixed bugs:

• MutableArrayData::extend does not copy child values for ListView arrays #​9561 [arrow]
• ListView interleave bug #​9559 [arrow]
• Flight encoding panics with "no dict id for field" with nested dict arrays #​9555 [arrow] [arrow-flight]
• "DeltaBitPackDecoder only supports Int32Type and Int64Type" but unsigned types are supported too #​9551 [parquet]
• Potential overflow when calling util::bit_mask::set_bits (soundness issue) #​9543 [arrow]
• handle Null type in try_merge for Struct, List, LargeList, and Union #​9523 [arrow]
• Invalid offset in sparse column chunk data for multiple predicates #​9516 [parquet]
• debug_assert_eq! in BatchCoalescer panics in debug mode when batch_size < 4 #​9506 [arrow]
• Parquet Statistics::null_count_opt wrongly returns Some(0) when stats are missing #​9451 [parquet]
• Error "Not all children array length are the same!" when decoding rows spanning across page boundaries in parquet file when using RowSelection #​9370 [parquet]
• Avro schema resolution not properly supported for complex types #​9336 [arrow]

Documentation updates:

• Update planned release schedule in README.md #​9466 (alamb)

Performance improvements:

• Introduce NullBuffer::try_from_unsliced to simplify array construction #​9385 [parquet] [arrow]
• perf: Coalesce page fetches when RowSelection selects all rows #​9578 [parquet] (Dandandan)
• Use chunks_exact for has_true/has_false to enable compiler unrolling #​9570 [arrow] (adriangb)
• pyarrow: Cache the imported classes to avoid importing them each time #​9439 (Tpt)

Closed issues:

• Duplicate macro definition: partially_shredded_variant_array_gen #​9492
• Enable LargeList / ListView / LargeListView for VariantArray::try_new #​9455
• Support variables/expressions in record_batch! macro #​9245 [arrow]

Merged pull requests:

• [Variant] Add unshred_variant support for Binary and LargeBinary types #​9576 (kunalsinghdadhwal)
• [Variant] Add variant_to_arrow Struct type support #​9572 (sdf-jkl)
• Make Sbbf Constructers Public #​9569 [parquet] (cetra3)
• fix: Used checked_add for bounds checks to avoid UB #​9568 [arrow] (etseidl)
• Add mutable operations to BooleanBuffer (Bit*Assign) #​9567 [arrow] (Dandandan)
• chore(deps): update lz4_flex requirement from 0.12 to 0.13 #​9565 [parquet] [arrow] (dependabot[bot])
• arrow-select: fix MutableArrayData interleave for ListView #​9560 [arrow] (asubiotto)
• Move ValueIter into own module, and add public record_count function #​9557 [arrow] (Rafferty97)
• arrow-flight: generate dict_ids for dicts nested inside complex types #​9556 [arrow] [arrow-flight] (asubiotto)
• add shred_variant support for LargeUtf8 and LargeBinary #​9554 (sdf-jkl)
• [minor] Download clickbench file when missing #​9553 [parquet] (Dandandan)
• DeltaBitPackEncoderConversion: Fix panic message on invalid type #​9552 [parquet] (progval)
• Replace interleave overflow panic with error #​9549 [arrow] (xudong963)
• feat(arrow-avro): HeaderInfo to expose OCF header #​9548 [arrow] (mzabaluev)
• chore: Protect main branch with required reviews #​9547 (comphead)
• Add benchmark for infer_json_schema #​9546 [arrow] (Rafferty97)
• chore(deps): bump black from 24.3.0 to 26.3.1 in /parquet/pytest #​9545 [parquet] (dependabot[bot])
• Unroll interleave -25-30% #​9542 [arrow] (Dandandan)
• Optimize take_fixed_size_binary For Predefined Value Lengths #​9535 [arrow] (tobixdev)
• feat: expose arrow schema on async avro reader #​9534 [arrow] (mzabaluev)
• Make with_file_decryption_properties pub instead of pub(crate) #​9532 [parquet] (Dandandan)
• fix: handle Null type in try_merge for Struct, List, LargeList, and Union #​9524 [arrow] (zhuqi-lucas)
• chore: extend record_batch macro to support variables and expressions #​9522 [arrow] (buraksenn)
• [Variant] clean up variant_get tests #​9518 (sdf-jkl)
• support large string for unshred variant #​9515 (friendlymatthew)
• support string view unshred variant #​9514 (friendlymatthew)
• Add has_true() and has_false() to BooleanArray #​9511 [arrow] (adriangb)
• Fix Invalid offset in sparse column chunk data error for multiple predicates #​9509 [parquet] (cetra3)
• fix: remove incorrect debug assertion in BatchCoalescer #​9508 [arrow] (Tim-53)
• [Json] Add benchmarks for list json reader #​9507 [arrow] (liamzwbao)
• fix: first next_back() on new RowsIter panics #​9505 [arrow] (rluvaton)
• Add some benchmarks for decoding delta encoded Parquet #​9500 [parquet] (etseidl)
• chore: remove duplicate macro partially_shredded_variant_array_gen #​9498 (codephage2020)
• Deprecate ArrowTimestampType::make_value in favor of from_naive_datetime #​9491 [arrow] (codephage2020)
• fix: Do not assume missing nullcount stat means zero nullcount #​9481 [parquet] (scovich)
• [Variant] Enahcne bracket access for VariantPath #​9479 (klion26)
• Optimize delta binary decoder in the case where bitwidth=0 #​9477 [parquet] (etseidl)
• Add PrimitiveRunBuilder::with_data_type() to customize the values' DataType #​9473 [arrow] (brunal)
• Convert prettyprint tests in arrow-cast to insta inline snapshots #​9472 [parquet] [arrow] (grtlr)
• Update strum_macros requirement from 0.27 to 0.28 #​9471 [arrow] (dependabot[bot])
• docs(parquet): Fix broken links in README #​9467 [parquet] (SYaoJun)
• Add list-like types support to VariantArray::try_new #​9457 (sdf-jkl)
• Simplify downcast_...!() macro definitions #​9454 [arrow] (brunal)
• feat(parquet): add content defined chunking for arrow writer #​9450 [parquet] (kszucs)
• refactor: simplify iterator using cloned().map(Some) #​9449 [parquet] (SYaoJun)
• feat: Optimize from_bitwise_binary_op with 64-bit alignment #​9441 [arrow] (kunalsinghdadhwal)
• docs: fix markdown link syntax in README #​9440 (SYaoJun)
• Move ListLikeArray to arrow-array to be shared with json writer and parquet unshredding #​9437 [arrow] (liamzwbao)
• Add claim method to recordbatch for memory accounting #​9433 [arrow] (cetra3)
• Add append_nulls to MapBuilder #​9432 [arrow] (Fokko)
• Add append_non_nulls to StructBuilder #​9430 [arrow] (Fokko)
• Add append_value_n to GenericByteBuilder #​9426 [arrow] (Fokko)
• refactor: simplify dynamic state for Avro record projection #​9419 [arrow] (mzabaluev)
• Add NullBuffer::from_unsliced_buffer helper and refactor call sites #​9411 [parquet] [arrow] (Eyad3skr)
• Implement min, max, sum for run-end-encoded arrays. #​9409 [arrow] (brunal)
• feat: add RunArray::new_unchecked and RunArray::into_parts #​9376 [arrow] (rluvaton)
• Fix skip_records over-counting when partial record precedes num_rows page skip #​9374 [parquet] (jonded94)
• fix: resolution of complex type variants in Avro unions #​9328 [arrow] (mzabaluev)
• feat(arrow-avro): Configurable Arrow timezone ID for Avro timestamps #​9280 [arrow] (mzabaluev)

* This Changelog was automatically generated by github_changelog_generator

v58.0.0

Compare Source

Full Changelog

Implemented enhancements:

• Reuse compression dict lz4_block #​9566
• [Variant] Add variant_to_arrow Struct type support #​9529
• [Variant] Add unshred_variant support for Binary and LargeBinary types #​9526
• [Variant] Add shred_variant support for LargeUtf8 and LargeBinary types #​9525
• [Variant] variant_get tests clean up #​9517
• parquet_variant: Support LargeUtf8 typed value in unshred_variant #​9513
• parquet-variant: Support string view typed value in unshred_variant #​9512
• Deprecate ArrowTimestampType::make_value in favor of from_naive_datetime #​9490 [arrow]
• Followup for support ['fieldName'] in VariantPath #​9478
• Speedup DELTA_BINARY_PACKED decoding when bitwidth is 0 #​9476 [parquet]
• Support CSV files encoded with charsets other than UTF-8 #​9465 [arrow]
• Expose Avro writer schema when building the reader #​9460 [arrow]
• Python: avoid importing pyarrow classes ever time #​9438
• Add append_nulls to MapBuilder #​9431 [arrow]
• Add append_non_nulls to StructBuilder #​9429 [arrow]
• Add append_value_n to GenericByteBuilder #​9425 [arrow]
• Optimize from_bitwise_binary_op #​9378 [arrow]
• Configurable Arrow representation of UTC timestamps for Avro reader #​9279 [arrow]

Fixed bugs:

• MutableArrayData::extend does not copy child values for ListView arrays #​9561 [arrow]
• ListView interleave bug #​9559 [arrow]
• Flight encoding panics with "no dict id for field" with nested dict arrays #​9555 [arrow] [arrow-flight]
• "DeltaBitPackDecoder only supports Int32Type and Int64Type" but unsigned types are supported too #​9551 [parquet]
• Potential overflow when calling util::bit_mask::set_bits (soundness issue) #​9543 [arrow]
• handle Null type in try_merge for Struct, List, LargeList, and Union #​9523 [arrow]
• Invalid offset in sparse column chunk data for multiple predicates #​9516 [parquet]
• debug_assert_eq! in BatchCoalescer panics in debug mode when batch_size < 4 #​9506 [arrow]
• Parquet Statistics::null_count_opt wrongly returns Some(0) when stats are missing #​9451 [parquet]
• Error "Not all children array length are the same!" when decoding rows spanning across page boundaries in parquet file when using RowSelection #​9370 [parquet]
• Avro schema resolution not properly supported for complex types #​9336 [arrow]

Documentation updates:

• Update planned release schedule in README.md #​9466 (alamb)

Performance improvements:

• Introduce NullBuffer::try_from_unsliced to simplify array construction #​9385 [parquet] [arrow]
• perf: Coalesce page fetches when RowSelection selects all rows #​9578 [parquet] (Dandandan)
• Use chunks_exact for has_true/has_false to enable compiler unrolling #​9570 [arrow] (adriangb)
• pyarrow: Cache the imported classes to avoid importing them each time #​9439 (Tpt)

Closed issues:

• Duplicate macro definition: partially_shredded_variant_array_gen #​9492
• Enable LargeList / ListView / LargeListView for VariantArray::try_new #​9455
• Support variables/expressions in record_batch! macro #​9245 [arrow]

Merged pull requests:

• [Variant] Add unshred_variant support for Binary and LargeBinary types #​9576 (kunalsinghdadhwal)
• [Variant] Add variant_to_arrow Struct type support #​9572 (sdf-jkl)
• Make Sbbf Constructers Public #​9569 [parquet] (cetra3)
• fix: Used checked_add for bounds checks to avoid UB #​9568 [arrow] (etseidl)
• Add mutable operations to BooleanBuffer (Bit*Assign) #​9567 [arrow] (Dandandan)
• chore(deps): update lz4_flex requirement from 0.12 to 0.13 #​9565 [parquet] [arrow] (dependabot[bot])
• arrow-select: fix MutableArrayData interleave for ListView #​9560 [arrow] (asubiotto)
• Move ValueIter into own module, and add public record_count function #​9557 [arrow] (Rafferty97)
• arrow-flight: generate dict_ids for dicts nested inside complex types #​9556 [arrow] [arrow-flight] (asubiotto)
• add shred_variant support for LargeUtf8 and LargeBinary #​9554 (sdf-jkl)
• [minor] Download clickbench file when missing #​9553 [parquet] (Dandandan)
• DeltaBitPackEncoderConversion: Fix panic message on invalid type #​9552 [parquet] (progval)
• Replace interleave overflow panic with error #​9549 [arrow] (xudong963)
• feat(arrow-avro): HeaderInfo to expose OCF header #​9548 [arrow] (mzabaluev)
• chore: Protect main branch with required reviews #​9547 (comphead)
• Add benchmark for infer_json_schema #​9546 [arrow] (Rafferty97)
• chore(deps): bump black from 24.3.0 to 26.3.1 in /parquet/pytest #​9545 [parquet] (dependabot[bot])
• Unroll interleave -25-30% #​9542 [arrow] (Dandandan)
• Optimize take_fixed_size_binary For Predefined Value Lengths #​9535 [arrow] (tobixdev)
• feat: expose arrow schema on async avro reader #​9534 [arrow] (mzabaluev)
• Make with_file_decryption_properties pub instead of pub(crate) #​9532 [parquet] (Dandandan)
• fix: handle Null type in try_merge for Struct, List, LargeList, and Union #​9524 [arrow] (zhuqi-lucas)
• chore: extend record_batch macro to support variables and expressions #​9522 [arrow] (buraksenn)
• [Variant] clean up variant_get tests #​9518 (sdf-jkl)
• support large string for unshred variant #​9515 (friendlymatthew)
• support string view unshred variant #​9514 (friendlymatthew)
• Add has_true() and has_false() to BooleanArray #​9511 [arrow] (adriangb)
• Fix Invalid offset in sparse column chunk data error for multiple predicates #​9509 [parquet] (cetra3)
• fix: remove incorrect debug assertion in BatchCoalescer #​9508 [arrow] (Tim-53)
• [Json] Add benchmarks for list json reader #​9507 [arrow] (liamzwbao)
• fix: first next_back() on new RowsIter panics #​9505 [arrow] (rluvaton)
• Add some benchmarks for decoding delta encoded Parquet #​9500 [parquet] (etseidl)
• chore: remove duplicate macro partially_shredded_variant_array_gen #​9498 (codephage2020)
• Deprecate ArrowTimestampType::make_value in favor of from_naive_datetime #​9491 [arrow] (codephage2020)
• fix: Do not assume missing nullcount stat means zero nullcount #​9481 [parquet] (scovich)
• [Variant] Enahcne bracket access for VariantPath #​9479 (klion26)
• Optimize delta binary decoder in the case where bitwidth=0 #​9477 [parquet] (etseidl)
• Add PrimitiveRunBuilder::with_data_type() to customize the values' DataType #​9473 [arrow] (brunal)
• Convert prettyprint tests in arrow-cast to insta inline snapshots #​9472 [parquet] [arrow] (grtlr)
• Update strum_macros requirement from 0.27 to 0.28 #​9471 [arrow] (dependabot[bot])
• docs(parquet): Fix broken links in README #​9467 [parquet] (SYaoJun)
• Add list-like types support to VariantArray::try_new #​9457 (sdf-jkl)
• Simplify downcast_...!() macro definitions #​9454 [arrow] (brunal)
• feat(parquet): add content defined chunking for arrow writer #​9450 [parquet] (kszucs)
• refactor: simplify iterator using cloned().map(Some) #​9449 [parquet] (SYaoJun)
• feat: Optimize from_bitwise_binary_op with 64-bit alignment #​9441 [arrow] (kunalsinghdadhwal)
• docs: fix markdown link syntax in README #​9440 (SYaoJun)
• Move ListLikeArray to arrow-array to be shared with json writer and parquet unshredding #​9437 [arrow] (liamzwbao)
• Add claim method to recordbatch for memory accounting #​9433 [arrow] (cetra3)
• Add append_nulls to MapBuilder #​9432 [arrow] (Fokko)
• Add append_non_nulls to StructBuilder #​9430 [arrow] (Fokko)
• Add append_value_n to GenericByteBuilder #​9426 [arrow] (Fokko)
• refactor: simplify dynamic state for Avro record projection #​9419 [arrow] (mzabaluev)
• Add NullBuffer::from_unsliced_buffer helper and refactor call sites #​9411 [parquet] [arrow] (Eyad3skr)
• Implement min, max, sum for run-end-encoded arrays. #​9409 [arrow] (brunal)
• feat: add RunArray::new_unchecked and RunArray::into_parts #​9376 [arrow] (rluvaton)
• Fix skip_records over-counting when partial record precedes num_rows page skip #​9374 [parquet] (jonded94)
• fix: resolution of complex type variants in Avro unions #​9328 [arrow] (mzabaluev)
• feat(arrow-avro): Configurable Arrow timezone ID for Avro timestamps #​9280 [arrow] (mzabaluev)

* This Changelog was automatically generated by github_changelog_generator

---

Configuration

📅 Schedule: (UTC)

• Branch creation
  • Between 12:00 AM and 03:59 AM, only on Monday (* 0-3 * * 1)
• Automerge
  • At any time (no schedule defined)

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about these updates again.

---

• If you want to rebase/retry this PR, check this box

---

This PR was generated by Mend Renovate. View the repository job log.

[robert3005]

this is tied to lance which is tied to msrv at this point

[renovate]

Renovate Ignore Notification

Because you closed this PR without merging, Renovate will ignore this update. You will not get PRs for any future 58.x releases. But if you manually upgrade to 58.x then Renovate will re-enable minor and patch updates automatically.

If you accidentally closed this PR, or if you changed your mind: rename this PR to get a fresh replacement PR.