feat: add VariantArrayBuilder::with_shredding

[Shekharrajak]

Ref #8480

Rationale for this change

Adds a builder method that configures VariantArrayBuilder to produce a shredded VariantArray at build() time.

What changes are included in this PR?

Adds build_shredded(self, as_type: &DataType) -> Result<VariantArray, ArrowError> as a fallible alternative to build() that shreds the built array in

Are these changes tested?

unit tests

Are there any user-facing changes?

Yes, usage:

let arr = VariantArrayBuilder::new(3)
    .with_shredding(DataType::Int64)
    .tap(|b| { b.append_variant(Variant::Int64(42)); })
    .build(); // returns shredded VariantArray

[Shekharrajak]

@alamb @scovich , please have a look.

[sdf-jkl]

Hi @Shekharrajak, is this purely an ergonomic change? Would be cool to actually build a shredded array in one pass without building it first.

[scovich]

One thought: Instead of adding a new with_shredding method, what if we added a fallible shred (or build_shredded) method alongside the current infallible build, that takes the shredding schema as an arg and returns a VariantArray?

[scovich]

Please no panics in prod code like this. Do we know what could go wrong? Does build need to return Result so we can surface the error properly?

(I expect that the schema could be invalid, for example, which is technically a user error but an easy one to make if you don't fully understand the intricacies of variant shredding)

[scovich]

Would be cool to actually build a shredded array in one pass without building it first

Interesting! In theory it should be possible with the shredded variant builders that shred_variant uses, but it would certainly take more than the current 10 LoC.

[Shekharrajak]

Updated as per review comments.

[scovich]

The updated code looks nice and clean. My only question is: If we want to eventually directly build shredded variant, this API won't work (because the builder doesn't know the shredding schema until the build_shredded call, after all the values were already appended). The original approach was more forward looking in that way, but hid a panic behind an infallible signature.

Is it better for build to become fallible so we can do with_shredding_schema and have the option to someday shred directly as we append values? Or better to leave as-is, so build can remain infallible? @alamb @sdf-jkl @klion26 any thoughts?