[GLUTEN-12447] Add Arrow large var types support

[wecharyu]

What changes are proposed in this pull request?

Support Spark conf spark.sql.execution.arrow.useLargeVarTypes when building arrow vectors to avoid OversizedAllocationException for very large string/binary columns.

Fix #12447.

How was this patch tested?

UT.

Was this patch authored or co-authored using generative AI tooling?

No.

[zml1206]

I think this does not cover all schema export paths that should honor spark.sql.execution.arrow.useLargeVarTypes.
This PR wires the flag through SparkSchemaUtil.toArrowSchema, but several code paths still call SparkArrowUtil.toArrowSchema(..., timeZoneId) directly, where the new largeVarTypes parameter defaults to false. For example:

• RowToVeloxColumnarExec.scala
• ColumnarBuildSideRelation.scala
• ColumnarCachedBatchSerializer.scala
• VeloxFormatWriterInjects.scala
• `IcebergDataWriteFactory.scala

[wecharyu]

@zml1206 Thanks for the review, make some change to cover all other calls:

• Mark the Spark config-related APIs in SparkArrowUtil as private to prevent incorrect usage.
• Expose these APIs through SparkSchemaUtil and ensure they respect Spark configs.

[Copilot]

Pull request overview

This PR adds support for Spark’s spark.sql.execution.arrow.useLargeVarTypes by propagating the flag into Arrow schema/field construction so that StringType/BinaryType can be backed by Arrow LargeUtf8/LargeBinary vectors, avoiding OversizedAllocationException when variable-width buffers grow beyond the 2GB limit.

Changes:

• Plumb spark.sql.execution.arrow.useLargeVarTypes through SparkSchemaUtil → SparkArrowUtil to emit LargeUtf8/LargeBinary for Spark StringType/BinaryType.
• Extend Arrow-backed column vector implementations to read/write LargeVarCharVector and LargeVarBinaryVector.
• Update Velox backend schema construction call sites to go through SparkSchemaUtil (and add a UT pass over both useLargeVarTypes=false/true).

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
gluten-arrow/src/main/scala/org/apache/spark/sql/utils/SparkSchemaUtil.scala	Adds config-driven large-var-type enablement and routes schema/field creation through it.
gluten-arrow/src/main/scala/org/apache/spark/sql/utils/SparkArrowUtil.scala	Adds largeVarTypes plumbing and maps String/Binary to Arrow large variants.
gluten-arrow/src/main/scala/org/apache/gluten/utils/ArrowUtil.scala	Rebuilds imported Arrow schema fields via SparkSchemaUtil.toArrowField (now large-var aware).
gluten-arrow/src/main/java/org/apache/gluten/vectorized/ArrowWritableColumnVector.java	Adds accessor/writer support for LargeVarCharVector / LargeVarBinaryVector.
gluten-arrow/src/main/java/org/apache/gluten/vectorized/ArrowColumnVector.java	Adds read-side accessors for large varchar/binary vectors.
backends-velox/src/test/scala/org/apache/gluten/expression/UDFPartialProjectSuite.scala	Runs the string concat test under both values of the large-var-types conf.
backends-velox/src/main/scala/org/apache/spark/sql/execution/unsafe/UnsafeColumnarBuildSideRelation.scala	Switches Arrow schema creation to SparkSchemaUtil (config-aware).
backends-velox/src/main/scala/org/apache/spark/sql/execution/datasources/velox/VeloxFormatWriterInjects.scala	Switches writer Arrow schema creation to SparkSchemaUtil.
backends-velox/src/main/scala/org/apache/spark/sql/execution/ColumnarCachedBatchSerializer.scala	Switches cached-batch deserialization schema creation to SparkSchemaUtil.
backends-velox/src/main/scala/org/apache/spark/sql/execution/ColumnarBuildSideRelation.scala	Switches build-side Arrow schema creation to SparkSchemaUtil.
backends-velox/src/main/scala/org/apache/gluten/execution/RowToVeloxColumnarExec.scala	Switches row-to-columnar Arrow schema creation to SparkSchemaUtil.
backends-velox/src-iceberg/main/scala/org/apache/gluten/connector/write/IcebergDataWriteFactory.scala	Switches Iceberg writer Arrow schema creation to SparkSchemaUtil.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.