Use Array[String] instead of defaulting to String

[addu390]

Proposed changes

Issue Number: close #341

Problem Summary:

Doris ARRAY columns are currently exposed to Spark as StringType, so SQL functions like size(col) fail with DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE.

This PR adds an opt-in config doris.read.array.native-type (default false) that surfaces them as ArrayType(StringType) instead. Covers both thrift and arrow read modes. Default-off keeps existing users on the legacy JSON-string behavior.

Element-type inference (e.g. array<int> → IntegerType) is intentionally out of scope and left as a follow-up.

Checklist(Required)

1. Does it affect the original behavior: No (gated by a default-off config)
2. Has unit tests been added: Yes (SchemaConvertors, RowConvertors, plus parameterized IT for both read modes)
3. Has document been added or modified: Yes (option description); upstream docs at apache/doris will be a follow-up
4. Does it need to update dependencies: No
5. Are there any changes that cannot be rolled back: No

Further comments

Built on top of the discussion in #341 and the design feedback on the stale #345. Happy to iterate on the option name or scope.

[Copilot]

Pull request overview

Adds an opt-in connector configuration to surface Doris ARRAY columns as Spark ArrayType(StringType) (instead of StringType) so Spark SQL array functions (e.g., size()) work correctly, while preserving backward compatibility via a default-off flag. This is implemented across both read paths (thrift-based scan and Arrow Flight SQL) and includes unit + integration test coverage.

Changes:

• Introduce doris.read.array.native-type (default false) and plumb it through schema inference in both DataSource V1 (DorisRelation) and V2 (DorisTableBase).
• Extend row/value conversion to support Spark ArrayType by converting Doris/Arrow list values into Spark internal ArrayData.
• Add unit tests for schema/value conversion and an IT case validating both the native array mode and the default legacy mode.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
spark-doris-connector/spark-doris-connector-spark-3-base/src/main/scala/org/apache/doris/spark/catalog/DorisTableBase.scala	DataSource V2 schema conversion now considers doris.read.array.native-type for ARRAY columns.
spark-doris-connector/spark-doris-connector-base/src/main/scala/org/apache/doris/spark/sql/sources/DorisRelation.scala	DataSource V1 schema conversion now considers doris.read.array.native-type for ARRAY columns.
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/config/DorisOptions.java	Adds the new connector option definition.
spark-doris-connector/spark-doris-connector-base/src/main/scala/org/apache/doris/spark/util/SchemaConvertors.scala	Adds an arrayNativeType flag to map Doris ARRAY to ArrayType(StringType) when enabled.
spark-doris-connector/spark-doris-connector-base/src/main/scala/org/apache/doris/spark/util/RowConvertors.scala	Adds ArrayType handling to convert Java List into Spark GenericArrayData.
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/client/read/RowBatch.java	Conditionally emits List<String> (native) vs String (legacy) for Doris ARRAY vectors.
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/client/read/DorisFlightSqlReader.java	Passes the option into RowBatch for Arrow Flight SQL reads.
spark-doris-connector/spark-doris-connector-base/src/main/java/org/apache/doris/spark/client/read/AbstractThriftReader.java	Passes the option into RowBatch for thrift reads.
spark-doris-connector/spark-doris-connector-base/src/test/scala/org/apache/doris/spark/util/SchemaConvertorsTest.scala	Adds unit test for ARRAY mapping with/without native type enabled.
spark-doris-connector/spark-doris-connector-base/src/test/scala/org/apache/doris/spark/util/RowConvertorsTest.scala	Adds unit test for converting Java List into Spark ArrayData.
spark-doris-connector/spark-doris-connector-it/src/test/java/org/apache/doris/spark/sql/DorisReaderITCase.scala	Adds IT coverage validating schema + size() behavior with native arrays enabled and default behavior disabled.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[JNSimba]

Thank you for your contribution, ITCase failed to run, PTAL @addu390

[addu390]

Hi @JNSimba, thanks for running the checks, I haven't tested the ITs across all Spark versions. But I covered a few and I pushed a new fix, would it be possible to give access to run checks to contributors?

No worries if not, but figured it would be easier to iterate, especially for ITs across versions.

[JNSimba]

Hi, @addu390 I'm not sure how to set up contributors to run CI automatically, but you can refer to this to run these cases locally: https://github.com/apache/doris-spark-connector/blob/master/.github/workflows/run-itcase.yml

[addu390]

@JNSimba Ran the tests across all spark versions, had to update the scala reader: a2165a6

[JNSimba]

@addu390 Could you add a V1 test covering doris.read.array.native-type=true? A simple case based on testDataFrameSource that verifies the schema is inferred as an array and size(col) works would be enough.

If you've already manually verified the V1 path, that's also fine for correctness.

Other than that, LGTM.

[addu390]

@JNSimba A good catch! Added a new test explicitly covering the V1 API.

[JNSimba]

LGTM