[Bug] Cross-cluster Catalog with Arrow Flight fails to read Variant type

[Aaron199]

Search before asking

• I had searched in the issues and found no similar issues.

Version

Environment

• Doris version: 4.0.3
• Deployment: Kubernetes (k8s) Operator
• Cluster Type: DorisDisaggregatedCluster (Cloud Mode / Storage-Compute separation)
• Configuration: Catalog property "use_arrow_flight" = "true" is set.

What's Wrong?

Describe the bug
In Doris 4.0.3, when using a cross-cluster Catalog (type = 'doris') with the Arrow Flight protocol enabled (use_arrow_flight = 'true') to read data from a remote Doris cluster, querying a table that contains a Variant column fails. The query throws a NOT_IMPLEMENTED_ERROR. It appears the underlying Arrow reader currently does not support the serialization/deserialization of the Variant data type.

To Reproduce

1. Create a Catalog connecting to a remote Doris cluster and force enable Arrow Flight:

CREATE CATALOG remote_doris PROPERTIES (
    "type" = "doris",
    "use_arrow_flight" = "true",
    "fe_http_hosts" = "...",
    "fe_arrow_hosts" = "...",
    "fe_thrift_hosts" = "...",
    "user" = "...",
    "password" = "..."
);

2. Ensure the target table being queried on the remote cluster contains a column of type Variant (e.g., ext_fields).
3. Execute a query from the local cluster via this Catalog:

SELECT ext_fields FROM remote_doris.db.table LIMIT 10;

4. The following error is triggered:

ERROR 1105 (HY000): errCode = 2, detailMessage = (...)[NOT_IMPLEMENTED_ERROR]read_column_from_arrow with type variant. cur path: /dummyPath

(Note: Even if attempting to use CAST(ext_fields AS STRING) in the query, the conversion cannot be pushed down before the error occurs, triggering the same interception error.)

What You Expected?

The Arrow Flight transmission link should natively support the cross-cluster reading and parsing of Variant types, allowing the query to return data successfully without throwing a NOT_IMPLEMENTED_ERROR.

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[Arun12-sketch]

hey @Aaron199 , I looked into this and the issue is that Variant type has no Arrow serialization support yet. Since Variant is stored as JSON internally, I think we can serialize it as Arrow LargeUtf8 string during flight transmission. Would need to update arrow_column_to_doris_column.cpp and doris_column_to_arrow_column.cpp to handle TYPE_VARIANT. Does this approach sound good before I start working on a PR?

[yuvarajl]

We are hit by this bug.