[Bug] org.apache.doris.shaded.org.apache.arrow.vector.VectorSchemaRoot#getFieldVectors中获取到了__DORIS_DELETE_SIGN__, 导致字段数量比较异常

[Wentao-Zhao]

Search before asking

• I had searched in the issues and found no similar issues.

Version

spark: 3.1.4
doris: doris-2.0.2.1-b387ac2
connector: spark-doris-connector-spark-3.1, version: 25.1.0

What's Wrong?

org.apache.doris.spark.client.read.RowBatch#readBatch
this.fieldVectors contains DORIS_DELETE_SIGN

What You Expected?

spark read doris table, compare field size need filter invisible_col

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dingyufei615]

I encountered the same problem, and I submitted a fix PR.

[dingyufei615]

I'm also using the latest version of Doris 4.0, and I also need to be compatible with Doris 2.0.x. I upgraded the spark-doris-connector and encountered the same problem. Reading and writing to Doris 4.0 is normal, but I encountered problems when reading and writing to Doris 2.0.x.

org.apache.doris.spark.exception.DorisException: Load Doris data failed, schema size of fetch data is wrong.
at org.apache.doris.spark.client.read.RowBatch.<init> (RowBatch.java:123)

After repair and verification #351 it is now running normally.

[JNSimba]

How about trying the latest version 2.0.x?