Skip to content

test(spark): characterize SparkToArrowSchema Spark-to-Arrow conversion#8782

Open
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:spark-to-arrow-schema-tests
Open

test(spark): characterize SparkToArrowSchema Spark-to-Arrow conversion#8782
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:spark-to-arrow-schema-tests

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

SparkToArrowSchema (java/vortex-spark/src/main/java/dev/vortex/spark/write/SparkToArrowSchema.java)
converts Spark SQL schemas to Arrow schemas on the write path, and currently has no
test coverage. It is the mirror image of ArrowUtils on the read path, which gained
characterization tests in #8770. Like that mapping table, this conversion is exactly the
kind of code that regresses quietly: a wrong bit width, a dropped timezone, or lost
nullability would silently corrupt written files rather than fail loudly.

What changes are included in this PR?

A new SparkToArrowSchemaTest that characterizes the conversion:

  • Primitive mappings: BooleanBool; Byte/Short/Integer/Long → signed Int
    8/16/32/64; Float/DoubleSINGLE/DOUBLE floating point; StringUtf8;
    BinaryBinary; Decimal → 128-bit Arrow Decimal preserving precision and
    scale; DateDate(DAY); TimestampTimestamp(MICROSECOND, "UTC") and
    TimestampNTZTimestamp(MICROSECOND, null).
  • Schema structure: field names and ordering are preserved; field-level nullability
    is carried over to the Arrow field.
  • Nested types: StructType converts to a Struct field with recursively converted
    children (including per-child nullability); ArrayType converts to a List whose
    element child carries containsNull.
  • Rejected types: unsupported Spark types (e.g. CalendarIntervalType) raise
    UnsupportedOperationException naming the offending type.

This is a test-only change; no production code is modified. 12 tests, all passing
locally (:vortex-spark_2.12:test --tests 'dev.vortex.spark.write.SparkToArrowSchemaTest').

What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant