Skip to content

test(spark): characterize PartitionPathUtils partition discovery and materialization#8783

Open
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:spark-partition-path-utils-tests
Open

test(spark): characterize PartitionPathUtils partition discovery and materialization#8783
jackylee-ch wants to merge 1 commit into
vortex-data:developfrom
jackylee-ch:spark-partition-path-utils-tests

Conversation

@jackylee-ch

Copy link
Copy Markdown
Contributor

Rationale for this change

PartitionPathUtils (java/vortex-spark/src/main/java/dev/vortex/spark/read/PartitionPathUtils.java)
discovers Hive-style partition columns from file paths and materializes them as constant
column vectors on the read path. It currently has no test coverage, despite encoding
several subtle behaviors: URL decoding of keys and values, the
__HIVE_DEFAULT_PARTITION__ null sentinel, first-= splitting, type inference
precedence (int → long → double → boolean → string), and per-type parsing into
ConstantColumnVector. This continues the characterization-test series from #8770
(read-path ArrowUtils) and #8782 (write-path SparkToArrowSchema).

What changes are included in this PR?

A new PartitionPathUtilsTest covering all three public methods:

  • parsePartitionValues: extracts ordered key=value segments from paths; ignores
    segments without the key=value shape (including empty keys or values); URL-decodes
    both keys and values (New%20YorkNew York); splits on the first = so values
    may themselves contain =.
  • inferPartitionColumnType: infers Integer for int-width values, Long for
    wider integrals, Double for decimal-point values, Boolean for case-insensitive
    true/false, and falls back to String (including for null and the
    __HIVE_DEFAULT_PARTITION__ sentinel).
  • createConstantVector: null and the Hive default-partition sentinel produce null
    vectors; string/int/long/short/byte/boolean/float/double values are parsed into their
    target types; DateType parses as days-since-epoch (int) and both timestamp types as
    micros-since-epoch (long); unrecognized target types fall back to UTF8 strings.

This is a test-only change; no production code is modified. 19 tests, all passing
locally (:vortex-spark_2.12:test --tests 'dev.vortex.spark.read.PartitionPathUtilsTest').

What APIs are changed? Are there any user-facing changes?

None. No production code or public API is touched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant