feat: implement native empty2null spark inner function#4683
feat: implement native empty2null spark inner function#4683kazantsev-maksim wants to merge 59 commits into
Conversation
This reverts commit 768b3e9.
comphead
left a comment
There was a problem hiding this comment.
Thanks @kazantsev-maksim can we investigate if this function can be implemented through codegen functions rather than native?
It doesn't seem to have intensive computations so codegen implementation should be fine I suppose. The example for codegen #4636
|
Thanks @comphead, it's work fine. |
|
Thanks, starting CI |
|
@comphead Spark uses this function to wrap string fields during partitioning; it cannot be called via Spark SQL, but we need this function to implement native partition writing. |
I see, perhaps we can come up with unit test? LLM suggests me something like Without unit test it would be pretty complicated to ensure no regressions happened And we need to check that execution actually happened with native writer. There are bunch of tests in |
Which issue does this PR close?
Part of: #4670
Rationale for this change
Empty2Nullis an internal Spark expression that converts an empty string "" into null. The logic is trivial: if the value is null or a zero-length string, it returns null; otherwise it returns the string unchanged.Purpose
The function is applied during partitioned file writes (parquet, orc, etc.) — specifically to the partition columns. The reason is the correctness of Hive-style partitioning.
In Hive-style directory naming, an empty string and null are indistinguishable: both would produce a path like
col1=, which is ambiguous and breaks reading the data back. To avoid this, Spark runs partition columns throughEmpty2Nullbefore writing, so empty strings end up in the same default partition as null:col1=__HIVE_DEFAULT_PARTITION__What changes are included in this PR?
How are these changes tested?
Add rust unit tests