perf: preserve dictionary encoding for lower/upper to avoid materializing low-cardinality columns by lyne7-sc · Pull Request #22905 · apache/datafusion

lyne7-sc · 2026-06-11T09:45:45Z

Which issue does this PR close?

Related to How Null/Dictionary/REE datatypes interact with Coercion API #19458
Related to String scalar functions should preserve Dictionary encoding for Dictionary-typed inputs #20935

Rationale for this change

When a Dictionary(K, Utf8) column is passed to a string scalar function, the type-coercion layer currently materializes it to flat Utf8/Utf8View before the function runs, so the operation is applied to every row instead of just the unique dictionary values, and the dictionary encoding is lost on the output. See #19458 for the underlying coercion behavior and #20935 for the string-function-specific impact.

This is wasteful for low-cardinality columns and inflates Arrow IPC/Flight message sizes downstream.

What changes are included in this PR?

Add EncodingPreservation { None, Dictionary } and opt-in constructors on Coercion (new_exact_preserving_encoding, with_encoding_preservation).
In get_valid_types, when a Coercible arg requests Dictionary preservation, run coercion against the dictionary's value type and re-wrap the result as Dictionary(K, V'), so the function receives a DictionaryArray.
Make lower/upper opt in and handle dictionary inputs (array + scalar) by converting only the dictionary values and re-wrapping with the original keys.

Are these changes tested?

Yes

Are there any user-facing changes?

Yes. upper/lower now return Dictionary(...) for dictionary inputs instead of the previously materialized Utf8View. New public API (EncodingPreservation, new Coercion constructors) is added — please add the api change label.

lyne7-sc · 2026-06-11T09:50:01Z

Hi @Jefffrey, following up on #19458. I prototyped one approach and wanted to check it with you before going further. Would you be interested in taking a look?

github-actions · 2026-06-11T10:05:58Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-expr v54.0.0 (current)
       Built [  27.621s] (current)
     Parsing datafusion-expr v54.0.0 (current)
      Parsed [   0.074s] (current)
    Building datafusion-expr v54.0.0 (baseline)
       Built [  26.909s] (baseline)
     Parsing datafusion-expr v54.0.0 (baseline)
      Parsed [   0.078s] (baseline)
    Checking datafusion-expr v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   1.348s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  57.057s] datafusion-expr
    Building datafusion-expr-common v54.0.0 (current)
       Built [  19.314s] (current)
     Parsing datafusion-expr-common v54.0.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-expr-common v54.0.0 (baseline)
       Built [  18.917s] (baseline)
     Parsing datafusion-expr-common v54.0.0 (baseline)
      Parsed [   0.019s] (baseline)
    Checking datafusion-expr-common v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.215s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure enum_struct_variant_field_added: pub enum struct variant field added ---

Description:
An enum's exhaustive struct variant has a new field, which has to be included when constructing or matching on this variant.
        ref: https://doc.rust-lang.org/reference/attributes/type_system.html#the-non_exhaustive-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/enum_struct_variant_field_added.ron

Failed in:
  field encoding_preservation of variant Coercion::Exact in /home/runner/work/datafusion/datafusion/datafusion/expr-common/src/signature.rs:1053
  field encoding_preservation of variant Coercion::Implicit in /home/runner/work/datafusion/datafusion/datafusion/expr-common/src/signature.rs:1063

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  39.179s] datafusion-expr-common
    Building datafusion-functions v54.0.0 (current)
       Built [  30.678s] (current)
     Parsing datafusion-functions v54.0.0 (current)
      Parsed [   0.085s] (current)
    Building datafusion-functions v54.0.0 (baseline)
       Built [  29.939s] (baseline)
     Parsing datafusion-functions v54.0.0 (baseline)
      Parsed [   0.085s] (baseline)
    Checking datafusion-functions v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.419s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  62.482s] datafusion-functions
    Building datafusion-sqllogictest v54.0.0 (current)
       Built [ 177.687s] (current)
     Parsing datafusion-sqllogictest v54.0.0 (current)
      Parsed [   0.021s] (current)
    Building datafusion-sqllogictest v54.0.0 (baseline)
       Built [ 176.598s] (baseline)
     Parsing datafusion-sqllogictest v54.0.0 (baseline)
      Parsed [   0.023s] (baseline)
    Checking datafusion-sqllogictest v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.089s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 357.278s] datafusion-sqllogictest

Jefffrey

just an initial comment, hope to fully review it soon

Jefffrey · 2026-06-13T05:23:32Z

+/// Controls whether a [`Coercion`] preserves an argument's physical encoding
+/// (e.g. dictionary) instead of materializing it to the coerced value type.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Hash)]
+pub enum EncodingPreservation {
+    /// Do not request preservation of a physical encoding.
+    None,
+    /// Preserve dictionary encoding and coerce only the dictionary values.
+    Dictionary,
+}


Something to consider is if an enum is best suited for this 🤔

For example, if we want to also include run encoded arrays as well (since they are similar to dictionaries), would this mean two new variants, one just for run arrays and one for dictionary + run arrays?

I don't know if arrow is planning to include any more types of encodings like this at the moment

So I was also thinking perhaps a bitflag approach (or just a struct with boolean flags) could be another approach:

struct Encoding { preserve_dictionary: bool, preserve_run: bool, }

But I don't know how common a use case would be to implement preservation only for dictionaries and not run arrays (or vice versa) so maybe thats overengineering 🤔

Considering RunEndEncoded and other potential encoding types, I agree that using a struct with boolean flags is a better approach. It keeps the complexity low while making it easier to add future preservation options without introducing enum variants for every possible combination.

Jefffrey · 2026-06-17T04:04:58Z

    }

+    #[test]
+    fn test_coercible_dictionary_preserves_encoding() -> Result<()> {


Could we also have a test for a TypeSignatureClass that isn't native? curious to see how it'll work, since #19458 highlighted that current behaviour for dictionaries is already different for TypeSignatureClass::Native and non-native (e.g. TypeSignatureClass::Integer)

Added tests for both cases:

The behavior now looks like this:

TypeSignatureClass no preservation dictionary preservation

Native(Int64) Int64 Dictionary(Int8, Int64)

Non-Native(Integer) Dictionary(Int8, Int32) Dictionary(Int8, Int32)

Thanks for checking this; I'm a little worried because of the discrepancy here now; is it a big task to try align them? Though we might need to check the UDFs of existing functions that use a non-native typesignatureclass in case they already knew this assumption and had dictionary paths 🤔

It looks like this discrepancy already exists today. Typed non-Native TypeSignatureClass values can pass Dictionary through, while Native coercions materialize to the target type. But this pr makes that contrast more visible by adding an explicit preservation API for the Native path.

If we want EncodingPreservation to be the general mechanism for deciding whether Dictionary is preserved, then I agree we should also think about typed non-Native TypeSignatureClass values. I took a quick pass over the affected built-ins. The good news is that I did not find an obvious core DataFusion function in the affected typed non-Native TypeSignatureClass path that intentionally depends on receiving Dictionary(_, T)🙂. Most seem to dispatch on T directly, and some would likely reject Dictionary(_, T) today after type coercion has accepted it. From that angle, aligning the default behavior may actually fix cases where dictionary inputs currently pass signature matching but fail later in the function implementation.

I did find a couple of Spark compatibility functions, such as spark hex and bitmap_count, that have dictionary paths and would likely need explicit opt-in if we align the default behavior.

From my side, the set of built-ins that may need changes looks bounded, so aligning the behavior seems manageable.

Thanks for checking this; I think we can separate that work into a followup PR 👍

lyne7-sc · 2026-06-24T08:31:08Z

@Jefffrey All feedback has been resolved, please take another look when you have a chance

Jefffrey

sorry i took so long to get back to this, ill try prioritize it now

Jefffrey · 2026-07-05T08:24:51Z

    }

+    #[test]
+    fn test_coercible_dictionary_preserves_encoding() -> Result<()> {


Thanks for checking this; I'm a little worried because of the discrepancy here now; is it a big task to try align them? Though we might need to check the UDFs of existing functions that use a non-native typesignatureclass in case they already knew this assumption and had dictionary paths 🤔

Jefffrey

I think we just need to add a note to the upgrade guide; other than that, this PR looks good to me

Jefffrey · 2026-07-06T14:51:06Z

    }

+    #[test]
+    fn test_coercible_dictionary_preserves_encoding() -> Result<()> {


Thanks for checking this; I think we can separate that work into a followup PR 👍

lyne7-sc · 2026-07-07T07:04:17Z

Thanks @Jefffrey , I added a note to the 55.0.0 upgrade guide for the new Coercion dictionary encoding preservation support.

As a follow-up, I will look into the differing coercion behavior, and check which remaining scalar functions can safely opt in to dictionary preservation. That should help address #20935 more completely.

github-actions Bot added logical-expr Logical plan and expressions sqllogictest SQL Logic Tests (.slt) functions Changes to functions implementation labels Jun 11, 2026

github-actions Bot added the auto detected api change Auto detected API change label Jun 11, 2026

Jefffrey reviewed Jun 13, 2026

View reviewed changes

github-actions Bot added the documentation Improvements or additions to documentation label Jun 15, 2026

Jefffrey added the api change Changes the API exposed to users of the crate label Jun 17, 2026

Jefffrey reviewed Jun 17, 2026

View reviewed changes

lyne7-sc requested a review from Jefffrey June 24, 2026 08:28

Jefffrey reviewed Jul 5, 2026

View reviewed changes

lyne7-sc and others added 6 commits July 6, 2026 22:09

perf: preserve dictionary encoding for case conversion

1896e06

Update signature.rs

4d32c12

docs: use Coercion constructor in UDF examples

8f0f701

use boolean flags for encoding preservation

b1bb38d

address dictionary preservation review feedback

789014d

add tests

adbdd13

lyne7-sc force-pushed the perf/dictionary_scalar branch from 6fc55ee to adbdd13 Compare July 6, 2026 14:21

Jefffrey reviewed Jul 6, 2026

View reviewed changes

added upgrading documentation

364d9d0

TypeSignatureClass	no preservation	dictionary preservation
`Native(Int64)`	`Int64`	`Dictionary(Int8, Int64)`
`Non-Native(Integer)`	`Dictionary(Int8, Int32)`	`Dictionary(Int8, Int32)`

Uh oh!

Conversation

lyne7-sc commented Jun 11, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

lyne7-sc commented Jun 11, 2026

Uh oh!

github-actions Bot commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lyne7-sc Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lyne7-sc commented Jun 24, 2026

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jefffrey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lyne7-sc commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Jun 11, 2026 •

edited

Loading

lyne7-sc Jun 18, 2026 •

edited

Loading