perf: Additional optimizations for cast from string to int by andygrove · Pull Request #3048 · apache/datafusion-comet

andygrove · 2026-01-06T20:12:07Z

Which issue does this PR close?

N/A

Rationale for this change

This follows on from #3017 and makes some additional optimizations.

See comment #3048 (comment) for criterion benchmark results.

What changes are included in this PR?

use trim_ascii instead of custom trim logic
use two loops in do_parse_string_to_int_legacy and remove mutable parse_sign_and_digits
stop matching on eval mode for every row (do it once per batch instead)

How are these changes tested?

Existing tests.

andygrove · 2026-01-06T20:26:10Z


 macro_rules! cast_utf8_to_int {
-    ($array:expr, $eval_mode:expr, $array_type:ty, $cast_method:ident) => {{
+    ($array:expr, $array_type:ty, $parse_fn:expr) => {{


Pass in the parsing function as a closure rather than passing in the eval mode and then matching on that for each row.

Thats super nice

andygrove · 2026-01-06T20:27:15Z

-    match eval_mode {
-        EvalMode::Legacy => do_parse_string_to_int_legacy(str, min_value),
-        EvalMode::Ansi => do_parse_string_to_int_ansi(str, type_name, min_value),
-        EvalMode::Try => do_parse_string_to_int_try(str, min_value),
-    }


This was happening once per row

andygrove · 2026-01-06T20:53:18Z

Current performance compared to main branch:

cast_string_to_int/legacy/i8
                        time:   [5.9209 µs 5.9392 µs 5.9592 µs]
                        change: [−11.695% −11.370% −11.025%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high mild
cast_string_to_int/legacy/i16
                        time:   [6.0520 µs 6.0623 µs 6.0752 µs]
                        change: [−19.180% −16.832% −14.698%] (p = 0.00 < 0.05)
                        Performance has improved.
cast_string_to_int/legacy/i32
                        time:   [10.351 µs 10.465 µs 10.588 µs]
                        change: [−8.5234% −7.6387% −6.8160%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
  11 (11.00%) low mild
cast_string_to_int/legacy/i64
                        time:   [9.8147 µs 9.8431 µs 9.8750 µs]
                        change: [−20.510% −20.227% −19.963%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

cast_string_to_int/ansi/i8
                        time:   [5.9662 µs 5.9738 µs 5.9833 µs]
                        change: [+3.5123% +3.8189% +4.1332%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 16 outliers among 100 measurements (16.00%)
  5 (5.00%) high mild
  11 (11.00%) high severe
cast_string_to_int/ansi/i16
                        time:   [6.0278 µs 6.0633 µs 6.0965 µs]
                        change: [+4.7100% +5.2768% +5.8408%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 18 outliers among 100 measurements (18.00%)
  17 (17.00%) high mild
  1 (1.00%) high severe
cast_string_to_int/ansi/i32
                        time:   [11.561 µs 12.055 µs 12.488 µs]
                        change: [−4.3485% +0.7326% +5.7014%] (p = 0.77 > 0.05)
                        No change in performance detected.
cast_string_to_int/ansi/i64
                        time:   [8.9904 µs 9.0207 µs 9.0561 µs]
                        change: [−12.821% −12.447% −12.056%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 21 outliers among 100 measurements (21.00%)
  6 (6.00%) high mild
  15 (15.00%) high severe

cast_string_to_int/try/i8
                        time:   [5.3065 µs 5.3077 µs 5.3088 µs]
                        change: [−12.228% −11.963% −11.694%] (p = 0.00 < 0.05)
                        Performance has improved.
cast_string_to_int/try/i16
                        time:   [5.3181 µs 5.3339 µs 5.3510 µs]
                        change: [−14.996% −12.856% −11.115%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 14 outliers among 100 measurements (14.00%)
  9 (9.00%) high mild
  5 (5.00%) high severe
cast_string_to_int/try/i32
                        time:   [8.9089 µs 8.9433 µs 8.9790 µs]
                        change: [−23.521% −21.359% −19.212%] (p = 0.00 < 0.05)
                        Performance has improved.
cast_string_to_int/try/i64
                        time:   [8.9110 µs 8.9753 µs 9.0582 µs]
                        change: [−33.677% −30.213% −26.785%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) high mild
  6 (6.00%) high severe

cast_string_to_int/legacy_decimals/i32
                        time:   [10.346 µs 10.361 µs 10.383 µs]
                        change: [−16.404% −15.946% −15.512%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 16 outliers among 100 measurements (16.00%)
  6 (6.00%) high mild
  10 (10.00%) high severe
cast_string_to_int/legacy_decimals/i64
                        time:   [10.409 µs 10.434 µs 10.467 µs]
                        change: [−18.687% −17.862% −17.208%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 18 outliers among 100 measurements (18.00%)
  7 (7.00%) high mild
  11 (11.00%) high severe

codecov-commenter · 2026-01-06T20:55:13Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.50%. Comparing base (f09f8af) to head (1eb4b39).
⚠️ Report is 832 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3048      +/-   ##
============================================
+ Coverage     56.12%   59.50%   +3.37%     
- Complexity      976     1381     +405     
============================================
  Files           119      167      +48     
  Lines         11743    15522    +3779     
  Branches       2251     2575     +324     
============================================
+ Hits           6591     9236    +2645     
- Misses         4012     4989     +977     
- Partials       1140     1297     +157

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

comphead · 2026-01-08T16:35:36Z

-    .map(|v| v as i16))
+fn parse_string_to_i8_legacy(str: &str) -> SparkResult<Option<i8>> {
+    match do_parse_string_to_int_legacy::<i32>(str, i32::MIN)? {
+        None => Ok(None),


Suggested change

None => Ok(None),

?

comphead

Thanks @andygrove it is similar to #2600 (comment) suggestion and thanks for addressing those issues

andygrove · 2026-01-08T17:02:43Z

Thanks @andygrove it is similar to #2600 (comment) suggestion and thanks for addressing those issues

I'll take a look next.

coderfender · 2026-01-09T05:39:29Z

thank you @andygrove

andygrove added 4 commits January 6, 2026 12:43

use trim_ascii

8ccd4aa

minor

06ae139

use two loops and remove mutable parse_sign_and_digits variable

cbf68bb

remove per-row eval mode check and expand benchmarks

e28d52a

andygrove commented Jan 6, 2026

View reviewed changes

more cleanup

37b90fb

andygrove changed the title ~~minor: small improvements in cast from string to int [WIP]~~ perf: Additional optimizations for cast from string to int [WIP] Jan 6, 2026

fix regression and update benchmark

b56aa04

andygrove changed the title ~~perf: Additional optimizations for cast from string to int [WIP]~~ perf: Additional optimizations for cast from string to int Jan 6, 2026

andygrove mentioned this pull request Jan 7, 2026

perf: Improve criterion benchmarks for cast string to int #3049

Merged

andygrove added 2 commits January 7, 2026 09:11

Merge remote-tracking branch 'apache/main' into cast-to-int-perf

b0548a0

upmerge and improve benchmark

1510f62

andygrove marked this pull request as ready for review January 7, 2026 16:19

comphead reviewed Jan 8, 2026

View reviewed changes

comphead approved these changes Jan 8, 2026

View reviewed changes

address feedback

1eb4b39

andygrove merged commit c7aad67 into apache:main Jan 8, 2026
119 checks passed

andygrove deleted the cast-to-int-perf branch January 8, 2026 18:34

andygrove mentioned this pull request Jan 23, 2026

Comet 0.13.0 Release (January 2026) #2845

Closed

andygrove mentioned this pull request Jan 30, 2026

Benchmark and optimize CAST from String to Integer #330

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

perf: Additional optimizations for cast from string to int#3048

perf: Additional optimizations for cast from string to int#3048
andygrove merged 9 commits into
apache:mainfrom
andygrove:cast-to-int-perf

andygrove commented Jan 6, 2026 •

edited

Loading

Uh oh!

andygrove Jan 6, 2026

Uh oh!

comphead Jan 8, 2026

Uh oh!

andygrove Jan 6, 2026

Uh oh!

andygrove commented Jan 6, 2026

Uh oh!

codecov-commenter commented Jan 6, 2026 •

edited

Loading

Uh oh!

comphead Jan 8, 2026

Uh oh!

comphead left a comment

Uh oh!

andygrove commented Jan 8, 2026

Uh oh!

Uh oh!

coderfender commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

andygrove commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

How are these changes tested?

Uh oh!

andygrove Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

comphead Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

andygrove Jan 6, 2026

Choose a reason for hiding this comment

Uh oh!

andygrove commented Jan 6, 2026

Uh oh!

codecov-commenter commented Jan 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

comphead Jan 8, 2026

Choose a reason for hiding this comment

Uh oh!

comphead left a comment

Choose a reason for hiding this comment

Uh oh!

andygrove commented Jan 8, 2026

Uh oh!

Uh oh!

coderfender commented Jan 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

andygrove commented Jan 6, 2026 •

edited

Loading

codecov-commenter commented Jan 6, 2026 •

edited

Loading