Skip to content

Avoid allocations for complex segmenter runs#8144

Draft
Xuanwo wants to merge 2 commits into
unicode-org:mainfrom
Xuanwo:xuanwo/icu-segmenter-complex-slice
Draft

Avoid allocations for complex segmenter runs#8144
Xuanwo wants to merge 2 commits into
unicode-org:mainfrom
Xuanwo:xuanwo/icu-segmenter-complex-slice

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Changelog

Performance: Avoid temporary complex-run input allocations in the published word and line segmenters.

Summary

This changes the published complex-language hot path to slice the original iterator input instead of building a temporary String or Vec<u16> for each complex run.

This is not a str specialization. The old iterator now keeps the iterator positioned at the current and previous code point, and the complex handoff is implemented through an internal ComplexRunSegmenter abstraction over Y::IterAttr:

  • Utf8 slices CharIndices::as_str() and uses byte offsets.
  • PotentiallyIllFormedUtf8 slices Utf8CharIndices::as_slice() and uses byte offsets.
  • Utf16 slices Utf16Indices::as_slice() and uses code unit offsets.

Potentially ill-formed UTF-8 still goes through the UTF-8 complex segmenter over bytes, so invalid byte sequences continue to be interpreted through Utf8CharIndices as replacement characters. UTF-16 continues to preserve unpaired surrogates in the rule iterator and now passes the original UTF-16 slice to complex segmentation.

The same pattern is applied to both published word and line segmenters. The neo segmenters already route complex runs through Y::IterAttr and are not changed here.

Performance

Baseline: origin/main at 4f4c9f1fa068b6e4e0331b385104a7b2872489c5 plus the benchmark code from draft PR #8141 at 044cc38edec9ae46d03db66e9f5016bcd85e5835. Current: this branch plus the same benchmark code. Criterion command:

cargo bench -p icu_segmenter --bench bench --features unstable -- 'Segmenter/(word|line)/segment_(str|utf8|utf16)/published_dictionary/thai_han' --sample-size 10 --warm-up-time 0.2 --measurement-time 0.5 --noplot

Allocation counts were measured with a temporary local complex_allocs example over thai_han and are per full segmentation call. The allocation probe was not included in this PR diff.

Case (thai_han) Allocs before Allocs after Bytes before Bytes after Criterion before Criterion after Time delta
word segment_str 16 11 760 512 1.0788 us 1.0423 us -3.4%
word segment_utf8 16 11 760 512 1.1390 us 1.1560 us +1.5%
word segment_utf16 16 11 634 512 0.9750 us 0.9984 us +2.4%
line segment_str 15 6 288 120 0.8382 us 0.6498 us -22.5%
line segment_utf8 15 6 288 120 0.9069 us 0.8195 us -9.6%
line segment_utf16 15 6 198 120 0.6945 us 0.6226 us -10.4%

The word segment_utf8 and segment_utf16 thai_han timings are small regressions in this quick Criterion run, but the temporary complex-run allocation is removed for all measured encodings and the measured line paths improve.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant