Avoid allocations for complex segmenter runs#8144
Draft
Xuanwo wants to merge 2 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changelog
Performance: Avoid temporary complex-run input allocations in the published word and line segmenters.
Summary
This changes the published complex-language hot path to slice the original iterator input instead of building a temporary
StringorVec<u16>for each complex run.This is not a
strspecialization. The old iterator now keeps the iterator positioned at the current and previous code point, and the complex handoff is implemented through an internalComplexRunSegmenterabstraction overY::IterAttr:Utf8slicesCharIndices::as_str()and uses byte offsets.PotentiallyIllFormedUtf8slicesUtf8CharIndices::as_slice()and uses byte offsets.Utf16slicesUtf16Indices::as_slice()and uses code unit offsets.Potentially ill-formed UTF-8 still goes through the UTF-8 complex segmenter over bytes, so invalid byte sequences continue to be interpreted through
Utf8CharIndicesas replacement characters. UTF-16 continues to preserve unpaired surrogates in the rule iterator and now passes the original UTF-16 slice to complex segmentation.The same pattern is applied to both published word and line segmenters. The neo segmenters already route complex runs through
Y::IterAttrand are not changed here.Performance
Baseline:
origin/mainat4f4c9f1fa068b6e4e0331b385104a7b2872489c5plus the benchmark code from draft PR #8141 at044cc38edec9ae46d03db66e9f5016bcd85e5835. Current: this branch plus the same benchmark code. Criterion command:cargo bench -p icu_segmenter --bench bench --features unstable -- 'Segmenter/(word|line)/segment_(str|utf8|utf16)/published_dictionary/thai_han' --sample-size 10 --warm-up-time 0.2 --measurement-time 0.5 --noplotAllocation counts were measured with a temporary local
complex_allocsexample overthai_hanand are per full segmentation call. The allocation probe was not included in this PR diff.thai_han)segment_strsegment_utf8segment_utf16segment_strsegment_utf8segment_utf16The word
segment_utf8andsegment_utf16thai_hantimings are small regressions in this quick Criterion run, but the temporary complex-run allocation is removed for all measured encodings and the measured line paths improve.