Encoding: Fix _wp_scan_utf8() budget scanning#49
Open
sirreal wants to merge 2 commits into
Open
Conversation
When _wp_scan_utf8() is called with max_code_points but no max_bytes, the ASCII fast path previously called strspn() across the entire remaining ASCII run before checking the code point limit. This made _wp_utf8_codepoint_span( large ASCII text, 0, 5 ) scan the full string. Bound strspn() by the remaining code point budget. ASCII code points are one byte, so this preserves the returned span and found count while avoiding work past the budget. Local benchmark, 10 MB ASCII _wp_utf8_codepoint_span( ..., 0, 5 ): original 3.183709 ms, patched 0.001250 ms. Differential fuzz: 189847 span/found cases and 2750 valid mb_substr sanity cases, no mismatches.
|
The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the Core Committers: Use this line as a base for the props when committing in SVN: To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This fixes a narrow performance issue in
_wp_scan_utf8(), a private but shared UTF-8 primitive used by WordPress UTF-8 internals and PHP compat/polyfill code.The ASCII fast path now caps its
strspn()length by the remaining code point budget:Previously, callers that passed
$max_code_pointswithout an equally small$max_byteslimit could still scan an entire remaining ASCII run before applying the code point budget.Impact
This matters when all of these are true:
_wp_scan_utf8()is called with a non-null$max_code_points.$max_bytesto an equally small range.Relevant paths include:
antispambot(), which scans one UTF-8 character at a time via_wp_scan_utf8( ..., null, 1 ), regardless of whethermbstringis available.mb_substr()polyfill path, when nativemb_substr()is unavailable and the encoding resolves to UTF-8._mb_substr()uses_wp_utf8_codepoint_span(), which calls_wp_scan_utf8()with a code point budget.mb_ord()polyfill path, when called with a long ASCII string, because it only needs the first code point.Mostly unaffected:
wp_is_valid_utf8()/wp_check_invalid_utf8()validation, which must scan the full string and do not use a code point budget.wp_scrub_utf8()fallback, which also scans full invalid spans.mb_strlen()polyfill /_wp_utf8_codepoint_count(), which counts the requested byte span using$max_bytes, not$max_code_points.The benchmarked case maps most directly to UTF-8 substring/span behavior, such as “give me the first 5 code points from a 10 MB ASCII string” on the polyfill path, plus
antispambot()’s one-code-point-at-a-time loop.Performance
Local benchmark:
Before:
3.183709 msAfter:
0.001250 msTesting
Adds direct PHPUnit coverage for
_wp_utf8_codepoint_span()covering:Ran focused syntax, PHPCS, PHPUnit, and differential validation:
mb_substr(): 0 mismatchesThis is intended to be a performance-only change. Returned spans and found code point counts should remain identical for all inputs.
Trac ticket:
Use of AI Tools
AI assistance: Yes
Tool(s): Claude (fable 5), Codex (GPT 5.5)
Used for: Fuzz test design and implementation, discovery, detection, diagnosis.
This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.