Skip to content

Encoding: Fix _wp_scan_utf8() budget scanning#49

Open
sirreal wants to merge 2 commits into
trunkfrom
fix/utf8-scan-codepoint-budget
Open

Encoding: Fix _wp_scan_utf8() budget scanning#49
sirreal wants to merge 2 commits into
trunkfrom
fix/utf8-scan-codepoint-budget

Conversation

@sirreal

@sirreal sirreal commented Jun 11, 2026

Copy link
Copy Markdown
Owner

Summary

This fixes a narrow performance issue in _wp_scan_utf8(), a private but shared UTF-8 primitive used by WordPress UTF-8 internals and PHP compat/polyfill code.

The ASCII fast path now caps its strspn() length by the remaining code point budget:

min( $end - $i, $max_count - $count )

Previously, callers that passed $max_code_points without an equally small $max_bytes limit could still scan an entire remaining ASCII run before applying the code point budget.

Impact

This matters when all of these are true:

  1. _wp_scan_utf8() is called with a non-null $max_code_points.
  2. The current scan position starts with an ASCII run.
  3. That ASCII run is longer than the remaining code point budget.
  4. The caller does not also constrain $max_bytes to an equally small range.

Relevant paths include:

  • antispambot(), which scans one UTF-8 character at a time via _wp_scan_utf8( ..., null, 1 ), regardless of whether mbstring is available.
  • The mb_substr() polyfill path, when native mb_substr() is unavailable and the encoding resolves to UTF-8. _mb_substr() uses _wp_utf8_codepoint_span(), which calls _wp_scan_utf8() with a code point budget.
  • The mb_ord() polyfill path, when called with a long ASCII string, because it only needs the first code point.

Mostly unaffected:

  • wp_is_valid_utf8() / wp_check_invalid_utf8() validation, which must scan the full string and do not use a code point budget.
  • wp_scrub_utf8() fallback, which also scans full invalid spans.
  • mb_strlen() polyfill / _wp_utf8_codepoint_count(), which counts the requested byte span using $max_bytes, not $max_code_points.

The benchmarked case maps most directly to UTF-8 substring/span behavior, such as “give me the first 5 code points from a 10 MB ASCII string” on the polyfill path, plus antispambot()’s one-code-point-at-a-time loop.

Performance

Local benchmark:

_wp_utf8_codepoint_span( str_repeat( 'a', 10 * 1024 * 1024 ), 0, 5 )

Before: 3.183709 ms
After: 0.001250 ms

Testing

Adds direct PHPUnit coverage for _wp_utf8_codepoint_span() covering:

  • zero code point budget
  • long ASCII runs
  • non-zero byte offsets
  • multibyte characters before and at the boundary
  • invalid byte spans before and at the boundary

Ran focused syntax, PHPCS, PHPUnit, and differential validation:

  • 189,847 randomized and edge UTF-8 span cases compared against the pre-change implementation: 0 mismatches
  • 2,750 valid-input sanity checks against mb_substr(): 0 mismatches

This is intended to be a performance-only change. Returned spans and found code point counts should remain identical for all inputs.

Trac ticket:

Use of AI Tools

AI assistance: Yes
Tool(s): Claude (fable 5), Codex (GPT 5.5)
Used for: Fuzz test design and implementation, discovery, detection, diagnosis.


This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.

sirreal added 2 commits June 11, 2026 12:34
When _wp_scan_utf8() is called with max_code_points but no max_bytes, the ASCII fast path previously called strspn() across the entire remaining ASCII run before checking the code point limit. This made _wp_utf8_codepoint_span( large ASCII text, 0, 5 ) scan the full string.

Bound strspn() by the remaining code point budget. ASCII code points are one byte, so this preserves the returned span and found count while avoiding work past the budget.

Local benchmark, 10 MB ASCII _wp_utf8_codepoint_span( ..., 0, 5 ): original 3.183709 ms, patched 0.001250 ms. Differential fuzz: 189847 span/found cases and 2750 valid mb_substr sanity cases, no mismatches.
@sirreal sirreal marked this pull request as ready for review June 11, 2026 11:26
@github-actions

Copy link
Copy Markdown

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Core Committers: Use this line as a base for the props when committing in SVN:

Props sergeybiryukov, dmsnell, jonsurrell.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant