CSS selector fuzz: harden fuzzer follow-ups

sirreal · sirreal · commit 9d1129c21352 · 2026-06-12T21:18:11.000+02:00
diff --git a/tools/css-selector-fuzz/FINDINGS.md b/tools/css-selector-fuzz/FINDINGS.md
@@ -17,9 +17,10 @@ completely clean, and the lexbor differential (third independent oracle) agreed
 with the reference matcher on every compared no-quirks case (0 `lexbor-divergence`).
 Caveats on the strength of that agreement: roughly half of the `compared`
 cases (and ~62% of all match assertions across buckets) are vacuous `[] == []`;
-quirks-mode class/ID matching is excluded from the differential (lexbor #368)
-and so rests on `ReferenceMatcher` alone. See `README.md` for the full
-disclosure.
+older lexbor builds with #368 exclude quirks-mode class/ID matching from the
+differential, though current harnesses include it when the startup probe reports
+reliable class/#id behavior in both no-quirks and quirks mode. See `README.md`
+for the full disclosure.
 
 Reproduce any case: `php tools/css-selector-fuzz/replay.php --selector '<sel>' [--html '<html>']`.
 Auto-minimize a failing seed: `php tools/css-selector-fuzz/minimize.php --seed <seed>`
diff --git a/tools/css-selector-fuzz/NEXT-STEPS.md b/tools/css-selector-fuzz/NEXT-STEPS.md
@@ -14,16 +14,17 @@
 > (`CSS selector:` commits `7419a9fef6` / `0cefeb2fc8` / `16d03e2c5f`), each
 > with PHPUnit regression tests. A post-fix 5000-seed run is clean.
 >
-> **Open follow-up hardening (post-review):** `tests/self-check.php` runs its
-> parse-expectation assertions over a fixed seed window (1–400) that, against
-> an *unfixed* core, dodges the known core bugs only by seed luck. On this
-> branch the bugs are fixed so the collision risk is gone, but the hazard
-> returns whenever the tooling runs against a core without the fixes (e.g.
-> cherry-picked onto trunk before the fixes land) or when a future unfixed bug
-> is found. Decouple self-check from unfixed core bugs — e.g. allowlist known
-> signatures in the parse-expectation loop — as a standalone hardening. This is
-> worth doing on its own (it makes self-check robust to *any* future generator
-> change) and is the prerequisite for randomized class-NUL document injection.
+> **Fuzzer-side follow-up hardening implemented (2026-06-12):**
+> `tests/self-check.php` now allowlists known core parse-bug signatures in its
+> fixed seed-window parse-expectation loop, while unknown mismatches still fail.
+> The safe and wild document generators now inject NUL into random class tokens
+> and expose the decoded U+FFFD token to class-selector generation, without
+> leaking raw class values into the generic attribute-value pool. The lexbor
+> differential includes quirks documents whenever the startup probe confirms
+> class/#id behavior in both no-quirks and quirks mode (local master-built
+> harness `3a2d595fe8c50e5076ac79c02b2ded79a777bb52` passes), and `runner.php`
+> reports per-bucket/per-target vacuous and non-vacuous match assertion rates
+> under `matchStats`.
 >
 > **Candidate finding 4 — FIXED:** per CSS Syntax 3 §4.3.8, `\` followed by
 > EOF is a valid escape (EOF is not a newline), and §4.3.7 says consuming it
@@ -175,19 +176,15 @@
 > was skipped deliberately: it is near-tautological (it could only catch
 > a `from_selectors()` bypass, and no public path bypasses it).
 >
-> **Still open from the original follow-up list:** the tooling items in
-> this file's hardening notes (self-check decoupling, class-NUL injection,
-> vacuous-assertion rate, quirks-mode single-oracle gap). New small item
-> from the 2026-06-11 review: `gen_chaos()`'s whole-codepoint `unicode`
-> branch is dead code — it compares the alphabet *string* against the key
-> `'unicode'` after the value lookup already happened — so the unicode
-> alphabet is byte-sliced by the generic fallback instead. That slicing is
-> what makes chaos emit invalid UTF-8 organically (~15% of chaos cases),
-> so making the branch live is a behavior decision, not just a cleanup:
-> it would remove chaos's organic ill-formed-byte production, leaving the
-> deliberate paths (`invalid-utf8` bucket, `mutated` splice) plus
-> `mutated`'s residual organic corruption of pool multibyte characters
-> (~2% of mutated cases even without the splice).
+> **Still open:** `gen_chaos()`'s whole-codepoint `unicode` branch is dead
+> code — it compares the alphabet *string* against the key `'unicode'` after
+> the value lookup already happened — so the unicode alphabet is byte-sliced
+> by the generic fallback instead. That slicing is what makes chaos emit
+> invalid UTF-8 organically (~15% of chaos cases), so making the branch live
+> is a behavior decision, not just a cleanup: it would remove chaos's organic
+> ill-formed-byte production, leaving the deliberate paths (`invalid-utf8`
+> bucket, `mutated` splice) plus `mutated`'s residual organic corruption of
+> pool multibyte characters (~2% of mutated cases even without the splice).
 
 Repo: `/Users/jonsurrell/a8c/wordpress-develop/html-css-fuzz`, branch
 `html-css-fuzz` (trunk + merged `html-api/add-css-selector-parser`).
diff --git a/tools/css-selector-fuzz/README.md b/tools/css-selector-fuzz/README.md
@@ -45,6 +45,9 @@ produces the same document, the same selector, and the same verdict.
      path-directed generation is that the *combinator/breadcrumb* walker —
      the part most likely to harbor a matching bug — is now exercised with
      real depth, not that every assertion is non-vacuous.
+     `runner.php` persists per-bucket/per-target match assertion counts and
+     vacuous/non-vacuous rates under `matchStats` in `state.json`, so this
+     distribution is reported on every run instead of relying on stale notes.
    - `unsupported` — valid CSS the API intentionally rejects (pseudo-classes
      and -elements, `+`/`~`/`||` combinators, namespaces, non-type context
      selectors); must not parse.
@@ -88,15 +91,16 @@ produces the same document, the same selector, and the same verdict.
      Skipped for ASTs containing invalid UTF-8 (reachable only from
      chaos/mutated inputs), which the renderer cannot round-trip.
    - lexbor differential (third, independent oracle; requires the harness —
-     see below): on no-quirks documents whose selector parsed, a canonical
+     see below): on full-document cases whose selector parsed, a canonical
      re-render of the verified AST is matched by liblexbor and compared,
-     as a multiset of fids, against the reference matcher. Gated on WP and
+     as a multiset of fids, against the reference matcher. Quirks documents
+     participate only when the startup probe confirms lexbor's class/#id
+     folding behavior in both no-quirks and quirks mode. Gated on WP and
      lexbor building the same element tree (fid/tag/ancestry), so it tests
-     the selector layer, not tree construction. Verdicts: `lexbor-divergence`
-     (lexbor ≠ reference) is a fuzzer-oracle problem; `match-mismatch-html`
-     with no accompanying divergence means reference == lexbor ≠ WP — a
-     high-confidence WP finding. (Roughly half of `compared` cases are
-     themselves non-vacuous; the rest assert `[] == []` on both engines.)
+     the selector layer, not tree construction. Verdicts:
+     `lexbor-divergence` (lexbor ≠ reference) is a fuzzer-oracle problem;
+     `match-mismatch-html` with no accompanying divergence means reference
+     == lexbor ≠ WP — a high-confidence WP finding.
    - Repeating a case yields a byte-identical result digest (determinism).
      Note the digest covers the WP-under-test surface (selector, html,
      parse-nullness, ASTs, failure invariants) but **not** the lexbor
@@ -112,7 +116,8 @@ and reports per-batch tallies, persisted to `state.json` under `lexbor`:
 
 - `compared` — the differential ran and matched fid-multisets.
 - `tree-gated` — WP and lexbor built different trees; differential skipped.
-- `skipped-quirks` / `skipped-utf8` — quirks document / non-UTF-8 AST.
+- `skipped-quirks` / `skipped-utf8` — quirks document while lexbor class/#id
+  case behavior is not trusted / non-UTF-8 AST.
 - `n/a` — the differential does not apply (unparseable selector, fragment, no
   captured tree).
 - `unavailable` / `error` — the harness was missing or died. The runner prints
@@ -127,14 +132,9 @@ Known lexbor issues compensated for when present:
   case-sensitive). Detected by a startup probe; when present, lexbor is
   compared against the reference matcher run with quirks-style class/ID
   folding, and quirks-mode documents are excluded from the differential
-  entirely. **Consequence — a real coverage hole:** quirks-mode class/ID
-  matching has no independent third oracle. `ReferenceMatcher` is the sole
-  authority there, and it encodes the same "ASCII-only case fold in quirks"
-  reading WP does (both fold via ASCII-only lowercasing), so if that reading
-  is wrong they would be wrong identically and lexbor — the one engine that
-  could disagree — is excluded. This is inherent to lexbor #368 being open;
-  it is the weakest-covered behavior in the suite and is called out here
-  rather than papered over.
+  entirely. The same startup probe also checks class and `#id` selectors in
+  quirks mode; only when all four probes pass is quirks-mode class/ID matching
+  included in the differential.
 - lexbor rejects uppercase `I`/`S` attribute-selector modifiers, and its
   non-ASCII ident-codepoint table omits U+00B7 and U+00C0–U+00F6 (it
   starts at U+00F8), rejecting e.g. `.Über` while accepting `.über`.
@@ -159,13 +159,11 @@ The match oracle's independence differs between class and attribute selectors:
   splits on ASCII whitespace and folds NUL → U+FFFD per token;
   `ReferenceMatcher::class_matches()` reimplements that independently (and is
   pinned against `class_list()` on NUL/FF boundary inputs by `self-check.php`).
-  The random document generators do **not** emit control bytes inside class
-  values, so the *randomized* fuzzing never exercises this boundary — it is
-  covered only by the deterministic self-check cases. Randomized document-side
-  injection is deliberately deferred: adding it to the hot path perturbs the
-  deterministic self-check seed space enough to surface the known Bug 3, which
-  would first require decoupling `self-check.php` from the unfixed core bugs.
-  A worthwhile, scoped future improvement.
+  The safe and wild random document generators now inject NUL into class
+  tokens occasionally and expose the decoded U+FFFD token to class-selector
+  generation. Raw class attribute values are intentionally kept out of the
+  generic `attrValues` pool so attribute-selector generation does not inherit
+  class-list-only decoding semantics.
 - **Attribute values are matched through a single shared read.** Both WP's
   attribute matcher and `ReferenceMatcher::attr_matches()` read the same
   `get_attribute()` output, so a value-decoding bug there would be shared and
diff --git a/tools/css-selector-fuzz/lib/DocumentGenerator.php b/tools/css-selector-fuzz/lib/DocumentGenerator.php
@@ -314,7 +314,7 @@ private function random_attrs(): array {
 			}
 
 			$this->pools['attrNames'][] = ascii_strtolower( $name );
-			if ( is_string( $value ) ) {
+			if ( is_string( $value ) && 'class' !== $lower ) {
 				$this->pools['attrValues'][] = $value;
 			}
 
@@ -328,9 +328,12 @@ private function random_class_value(): string {
 		$count   = $this->prng->int( 1, 4 );
 		$classes = array();
 		for ( $i = 0; $i < $count; $i++ ) {
-			$class                    = $this->random_word( true );
-			$classes[]                = $class;
-			$this->pools['classes'][] = $class;
+			$class     = $this->random_word( true );
+			$raw_class = $this->maybe_inject_class_nul( $class );
+			$classes[] = $raw_class;
+			foreach ( self::class_tokens( $raw_class ) as $token ) {
+				$this->pools['classes'][] = $token;
+			}
 		}
 
 		$ws    = array( ' ', ' ', ' ', "\t", "\n", "\f", '  ' );
@@ -347,6 +350,23 @@ private function random_class_value(): string {
 		return $value;
 	}
 
+	private function maybe_inject_class_nul( string $class ): string {
+		if ( '' === $class || ! $this->prng->chance( 12 ) ) {
+			return $class;
+		}
+
+		$points = utf8_codepoints( $class );
+		$at     = $this->prng->int( 0, count( $points ) );
+		$out    = '';
+		foreach ( $points as $i => $point ) {
+			if ( $i === $at ) {
+				$out .= "\0";
+			}
+			$out .= $point[0];
+		}
+		return $at === count( $points ) ? $out . "\0" : $out;
+	}
+
 	private function random_id_value(): string {
 		$id                   = $this->random_word( true );
 		$this->pools['ids'][] = $id;
@@ -581,4 +601,28 @@ public static function get_attribute_value( array $element, string $name ) {
 		}
 		return null;
 	}
+
+	/**
+	 * Class tokens as seen by selector matching: ASCII whitespace separates
+	 * tokens, and NUL inside a token is exposed as U+FFFD by class_list().
+	 *
+	 * @return string[]
+	 */
+	public static function class_tokens( string $class_value ): array {
+		$tokens = array();
+		$length = strlen( $class_value );
+		$at     = 0;
+		$ws     = " \t\r\n\f";
+		while ( $at < $length ) {
+			$at += strspn( $class_value, $ws, $at );
+			if ( $at >= $length ) {
+				break;
+			}
+
+			$token_length = strcspn( $class_value, $ws, $at );
+			$tokens[]     = str_replace( "\0", "\u{FFFD}", substr( $class_value, $at, $token_length ) );
+			$at          += $token_length;
+		}
+		return $tokens;
+	}
 }
diff --git a/tools/css-selector-fuzz/lib/LexborOracle.php b/tools/css-selector-fuzz/lib/LexborOracle.php
@@ -19,7 +19,8 @@
  * ASCII case-insensitively even in no-quirks mode ( attribute selectors
  * like [id=x] are correctly case-sensitive ). Detected by probe at startup;
  * when present, lexbor is compared against the reference matcher run with
- * quirks-style class/ID folding.
+ * quirks-style class/ID folding. Quirks documents are compared only when
+ * the probe also confirms class and #id selectors fold in quirks mode.
  */
 class LexborOracle {
 
@@ -33,6 +34,8 @@ class LexborOracle {
 	private static $available = null;
 	/** @var bool */
 	private static $issue368 = false;
+	/** @var bool */
+	private static $quirks_class_id_reliable = false;
 
 	public static function harness_path(): string {
 		return dirname( __DIR__ ) . '/lexbor/harness';
@@ -49,15 +52,31 @@ public static function available(): bool {
 			return false;
 		}
 
-		// Probe: sanity plus issue-#368 detection.
+		// Probe: sanity plus class/#id case-sensitivity behavior.
 		$sane = self::query( '<!DOCTYPE html><div class="a" data-fid="x"></div>', 'div.a' );
 		if ( null === $sane || array( 'x' ) !== $sane['matches'] ) {
 			self::stop();
 			return false;
 		}
 
-		$folded          = self::query( '<!DOCTYPE html><div class="a" data-fid="x"></div>', '.A' );
-		self::$issue368  = null !== $folded && array( 'x' ) === $folded['matches'];
+		$no_quirks_class = self::query( '<!DOCTYPE html><div class="a" data-fid="x"></div>', '.A' );
+		$no_quirks_id    = self::query( '<!DOCTYPE html><div id="a" data-fid="x"></div>', '#A' );
+		$quirks_class    = self::query( '<div class="a" data-fid="x"></div>', '.A' );
+		$quirks_id       = self::query( '<div id="a" data-fid="x"></div>', '#A' );
+		foreach ( array( $no_quirks_class, $no_quirks_id, $quirks_class, $quirks_id ) as $probe ) {
+			if ( null === $probe || null !== $probe['error'] ) {
+				self::stop();
+				return false;
+			}
+		}
+
+		self::$issue368 = array( 'x' ) === $no_quirks_class['matches']
+			|| array( 'x' ) === $no_quirks_id['matches'];
+		self::$quirks_class_id_reliable = ! self::$issue368
+			&& array() === $no_quirks_class['matches']
+			&& array() === $no_quirks_id['matches']
+			&& array( 'x' ) === $quirks_class['matches']
+			&& array( 'x' ) === $quirks_id['matches'];
 		self::$available = true;
 		return true;
 	}
@@ -67,6 +86,11 @@ public static function has_issue_368(): bool {
 		return self::$issue368;
 	}
 
+	/** Whether lexbor can be trusted on quirks class/#id case folding. */
+	public static function quirks_class_id_reliable(): bool {
+		return self::$quirks_class_id_reliable;
+	}
+
 	/**
 	 * Runs one case through lexbor.
 	 *
diff --git a/tools/css-selector-fuzz/lib/ReferenceMatcher.php b/tools/css-selector-fuzz/lib/ReferenceMatcher.php
@@ -223,25 +223,7 @@ private static function class_matches( string $wanted, array $row, bool $quirks
 			return false;
 		}
 
-		$length = strlen( $class_value );
-		$at     = 0;
-		while ( $at < $length ) {
-			$at += strspn( $class_value, self::WHITESPACE, $at );
-			if ( $at >= $length ) {
-				break;
-			}
-			$word_length = strcspn( $class_value, self::WHITESPACE, $at );
-			$word        = substr( $class_value, $at, $word_length );
-			$at         += $word_length;
-
-			/*
-			 * WP_HTML_Tag_Processor::class_list() replaces NUL with U+FFFD in
-			 * each class token before comparison; model that so a class value
-			 * containing a raw NUL matches a `\0`-escaped ( U+FFFD ) selector
-			 * the same way select() does.
-			 */
-			$word = str_replace( "\0", "\u{FFFD}", $word );
-
+		foreach ( DocumentGenerator::class_tokens( $class_value ) as $word ) {
 			if (
 				$quirks
 					? ascii_strtolower( $word ) === ascii_strtolower( $wanted )
diff --git a/tools/css-selector-fuzz/lib/SelectorGenerator.php b/tools/css-selector-fuzz/lib/SelectorGenerator.php
@@ -957,7 +957,7 @@ private function path_compound_for( array $element ): array {
 
 		$class_value = DocumentGenerator::get_attribute_value( $element, 'class' );
 		if ( is_string( $class_value ) ) {
-			foreach ( preg_split( '/[ \t\n\f\r]+/', $class_value, -1, PREG_SPLIT_NO_EMPTY ) as $word ) {
+			foreach ( DocumentGenerator::class_tokens( $class_value ) as $word ) {
 				$features[] = array( 'kind' => 'class', 'name' => $word );
 			}
 		}
@@ -974,6 +974,9 @@ private function path_compound_for( array $element ): array {
 				continue;
 			}
 			$seen_attrs[ $lower ] = true;
+			if ( 'class' === $lower && is_string( $attr[1] ) && false !== strpos( $attr[1], "\0" ) ) {
+				continue;
+			}
 			$features[]           = $this->path_attr_feature( $lower, $attr[1], 'html' === ( $element['namespace'] ?? 'html' ) );
 		}
 
diff --git a/tools/css-selector-fuzz/lib/WildDocumentGenerator.php b/tools/css-selector-fuzz/lib/WildDocumentGenerator.php
@@ -299,9 +299,11 @@ private function random_attrs(): array {
 				$words = array();
 				$n     = $this->prng->int( 1, 3 );
 				for ( $j = 0; $j < $n; $j++ ) {
-					$word                     = $this->random_word();
-					$words[]                  = $word;
-					$this->pools['classes'][] = $word;
+					$word    = $this->maybe_inject_class_nul( $this->random_word() );
+					$words[] = $word;
+					foreach ( DocumentGenerator::class_tokens( $word ) as $token ) {
+						$this->pools['classes'][] = $token;
+					}
 				}
 				$value = implode( ' ', $words );
 			} elseif ( 'id' === $lower ) {
@@ -317,7 +319,7 @@ private function random_attrs(): array {
 			}
 
 			$this->pools['attrNames'][] = $lower;
-			if ( is_string( $value ) ) {
+			if ( is_string( $value ) && 'class' !== $lower ) {
 				$this->pools['attrValues'][] = $value;
 			}
 			$attrs[] = array( $name, $value );
@@ -326,6 +328,23 @@ private function random_attrs(): array {
 		return $attrs;
 	}
 
+	private function maybe_inject_class_nul( string $class ): string {
+		if ( '' === $class || ! $this->prng->chance( 12 ) ) {
+			return $class;
+		}
+
+		$points = utf8_codepoints( $class );
+		$at     = $this->prng->int( 0, count( $points ) );
+		$out    = '';
+		foreach ( $points as $i => $point ) {
+			if ( $i === $at ) {
+				$out .= "\0";
+			}
+			$out .= $point[0];
+		}
+		return $at === count( $points ) ? $out . "\0" : $out;
+	}
+
 	private function render_attrs( array $attrs ): string {
 		$out = '';
 		foreach ( $attrs as $attr ) {
diff --git a/tools/css-selector-fuzz/lib/Worker.php b/tools/css-selector-fuzz/lib/Worker.php
diff --git a/tools/css-selector-fuzz/runner.php b/tools/css-selector-fuzz/runner.php
diff --git a/tools/css-selector-fuzz/tests/self-check.php b/tools/css-selector-fuzz/tests/self-check.php

Original file line number	Diff line number	Diff line change
`@@ -957,7 +957,7 @@ private function path_compound_for( array $element ): array {`
`957`	`957`
`958`	`958`	`$class_value = DocumentGenerator::get_attribute_value( $element, 'class' );`
`959`	`959`	`if ( is_string( $class_value ) ) {`
`960`		`- foreach ( preg_split( '/[ \t\n\f\r]+/', $class_value, -1, PREG_SPLIT_NO_EMPTY ) as $word ) {`
	`960`	`+ foreach ( DocumentGenerator::class_tokens( $class_value ) as $word ) {`
`961`	`961`	`$features[] = array( 'kind' => 'class', 'name' => $word );`
`962`	`962`	`}`
`963`	`963`	`}`
`@@ -974,6 +974,9 @@ private function path_compound_for( array $element ): array {`
`974`	`974`	`continue;`
`975`	`975`	`}`
`976`	`976`	`$seen_attrs[ $lower ] = true;`
	`977`	`+ if ( 'class' === $lower && is_string( $attr[1] ) && false !== strpos( $attr[1], "\0" ) ) {`
	`978`	`+ continue;`
	`979`	`+ }`
`977`	`980`	`$features[] = $this->path_attr_feature( $lower, $attr[1], 'html' === ( $element['namespace'] ?? 'html' ) );`
`978`	`981`	`}`
`979`	`982`