scan all tokens

Squashed commit of the following:

commit 0ca080a
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Mon Jan 15 15:56:54 2024 -0600

    WPCS

commit 2523382
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Mon Jan 15 15:54:41 2024 -0600

    Fix span-of-dashes comment modifiable text

commit 2481fcb
Author: Jon Surrell <sirreal@users.noreply.github.com>
Date:   Mon Jan 15 22:07:27 2024 +0100

    Add failing test for `<!----->`

commit 1ced6cf
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Mon Jan 15 11:55:30 2024 -0600

    Expand comment introducing modifiable text.

commit fe79692
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Mon Jan 15 11:38:59 2024 -0600

    Rename INCOMPLETE state to INCOMPLETE_INPUT

commit 1098c19
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Fri Jan 12 13:49:12 2024 -0500

    PR Feedback

    Co-authored-by: Jon Surrell <sirreal@users.noreply.github.com>

commit e9e941a
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Fri Jan 12 13:23:12 2024 -0500

    Remove early bailout of special elements. It's duplicated.

commit 928f12e
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Fri Jan 12 13:10:53 2024 -0500

    Feedback updates.

    Co-authored-by: David Herrera <mail@dlh01.info>
    Co-authored-by: Jon Surrell <sirreal@users.noreply.github.com>

commit e441790
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Fri Jan 12 12:54:22 2024 -0500

    Expand docblocks for CDATA/PINodes and re-add removed tests

commit 3d71c07
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Fri Jan 12 07:53:28 2024 -0500

    Provisionarily: add back CDATA and PI nodes

commit 21bfbca
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Wed Jan 10 12:05:45 2024 -0500

    Fix + WPCS

commit 783e9d6
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Thu Jan 11 21:22:51 2024 -0500

    Remove support for CDATA sections.

commit 72306c3
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Wed Jan 10 11:51:17 2024 -0500

    Remove support for Processing Instructions

    Attempting to parse processing instructions conflicts with parsing bogus
    comments when a document may be incomplete, which might create a
    divergence in the HTML API from browser behavior.

commit 3d5bae3
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Wed Jan 10 11:17:57 2024 -0500

    Fix non-PI-node tests

commit a7554ad
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Wed Jan 10 11:09:20 2024 -0500

    Add basic conformance tests

commit fa4101b
Author: Dennis Snell <dennis.snell@automattic.com>
Date:   Sun Dec 10 15:17:01 2023 +0100

    HTML API: Avoid processing incomplete syntax elements.

    The HTML Tag Processor is able to know if it starts parsing a syntax element
    and reaches the end of the document before it reaches the end of the element.
    In these cases, after this patch, the processor will indicate this condition.

    For example, when processing `<div><input type="te` there is an incomplete INPUT
    element. The processor will fail to find the INPUT, it will pause right after
    the DIV, and `paused_at_incomplete_token()` will return `true`.

    This patch doesn't change any existing behaviors, but it adds the new method
    to report on the final failure condition. It provides a mechanism for later
    use to add chunked parsing to the class, wherein it will be possible to process
    a document without having the entire document loaded in memory, for example
    when processing unbuffered output.

    This is also a necessary change for adding the ability to scan every token in
    the document. Currently the Tag Processor only exposes tags as tokens, but it
    will need to process `#text` nodes, HTML comments, and other markup in order
    to enable behaviors in the HTML Processor and in refactors of existing HTML
    processing in Core.

Author: sirreal

src/wp-includes/html-api/class-wp-html-processor.php

@@ -149,17 +149,6 @@ class WP_HTML_Processor extends WP_HTML_Tag_Processor {
 	 */
 	const MAX_BOOKMARKS = 100;
 
-	/**
-	 * Static query for instructing the Tag Processor to visit every token.
-	 *
-	 * @access private
-	 *
-	 * @since 6.4.0
-	 *
-	 * @var array
-	 */
-	const VISIT_EVERYTHING = array( 'tag_closers' => 'visit' );
-
 	/**
 	 * Holds the working state of the parser, including the stack of
 	 * open elements and the stack of active formatting elements.
@@ -520,7 +509,9 @@ public function step( $node_to_process = self::PROCESS_NEXT_NODE ) {
 				$this->state->stack_of_open_elements->pop();
 			}
 
-			parent::next_tag( self::VISIT_EVERYTHING );
+			while ( parent::next_token() && '#tag' !== $this->get_token_type() ) {
+				continue;
+			}
 		}
 
 		// Finish stepping when there are no more tokens in the document.