Allow rewriter to gracefully bail out on content handler errors.

This is the symmetric companion to the previous `MemoryLimitExceeded` bail-out. Until now,
a content handler returning `Err` would abort the rewriter and leave the output sink in a
potentially inconsistent state (the sink has whatever transformed bytes the rewriter had
already produced, but the remaining input bytes are lost). Downstream this typically
manifests as a truncated, broken HTTP response.

This change adds an opt-in `Settings::graceful_bail_out_on_content_handler_error` flag.
When set, before propagating `RewritingError::ContentHandlerError` the rewriter flushes
every input byte it has received but not yet emitted to the sink, as-is. The caller can
then continue the response by writing subsequent bytes directly to its own downstream
sink, bypassing the (now poisoned) rewriter.

The flag lives on `Settings` rather than `MemorySettings` because the underlying error has
nothing to do with memory: a handler returning `Err` is an explicit signal from the
application, not an environmental constraint. The two flags are intentionally independent
so callers can pick the recovery posture per error class.

The implementation requires a small but important refactor of the dispatcher. Previously,
`lexeme_consumed()` did two things: emit the chunk of input before the lexeme, and advance
`remaining_content_start` past the lexeme. The second step happened before the lexeme's
token was actually serialized to the sink, so a handler error from `token_produced()` (or
text emission) left `remaining_content_start` past a lexeme whose bytes had never been
emitted. Flushing from there as-is would lose the lexeme.

We split that into:

1. `emit_chunk_before_lexeme()`: emit the raw input chunk between the previously emitted
   byte and the start of the lexeme, advance `remaining_content_start` to the lexeme's
   start.
2. `consume_lexeme()`: advance `remaining_content_start` past the lexeme's end. Called
   only after the lexeme's token has been successfully emitted.

With this split, on a `ContentHandlerError` mid-lexeme `remaining_content_start` still
points at the failing lexeme's start, and a graceful bail-out can flush the lexeme + the
rest of the chunk raw.

The bail-out decision in `TransformStream::write()` and `end()` is centralized in
`should_bail_out_for()`, which considers each `RewritingError` variant against its own
flag. `ParsingAmbiguity` is never recovered from (the whole point of strict mode is to
refuse uncertain markup).

The C API is left on the original behavior for now. The previous memory flag fit cleanly
into `MemorySettings` (a `#[repr(C)]` struct), so it was an additive change to the C ABI.
The new flag lives on `Settings`, which is not `repr(C)`, so exposing it through the C
API needs either a new function signature or a new settings struct — a maintainer call.
A TODO captures this in `c-api/src/rewriter.rs`, and the CHANGELOG notes it.

Tested with 6 new tests in `rewriter::tests::fatal_errors`:

1. `test_graceful_bail_out_on_element_handler_error`: element handler returns `Err`, sink
   ends up with the transformed prefix + the failing element and everything after it raw.
2. `test_no_graceful_bail_out_on_content_handler_error_by_default`: sanity check that the
   flag is opt-in.
3. `test_graceful_bail_out_on_comment_handler_error`: comment handler error path
   (exercises the same `lexeme_consumed` restructure).
4. `test_graceful_bail_out_on_end_handler_error`: `handle_end()` returns `Err` after
   `flush_remaining_input` has already emitted every input byte; sink already has the
   complete document.
5. `test_bail_out_reconstruct_handler_error_midstream`: end-to-end reconstruction
   (`sink_output == original_html`).
6. `test_bail_out_flags_independent`: enabling content-handler bail-out doesn't
   accidentally enable memory bail-out, and vice versa.

Caveats documented on the new field's doc comment:

1. Active content removal at the time of the bail-out can produce mismatched tags. Same
   caveat as the memory flag; uncommon in practice and not byte-truncating.
2. Multi-chunk text emission where a later chunk's handler errors will re-emit the raw
   input bytes, duplicating the already-emitted chunks. Rare (only on encoding-converted
   text); the response is byte-bigger but not truncated.

Author: orium

CHANGELOG.md

@@ -6,6 +6,10 @@
   flushes every input byte it has received but not yet emitted to the sink (as-is) before
   returning `MemoryLimitExceededError`, so callers can continue the response by writing
   subsequent bytes directly to their downstream sink instead of breaking it.
+- Added `Settings::graceful_bail_out_on_content_handler_error`: symmetric to the memory flag
+  above, but for `RewritingError::ContentHandlerError`. When set, the rewriter flushes
+  remaining input bytes before propagating a handler error, preserving the response.
+  Currently exposed via the Rust API only; the C API still uses the original behavior.
 
 ## v2.9.0
 

c-api/src/rewriter.rs

@@ -61,6 +61,10 @@ fn lol_html_rewriter_build_inner(
         strict,
         enable_esi_tags,
         adjust_charset_on_meta_tag: false,
+        // TODO: expose `graceful_bail_out_on_content_handler_error` through the C API. Adding
+        // a new parameter to `lol_html_rewriter_build()` is a breaking ABI change, so it
+        // belongs behind a new function variant or a settings struct.
+        graceful_bail_out_on_content_handler_error: false,
     };
 
     let output_sink = ExternOutputSink::new(output_sink, output_sink_user_data);

fuzz/test_case/src/lib.rs

@@ -176,6 +176,7 @@ fn run_rewriter_iter(data: &[u8], selector: &str, encoding: &'static Encoding) {
             memory_settings: MemorySettings::new(),
             strict: false,
             adjust_charset_on_meta_tag: false,
+            graceful_bail_out_on_content_handler_error: false,
         },
         |_: &[u8]| {},
     );

src/rewriter/mod.rs

@@ -167,6 +167,8 @@ impl<'h, O: OutputSink, H: HandlerTypes> HtmlRewriter<'h, O, H> {
         let graceful_bail_out_on_memory_limit_exceeded = settings
             .memory_settings
             .graceful_bail_out_on_memory_limit_exceeded;
+        let graceful_bail_out_on_content_handler_error =
+            settings.graceful_bail_out_on_content_handler_error;
         let strict = settings.strict;
 
         let encoding = settings.encoding;
@@ -188,6 +190,7 @@ impl<'h, O: OutputSink, H: HandlerTypes> HtmlRewriter<'h, O, H> {
             next_encoding,
             strict,
             graceful_bail_out_on_memory_limit_exceeded,
+            graceful_bail_out_on_content_handler_error,
         });
 
         HtmlRewriter {
@@ -1443,6 +1446,194 @@ mod tests {
             );
         }
 
+        // --- Content-handler-error bail-out tests ---
+        //
+        // These mirror the memory bail-out tests but with handler errors as the trigger. The
+        // contract is the same: when `graceful_bail_out_on_content_handler_error = true`, the
+        // sink receives every input byte the rewriter had been given before the error.
+
+        /// An element handler that returns `Err` aborts the rewriter. With graceful bail-out
+        /// enabled, the sink keeps every byte the caller had fed in, with the failing element
+        /// and everything after it flushed raw (no transformation), and earlier elements
+        /// transformed normally.
+        #[test]
+        fn test_graceful_bail_out_on_element_handler_error() {
+            let html = b"<a>first</a><stop>middle</stop><b>last</b>";
+
+            let mut output = Vec::<u8>::new();
+            let mut rewriter = HtmlRewriter::new(
+                Settings {
+                    element_content_handlers: vec![
+                        element!("a", |el| {
+                            el.set_attribute("rewritten", "yes").unwrap();
+                            Ok(())
+                        }),
+                        element!("stop", |_| Err("handler refused".into())),
+                    ],
+                    graceful_bail_out_on_content_handler_error: true,
+                    ..Settings::new()
+                },
+                |c: &[u8]| output.extend_from_slice(c),
+            );
+
+            let err = rewriter.write(html).unwrap_err();
+
+            assert!(
+                matches!(err, RewritingError::ContentHandlerError(_)),
+                "expected ContentHandlerError, got {err}",
+            );
+
+            // The full original bytes from `<stop>` onwards are present raw, and the `<a>` tag
+            // before that has been transformed.
+            let output_str = std::str::from_utf8(&output).unwrap();
+
+            assert!(
+                output_str.starts_with("<a rewritten=\"yes\">first</a>"),
+                "expected transformed prefix, got {output_str:?}",
+            );
+            assert!(
+                output_str.ends_with("<stop>middle</stop><b>last</b>"),
+                "expected raw bytes from the failing tag onwards, got {output_str:?}",
+            );
+        }
+
+        /// Without the opt-in flag, an element-handler error still aborts processing without
+        /// flushing remaining bytes (existing behavior preserved).
+        #[test]
+        fn test_no_graceful_bail_out_on_content_handler_error_by_default() {
+            let html = b"<a>first</a><stop>middle</stop>";
+
+            let mut output = Vec::<u8>::new();
+            let mut rewriter = HtmlRewriter::new(
+                Settings {
+                    element_content_handlers: vec![element!("stop", |_| Err(
+                        "handler refused".into()
+                    ))],
+                    ..Settings::new()
+                },
+                |c: &[u8]| output.extend_from_slice(c),
+            );
+
+            let err = rewriter.write(html).unwrap_err();
+
+            assert!(matches!(err, RewritingError::ContentHandlerError(_)));
+            assert!(
+                !output.ends_with(b"<stop>middle</stop>"),
+                "without graceful bail-out the sink must NOT contain the failing tag, got {output:?}",
+            );
+        }
+
+        /// A comment handler that returns `Err` is recoverable too. Comments live on the same
+        /// `lexeme_consumed` path as elements, so this exercises the same restructured ordering.
+        #[test]
+        fn test_graceful_bail_out_on_comment_handler_error() {
+            let html = b"<div>Before<!--FAIL-->After</div>";
+
+            let mut output = Vec::<u8>::new();
+            let mut rewriter = HtmlRewriter::new(
+                Settings {
+                    element_content_handlers: vec![comments!("div", |_| {
+                        Err("comment refused".into())
+                    })],
+                    graceful_bail_out_on_content_handler_error: true,
+                    ..Settings::new()
+                },
+                |c: &[u8]| output.extend_from_slice(c),
+            );
+
+            let err = rewriter.write(html).unwrap_err();
+            assert!(matches!(err, RewritingError::ContentHandlerError(_)));
+
+            // The whole document is in the sink (handler error doesn't lose bytes).
+            assert_eq!(output, html);
+        }
+
+        /// A handler error from `handle_end` arrives after `flush_remaining_input` has already
+        /// emitted every input byte, so the sink already has the complete document. Bail-out
+        /// just propagates the error without losing anything.
+        #[test]
+        fn test_graceful_bail_out_on_end_handler_error() {
+            let html = b"<div>content</div>";
+
+            let mut output = Vec::<u8>::new();
+            let mut rewriter = HtmlRewriter::new(
+                Settings {
+                    document_content_handlers: vec![end!(|_| Err("end refused".into()))],
+                    graceful_bail_out_on_content_handler_error: true,
+                    ..Settings::new()
+                },
+                |c: &[u8]| output.extend_from_slice(c),
+            );
+
+            rewriter.write(html).unwrap();
+
+            let err = rewriter.end().unwrap_err();
+
+            assert!(matches!(err, RewritingError::ContentHandlerError(_)));
+            // All input bytes already in sink before `handle_end()` runs.
+            assert_eq!(output, html);
+        }
+
+        /// Reconstruction test: when a handler in the middle of a document errors, the sink
+        /// output plus any unfed bytes must equal the original document.
+        #[test]
+        fn test_bail_out_reconstruct_handler_error_midstream() {
+            let html = b"<p>before</p><div>middle</div><span>after</span>";
+
+            let mut output = Vec::<u8>::new();
+            let mut rewriter = HtmlRewriter::new(
+                Settings {
+                    element_content_handlers: vec![element!("div", |_| {
+                        Err("div refused".into())
+                    })],
+                    graceful_bail_out_on_content_handler_error: true,
+                    ..Settings::new()
+                },
+                |c: &[u8]| output.extend_from_slice(c),
+            );
+
+            let err = rewriter.write(html).unwrap_err();
+            assert!(matches!(err, RewritingError::ContentHandlerError(_)));
+
+            assert_eq!(
+                output, html,
+                "response must be reconstructable byte-for-byte when handler errors midstream",
+            );
+        }
+
+        /// The two bail-out flags are independent: enabling content-handler bail-out does not
+        /// affect memory-limit behavior, and vice versa.
+        #[test]
+        fn test_bail_out_flags_independent() {
+            // Memory limit error with content-handler bail-out only: should NOT bail out.
+            const MAX: usize = 100;
+            let mut output = Vec::<u8>::new();
+            let mut rewriter = HtmlRewriter::new(
+                Settings {
+                    element_content_handlers: vec![element!("*", |_| Ok(()))],
+                    memory_settings: MemorySettings {
+                        max_allowed_memory_usage: MAX,
+                        preallocated_parsing_buffer_size: 0,
+                        graceful_bail_out_on_memory_limit_exceeded: false,
+                    },
+                    graceful_bail_out_on_content_handler_error: true,
+                    ..Settings::new()
+                },
+                |c: &[u8]| output.extend_from_slice(c),
+            );
+
+            let chunk_1 = format!("<img alt=\"{}", "l".repeat(MAX / 2));
+            let chunk_2 = format!("{}\" />", "r".repeat(MAX / 2));
+            rewriter.write(chunk_1.as_bytes()).unwrap();
+            let err = rewriter.write(chunk_2.as_bytes()).unwrap_err();
+
+            assert!(matches!(err, RewritingError::MemoryLimitExceeded(_)));
+            assert!(
+                output.is_empty(),
+                "content-handler flag must not enable memory bail-out, got {output:?}",
+            );
+        }
+
         #[test]
         fn content_handler_error_propagation() {
             fn assert_err<'h>(

src/rewriter/settings.rs

@@ -1034,6 +1034,54 @@ pub struct Settings<'handlers, 'selectors, H: HandlerTypes = LocalHandlerTypes>
     ///
     /// `false` when constructed with `Settings::new()`.
     pub adjust_charset_on_meta_tag: bool,
+
+    /// Controls how the rewriter recovers when a content handler returns an `Err`.
+    ///
+    /// When `false` (the default), the rewriter aborts processing the response, returns the
+    /// handler's [`RewritingError::ContentHandlerError`], and leaves the output sink in a
+    /// potentially inconsistent state. Downstream this typically manifests as a truncated,
+    /// broken response.
+    ///
+    /// When `true`, before propagating [`RewritingError::ContentHandlerError`] the rewriter
+    /// flushes every input byte it has received but not yet emitted to the sink, *as-is*. The
+    /// caller can then continue the response by writing any subsequent input bytes directly
+    /// to its own downstream sink, bypassing the (now poisoned) rewriter. The resulting
+    /// response will have the rewriter's transformations applied up to some boundary,
+    /// followed by the original bytes after that boundary, but the response will not be
+    /// broken.
+    ///
+    /// The rewriter is still poisoned after the error and must not be used again, regardless
+    /// of this setting.
+    ///
+    /// This is symmetric with
+    /// [`MemorySettings::graceful_bail_out_on_memory_limit_exceeded`], but kept as a separate
+    /// flag because the underlying error has different semantics: a memory limit is an
+    /// environmental constraint, whereas a content handler returning `Err` is an explicit
+    /// signal from the application that something is wrong with the input. Some callers will
+    /// want graceful recovery for one but not the other.
+    ///
+    /// ### Caveats
+    ///
+    /// 1. If a handler was actively removing element content (e.g. via [`Element::remove()`])
+    ///    at the time of the bail-out, the surrounding tags can end up mismatched in the
+    ///    resulting response. In practice removing content is uncommon, and a
+    ///    well-formed-but-imperfect response is still much better than a truncated one.
+    /// 2. If a text content handler returns an error after some chunks of the same text node
+    ///    have already been emitted (rare; typically only happens with multi-chunk
+    ///    encoding-converted text), the bail-out flush will re-emit the input bytes raw,
+    ///    duplicating the already-emitted chunks. The response is byte-bigger but not
+    ///    truncated.
+    ///
+    /// ### Default
+    ///
+    /// `false` when constructed with `Settings::new()`.
+    ///
+    /// [`MemorySettings::graceful_bail_out_on_memory_limit_exceeded`]:
+    ///     struct.MemorySettings.html#structfield.graceful_bail_out_on_memory_limit_exceeded
+    /// [`RewritingError::ContentHandlerError`]:
+    ///     errors/enum.RewritingError.html#variant.ContentHandlerError
+    /// [`Element::remove()`]: html_content/struct.Element.html#method.remove
+    pub graceful_bail_out_on_content_handler_error: bool,
 }
 
 impl Default for Settings<'_, '_, LocalHandlerTypes> {
@@ -1074,6 +1122,7 @@ impl<H: HandlerTypes> Settings<'_, '_, H> {
             strict: true,
             enable_esi_tags: false,
             adjust_charset_on_meta_tag: false,
+            graceful_bail_out_on_content_handler_error: false,
         }
     }
 }

src/selectors_vm/tests.rs

@@ -90,6 +90,7 @@ mod vm_tests {
             memory_limiter: SharedMemoryLimiter::new(2048),
             strict: true,
             graceful_bail_out_on_memory_limit_exceeded: false,
+            graceful_bail_out_on_content_handler_error: false,
         });
 
         transform_stream.write(&html).unwrap();

src/transform_stream/dispatcher.rs

@@ -124,7 +124,18 @@ where
         Ok(())
     }
 
-    fn lexeme_consumed<T>(&mut self, lexeme: &Lexeme<'_, T>) {
+    /// Emit the raw chunk of input that lies between the previously emitted byte and the
+    /// start of `lexeme`, and move `remaining_content_start` to the start of `lexeme`.
+    ///
+    /// This is the first half of what used to be `lexeme_consumed()`. It is split out so that
+    /// the second half ([`consume_lexeme()`]) can be deferred until after the lexeme's token
+    /// has been successfully emitted; that way, if token emission fails (e.g. a content
+    /// handler returns an error), `remaining_content_start` still points at the failing
+    /// lexeme's start and a graceful bail-out can flush the lexeme bytes raw rather than
+    /// losing them.
+    ///
+    /// [`consume_lexeme()`]: Self::consume_lexeme
+    fn emit_chunk_before_lexeme<T>(&mut self, lexeme: &Lexeme<'_, T>) {
         let lexeme_range = lexeme.raw_range();
 
         let chunk_range = Range {
@@ -138,7 +149,16 @@ where
             self.output_sink.handle_chunk(&chunk);
         }
 
-        self.remaining_content_start = lexeme_range.end;
+        self.remaining_content_start = lexeme_range.start;
+    }
+
+    /// Advance `remaining_content_start` past the end of `lexeme`, marking it as committed.
+    /// Must only be called after the lexeme's token has been successfully emitted; see
+    /// [`emit_chunk_before_lexeme()`].
+    ///
+    /// [`emit_chunk_before_lexeme()`]: Self::emit_chunk_before_lexeme
+    fn consume_lexeme<T>(&mut self, lexeme: &Lexeme<'_, T>) {
+        self.remaining_content_start = lexeme.raw_range().end;
     }
 
     #[inline]
@@ -234,13 +254,14 @@ where
     {
         match lexeme.to_token(&mut self.delegate.capture_flags, self.encoding.get()) {
             ToTokenResult::Token(token) => {
-                self.delegate.lexeme_consumed(lexeme);
+                self.delegate.emit_chunk_before_lexeme(lexeme);
                 self.delegate.token_produced(token)?;
+                self.delegate.consume_lexeme(lexeme);
                 // handler for <meta charset> may have changed encoding
                 self.flush_encoding_change();
             }
             ToTokenResult::Text(text_type) => {
-                self.delegate.lexeme_consumed(lexeme);
+                self.delegate.emit_chunk_before_lexeme(lexeme);
                 self.last_text_type = text_type;
                 self.text_decoder.feed_text(
                     lexeme.spanned(),
@@ -255,6 +276,7 @@ where
                         )
                     },
                 )?;
+                self.delegate.consume_lexeme(lexeme);
             }
             ToTokenResult::None => {}
         }