fix: gracefully handle invalide html string during chunking (#4243)

This PR fixes an issue where an invalid `text_as_html` input into html
based table chunking logic can lead to chunking failing. Like the
following stack trace shows:

```
    |   File "/app/unstructured/unstructured/chunking/base.py", line 594, in iter_chunks
    |     yield from _TableChunker.iter_chunks(
    |   File "/app/unstructured/unstructured/chunking/base.py", line 837, in _iter_chunks
    |     html_size = measure(self._html) if self._html else 0
    |                                        ^^^^^^^^^^
    |   File "/app/unstructured/unstructured/utils.py", line 154, in __get__
    |     value = self._fget(obj)
    |             ^^^^^^^^^^^^^^^
    |   File "/app/unstructured/unstructured/chunking/base.py", line 866, in _html
    |     if not (html_table := self._html_table):
    |                           ^^^^^^^^^^^^^^^^
    |   File "/app/unstructured/unstructured/utils.py", line 154, in __get__
    |     value = self._fget(obj)
    |             ^^^^^^^^^^^^^^^
    |   File "/app/unstructured/unstructured/chunking/base.py", line 884, in _html_table
    |     return HtmlTable.from_html_text(text_as_html)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/unstructured/unstructured/common/html_table.py", line 61, in from_html_text
    |     root = fragment_fromstring(html_text)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.12/site-packages/lxml/html/__init__.py", line 810, in fragment_fromstring
    |     elements = fragments_fromstring(
    |                ^^^^^^^^^^^^^^^^^^^^^
    |   File "/app/.venv/lib/python3.12/site-packages/lxml/html/__init__.py", line 780, in fragments_fromstring
    |     raise etree.ParserError(
    | lxml.etree.ParserError: There is leading text: '```html\n'
```

The solution is to catch the parser error and return a `None` instead in
`unstructured/chunking/base.py` in `_html_table`. This way we fallback
to text based chunking for this element with a warning log.

Author: badGarnet

.github/workflows/ci.yml

@@ -191,10 +191,10 @@ jobs:
     - uses: ./.github/actions/base-cache
       with:
         python-version: ${{ matrix.python-version }}
-    - name: Setup docker-compose
-      uses: KengoTODA/actions-setup-docker-compose@v1
-      with:
-        version: '2.22.0'
+    - name: Set up Docker
+      uses: docker/setup-docker-action@v4
+    - name: Set up Docker Compose
+      uses: docker/setup-compose-action@v1
     - name: Test (end-to-end)
       env:
         AIRTABLE_PERSONAL_ACCESS_TOKEN: ${{ secrets.AIRTABLE_PERSONAL_ACCESS_TOKEN }}

.github/workflows/ingest-test-fixtures-update-pr.yml

@@ -29,10 +29,8 @@ jobs:
       - uses: ./.github/actions/base-cache
         with:
           python-version: ${{ env.PYTHON_VERSION }}
-      - name: Setup docker-compose
-        uses: KengoTODA/actions-setup-docker-compose@v1
-        with:
-          version: "2.22.0"
+      - name: Set up Docker Compose
+        uses: docker/setup-compose-action@v1
       - name: Update test fixtures
         env:
           AIRTABLE_PERSONAL_ACCESS_TOKEN: ${{ secrets.AIRTABLE_PERSONAL_ACCESS_TOKEN }}

CHANGELOG.md

@@ -1,3 +1,8 @@
+## 0.20.5
+
+### Fixes
+- **Gracefully handle invalid `text_as_html` during chunking**: `_TableChunker` now catches parse errors (e.g. `lxml.etree.ParserError` when `text_as_html` contains a markdown code-fence like `` ```html\n ``) and returns `None` instead of raising, allowing chunking to continue using plain-text fallback. A `WARNING` log is emitted with a truncated preview of the offending value.
+
 ## 0.20.4
 
 ### Enhancements

scripts/kafka-test-helpers/create-kafka-instance.sh

@@ -5,8 +5,8 @@ set -e
 SCRIPT_DIR=$(dirname "$(realpath "$0")")
 
 # Create the Weaviate instance
-docker-compose version
-docker-compose -f "$SCRIPT_DIR"/docker-compose.yml up --wait
-docker-compose -f "$SCRIPT_DIR"/docker-compose.yml ps
+docker compose version
+docker compose -f "$SCRIPT_DIR"/docker-compose.yml up --wait
+docker compose -f "$SCRIPT_DIR"/docker-compose.yml ps
 
 echo "Instance is live."

test_unstructured/chunking/test_base.py

@@ -4,6 +4,7 @@
 
 from __future__ import annotations
 
+import logging
 from typing import Any, Sequence
 
 import pytest
@@ -1322,6 +1323,56 @@ def it_handles_table_with_none_text_without_error(self):
 
         assert len(chunks) == 0
 
+    def it_handles_invalid_html_in_text_as_html_without_error(self, caplog):
+        """Regression test: gracefully skip HTML-based chunking when text_as_html is not valid HTML.
+
+        `lxml` raises `ParserError` for strings that are not valid HTML fragments (e.g. plain text
+        with no tags). The chunker should log a warning and fall back to text-only chunking rather
+        than raising.
+        """
+        table = Table(
+            "Header Col 1  Header Col 2\nLorem ipsum   dolor sit amet",
+            metadata=ElementMetadata(text_as_html="not valid html"),
+        )
+
+        caplog.set_level(logging.WARNING)
+        # -- should not raise ParserError --
+        chunks = list(_TableChunker.iter_chunks(table, "", ChunkingOptions()))
+
+        # -- falls back to text-only: a single Table chunk with no .text_as_html --
+        assert len(chunks) == 1
+        chunk = chunks[0]
+        assert isinstance(chunk, Table)
+        assert chunk.metadata.text_as_html is None
+        assert len(caplog.records) == 1
+        assert caplog.records[0].message.startswith("Could not parse text_as_html")
+        assert caplog.records[0].message.endswith("not valid html")
+
+    def it_handles_html_without_table_element_in_text_as_html_without_error(self, caplog):
+        """Regression test: gracefully skip HTML-based chunking when text_as_html has no <table>.
+
+        `HtmlTable.from_html_text` raises `ValueError` when the HTML is valid but contains no
+        `<table>` element. The chunker should log a warning and fall back to text-only chunking
+        rather than raising.
+        """
+        table = Table(
+            "Header Col 1  Header Col 2\nLorem ipsum   dolor sit amet",
+            metadata=ElementMetadata(text_as_html="<div>no table here</div>"),
+        )
+
+        caplog.set_level(logging.WARNING)
+        # -- should not raise ValueError --
+        chunks = list(_TableChunker.iter_chunks(table, "", ChunkingOptions()))
+
+        # -- falls back to text-only: a single Table chunk with no .text_as_html --
+        assert len(chunks) == 1
+        chunk = chunks[0]
+        assert isinstance(chunk, Table)
+        assert chunk.metadata.text_as_html is None
+        assert len(caplog.records) == 1
+        assert caplog.records[0].message.startswith("Could not parse text_as_html")
+        assert "<div>no table here</div>" in caplog.records[0].message
+
 
 # ================================================================================================
 # HTML SPLITTERS
@@ -1333,7 +1384,8 @@ class Describe_HtmlTableSplitter:
 
     def it_splits_an_HTML_table_on_whole_row_boundaries_when_possible(self):
         opts = ChunkingOptions(max_characters=(40))
-        html_table = HtmlTable.from_html_text("""
+        html_table = HtmlTable.from_html_text(
+            """
             <table border="1" class="dataframe">
               <tbody>
                 <tr>
@@ -1364,7 +1416,8 @@ def it_splits_an_HTML_table_on_whole_row_boundaries_when_possible(self):
                 </tr>
               </tbody>
             </table>
-            """)
+            """
+        )
 
         assert list(_HtmlTableSplitter.iter_subtables(html_table, opts)) == [
             (
@@ -1389,7 +1442,8 @@ def it_splits_an_HTML_table_on_whole_row_boundaries_when_possible(self):
 
     def and_it_splits_an_oversized_row_on_an_even_cell_boundary_when_possible(self):
         opts = ChunkingOptions(max_characters=(93))
-        html_table = HtmlTable.from_html_text("""
+        html_table = HtmlTable.from_html_text(
+            """
             <html><body><table>
               <tr>
                 <td>Lorem ipsum dolor sit amet.</td>
@@ -1408,7 +1462,8 @@ def and_it_splits_an_oversized_row_on_an_even_cell_boundary_when_possible(self):
                 <td>Cillum</td>
               </tr>
             </table></body></html>
-            """)
+            """
+        )
 
         assert list(_HtmlTableSplitter.iter_subtables(html_table, opts)) == [
             (
@@ -1433,7 +1488,8 @@ def and_it_splits_an_oversized_row_on_an_even_cell_boundary_when_possible(self):
 
     def and_it_splits_an_oversized_cell_on_an_even_word_boundary(self):
         opts = ChunkingOptions(max_characters=(100))
-        html_table = HtmlTable.from_html_text("""
+        html_table = HtmlTable.from_html_text(
+            """
             <table>
               <thead>
                 <tr>
@@ -1452,7 +1508,8 @@ def and_it_splits_an_oversized_cell_on_an_even_word_boundary(self):
                 <tr><td>In reprehenderit voluptate.</td></tr>
               </tbody>
             </table
-            """)
+            """
+        )
 
         assert list(_HtmlTableSplitter.iter_subtables(html_table, opts)) == [
             (

test_unstructured_ingest/src/elasticsearch.sh

@@ -22,7 +22,7 @@ source scripts/elasticsearch-test-helpers/common/es-dest-ingest-test-creds.env
 function cleanup() {
   # Kill the container so the script can be repeatedly run using the same ports
   echo "Stopping Elasticsearch Docker container"
-  docker-compose -f scripts/elasticsearch-test-helpers/common/docker-compose.yaml down --remove-orphans -v
+  docker compose -f scripts/elasticsearch-test-helpers/common/docker-compose.yaml down --remove-orphans -v
 
   cleanup_dir "$OUTPUT_DIR"
   cleanup_dir "$WORK_DIR"

test_unstructured_ingest/src/kafka-local.sh

@@ -33,7 +33,7 @@ function cleanup() {
   fi
 
   echo "Stopping local Kafka instance"
-  docker-compose -f scripts/kafka-test-helpers/docker-compose.yml down --remove-orphans -v
+  docker compose -f scripts/kafka-test-helpers/docker-compose.yml down --remove-orphans -v
 
 }
 trap cleanup EXIT

test_unstructured_ingest/src/opensearch.sh

@@ -20,7 +20,7 @@ source "$SCRIPT_DIR"/cleanup.sh
 function cleanup() {
   # Kill the container so the script can be repeatedly run using the same ports
   echo "Stopping OpenSearch Docker container"
-  docker-compose -f scripts/opensearch-test-helpers/common/docker-compose.yaml down --remove-orphans -v
+  docker compose -f scripts/opensearch-test-helpers/common/docker-compose.yaml down --remove-orphans -v
 
   cleanup_dir "$OUTPUT_DIR"
   cleanup_dir "$WORK_DIR"

test_unstructured_ingest/src/s3-minio.sh

@@ -22,7 +22,7 @@ source "$SCRIPT_DIR"/cleanup.sh
 function cleanup() {
   # Kill the container so the script can be repeatedly run using the same ports
   echo "Stopping Minio Docker container"
-  docker-compose -f scripts/minio-test-helpers/docker-compose.yaml down --remove-orphans -v
+  docker compose -f scripts/minio-test-helpers/docker-compose.yaml down --remove-orphans -v
 
   cleanup_dir "$OUTPUT_DIR"
   cleanup_dir "$WORK_DIR"

test_unstructured_ingest/src/sftp.sh

@@ -19,7 +19,7 @@ source "$SCRIPT_DIR"/cleanup.sh
 function cleanup() {
   # Kill the container so the script can be repeatedly run using the same ports
   echo "Stopping Sftp Docker container"
-  docker-compose -f scripts/sftp-test-helpers/docker-compose.yaml down --remove-orphans -v
+  docker compose -f scripts/sftp-test-helpers/docker-compose.yaml down --remove-orphans -v
 
   cleanup_dir "$OUTPUT_DIR"
   cleanup_dir "$WORK_DIR"