fix: handle deeply nested HTML that triggers RecursionError

[jigangz]

Summary

Fix large HTML files (>3MB) with deep DOM nesting silently returning unconverted HTML instead of markdown.

Problem

When converting deeply nested HTML documents (e.g., SEC EDGAR filings like Tesla's DEF 14A proxy statement), markdownify's recursive DOM traversal exceeds Python's default recursion limit (~400 nesting levels). The RecursionError is caught by the top-level _convert() dispatcher's except Exception block, and the request falls through to PlainTextConverter which returns the raw HTML as-is — with no error or warning.

Root cause chain:

1. HtmlConverter.convert() → markdownify.convert_soup() (recursive traversal)
2. Deep nesting (>~400 levels) → RecursionError
3. _convert() catches it via except Exception, stores in failed_attempts
4. PlainTextConverter.accepts() matches text/html via text/ prefix → true
5. PlainTextConverter.convert() returns raw HTML bytes as text
6. Caller gets "markdown" that is actually unconverted HTML

Fix

Catch RecursionError in HtmlConverter.convert() and fall back to BeautifulSoup's iterative get_text() method, which handles arbitrary nesting depths. A UserWarning is emitted so callers know the output is plain text rather than full markdown.

Changes

• packages/markitdown/src/markitdown/converters/_html_converter.py: catch RecursionError, fall back to get_text(), emit warning
• packages/markitdown/tests/test_module_misc.py: add test_deeply_nested_html_fallback verifying the fallback behavior and warning

Fixes #1636

[jigangz]

@microsoft-github-policy-service agree

[VANDRANKI]

Good fix for a real production issue. SEC EDGAR filings and similar regulatory documents are exactly the kind of deeply nested HTML that hits this in practice, and silently returning raw HTML instead of markdown is a worse outcome than a graceful fallback with a warning. The approach is correct.

Suggestion: move import warnings to the top of the file

import warnings is currently inside the except RecursionError block. Imports inside exception handlers work correctly in Python but are unconventional: the import happens at exception time rather than at module load time, and static analysis tools may flag it. Moving it to the top of the file with the other imports costs nothing and keeps the import layout consistent with the rest of the codebase.

Question: test recursion depth is environment-dependent

The test constructs HTML with 500 levels of nesting and comments that this is "deep enough to trigger RecursionError". Whether 500 levels actually triggers the error depends on:

• Python's current sys.getrecursionlimit() (default 1000, but varies by environment)
• The number of stack frames markdownify consumes per DOM level (typically more than 1)

In some environments (PyPy, or Python builds with a raised recursion limit), 500 levels may not trigger the error at all, causing the test to pass the fallback assertion vacuously. A more reliable approach is to temporarily lower the recursion limit in the test itself:

import sys
old_limit = sys.getrecursionlimit()
try:
    sys.setrecursionlimit(100)
    result = markitdown.convert_stream(...)
finally:
    sys.setrecursionlimit(old_limit)

This makes the test deterministic regardless of the environment's default limit.

Suggestion: no way to opt out of the fallback

The fallback to get_text() is always applied when a RecursionError occurs. For callers who need markdown specifically (not plain text) and would prefer an exception over degraded output, there is currently no way to opt out. A strict=False parameter on HtmlConverter that, when set to True, re-raises the RecursionError instead of falling back would give callers the choice. Not blocking for this PR, but worth a follow-up issue.

---

The core fix is correct and the test covers the main case well. The import placement is a quick cleanup and the recursion depth concern is worth resolving to make the test reliable across environments.

[jigangz]

Thanks for the thorough review! I've addressed all three points in the latest commit:

1. import warnings moved to top of file — removed the inline import from inside the except block and added it at the module level.

2. Environment-independent test — the test now temporarily lowers sys.setrecursionlimit(200) (restoring it in a finally block) so the RecursionError is guaranteed regardless of the host's default limit. This makes it reliable on all platforms and CI environments.

3. strict parameter added — convert() now accepts a strict=True keyword argument. When set, the RecursionError is re-raised instead of falling back to plain-text extraction, giving callers explicit opt-out control.

[VANDRANKI]

All three points from the original review are addressed. The implementation looks correct:

• import warnings moved to module level - clean
• sys.setrecursionlimit(200) + finally restore - exactly what was needed to make the test deterministic across CI environments
• strict=True parameter via kwargs.pop before forwarding to _CustomMarkdownify - the right pattern; no risk of markdownify receiving an unexpected keyword
• warnings.warn(..., stacklevel=2) - correct stacklevel so the warning points at the caller, not the converter internals

One minor nit: the test assertions use result.text_content which is the soft-deprecated alias for result.markdown. New code in the same file uses result.markdown directly. Not a blocker, but worth updating for consistency:

# Current:
assert "Deep content" in result.text_content

# Preferred:
assert "Deep content" in result.markdown

LGTM otherwise. The fallback is safe, the strict opt-out is well-designed, and the test is solid.

[VANDRANKI]

Perfect - result.markdown is now used consistently throughout the test. Everything is clean. Fully LGTM.

[jigangz]

Hi @afourney, sorry for the ping — this PR has been sitting for a couple weeks. CI workflows are pending maintainer approval (action_required). The fix addresses a silent-failure bug (#1636) where large, deeply-nested HTML documents like SEC EDGAR filings return raw HTML instead of markdown. Would you mind triggering the workflow run when you have a moment? Thanks!