fix: MarkdownHeaderSplitter drops content before an embedded header or code-fence comment

[otiscuilei]

What

MarkdownHeaderSplitter silently drops document content when keep_headers=False and a secondary split is configured.

In _apply_secondary_splitting, the header-stripping step used a non-anchored re.search(self._header_pattern, doc.content). Because the header pattern is (?m)^(#{1,6}) (.+)$ (MULTILINE), re.search matches the first # line anywhere in the chunk, not just a genuine leading header. Everything before that match (doc.content[header_match.end():]) is then discarded.

This triggers whenever a chunk contains a # line that is not its leading header, for example:

• an embedded lower-level header that was merged into the chunk (e.g. a ## when header_split_levels=[1]), or
• a # comment inside a fenced code block (e.g. a Python comment).

Anyone using MarkdownHeaderSplitter(keep_headers=False, secondary_split=...) on realistic Markdown (nested sections or code samples) loses the prose that precedes the first such # line.

Fix

Anchor the match to the start of the chunk with re.match instead of re.search, so only a header at the very beginning of the content is stripped. re.match anchors to position 0 regardless of the MULTILINE flag, leaving embedded headers and code-fence comments untouched.

-header_match = re.search(self._header_pattern, doc.content)
+header_match = re.match(self._header_pattern, doc.content)

Before / after

from haystack import Document
from haystack.components.preprocessors.markdown_header_splitter import MarkdownHeaderSplitter

splitter = MarkdownHeaderSplitter(
    keep_headers=False, header_split_levels=[1], secondary_split="word", split_length=100
)
content = "# Main\nsome intro text\n```python\n# a comment in code\n```\nmore text\n"
docs = splitter.run(documents=[Document(content=content)])["documents"]
combined = "".join(d.content or "" for d in docs)

# Before: "\n```\nmore text\n"   -> "some intro text" and the code fence are lost
# After:  "\nsome intro text\n```python\n# a comment in code\n```\nmore text\n"

Tests

Two regression tests in test/components/preprocessors/test_markdown_header_splitter.py, one per trigger case:

• test_secondary_split_keeps_content_before_embedded_header
• test_secondary_split_keeps_content_before_code_fence_comment

Both fail on the pre-fix source (content dropped) and pass with the fix; the full file (34 tests) passes. A release note is included under releasenotes/notes/.

[vercel]

@Otis0408 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/preprocessors
markdown_header_splitter.py					228
Project Total

This report was generated by python-coverage-comment-action

[sjrl]

Thanks for the changes!

[otiscuilei]

Thanks for the quick review and merge!