fix: MarkdownHeaderSplitter silently drops a trailing header with no body text - #12064
Conversation
With keep_headers=True (the default), a header whose only content is whitespace is buffered into pending_headers to be prepended to the next content chunk. A header (or run of headers) at the very end of the document has no following chunk, so the buffer was never flushed and the header text was dropped -- breaking the reconstruction invariant that the split contents rejoin to the original, and inconsistent with a middle empty header, which is preserved. Flush any trailing buffered headers as a final chunk, sliced from the original text so reconstruction stays exact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N6RtoHuxrDqTUo9Mw9h4Cv
|
@lntutor is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
|
|
|
Hi @lntutor, thanks a lot for your contribution! 🙏 We noticed that the Contributor License Agreement (CLA) check ( To get your PR reviewed, please sign the CLA via the link in the |
HaystackBot
added
the
cla-pending
|
Hi @lntutor, just a friendly reminder: this PR is still in draft because the Contributor License Agreement (CLA) hasn't been signed yet. We'd love to review your contribution! Please sign the CLA via the link in the |
HaystackBot
removed
the
cla-pending
|
Thanks for signing the CLA, @lntutor! 🎉 This PR is now ready for review again and the reviewer has been re-assigned. |
…eader metadata
Follow-up to the trailing-header fix in this PR, addressing two coupled review
findings.
The trailing-header flush already sliced the original text, but the pre-existing
path for a *middle* header with no body still rebuilt content by joining buffered
header lines with "\n". That normalized whitespace, so blank lines between such a
header and the next contentful one were collapsed and reconstruction stayed
lossy:
"# H1\n\n\n# H2\nContent.\n" -> "# H1\n# H2\nContent.\n"
Both paths now slice from the original text, so the two agree on fidelity and
keep_headers=True reconstruction is byte-exact in every buffered-header case.
Because content is no longer rebuilt from header_prefix/header_text, the
pending_headers list is only ever read as a boolean -- "pending_start is not
None" already carries that signal -- so it and both header_line constructions
are removed. Net less state in the loop than before.
That in turn makes it safe to strip the captured header text, which the regex
^(#{1,6}) (.+)$ had always let leak into metadata ("# Header 1 " gave
header == "Header 1 ", and a trailing "# " gave header == " "). Stripping
now only affects header / parent_headers; chunk content keeps the header line's
original whitespace. Done before this change it would have silently mutated
emitted content.
Adds three regression tests, each failing before this commit. Also applies the
release-note double-backtick convention that pre-commit enforces.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
julian-risch
left a comment
julian-risch
left a comment
There was a problem hiding this comment.
@lntutor Thanks for opening this PR! I made some small changes and the PR is ready to be merged now:
- The middle-empty-header path still rebuilt content by joining header lines with
"\n", which collapsed blank lines. Both paths slice the original text now - Stripped whitespaces from the captured header text
- Added regression tests and adjusted release notes
Coverage report
Click to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||
4 participants
What's broken
With
keep_headers=True(the default),MarkdownHeaderSplitterbuffers a header whose only content is whitespace intopending_headersand prepends it to the next content chunk. But a header (or run of headers) at the end of the document has no next chunk, so the buffer is never flushed and the header text is dropped:This breaks the reconstruction invariant the existing tests assert (
test_basic_split,test_keep_headers_preserves_parent_headers_for_first_childboth check"".join(doc.content) == original), and it's inconsistent: a middle empty header is preserved (prepended to the next chunk), a trailing one is dropped.Fix
Track the start offset / text / level of buffered trailing headers and, before returning, flush them as a final chunk sliced from the original text (so byte-level reconstruction and downstream page counting stay exact) — mirroring how a middle empty header is already preserved.
Distinct from #11920 (content dropped before an embedded header/code-fence).
Tests
Adds
test_trailing_header_without_content_is_not_dropped. Fulltest/components/preprocessors/test_markdown_header_splitter.pypasses (35). Release note included.