fix: DocumentCleaner wiped the middle page of a 3-page document

[otiscuilei]

Who hits this: anyone running DocumentCleaner with remove_repeated_substrings=True over documents that use form-feed (\f) page separators and have exactly three pages — a common shape for short PDFs. The unique middle page silently disappears from the cleaned output.

Root cause: header/footer detection derives a common ngram only from the pages strictly between the first and the last (pages[n_first_pages_to_ignore:-n_last_pages_to_ignore]). For a 3-page document that slice is a single page. _find_longest_common_ngram was written to guard only the empty case, so with one sequence reduce(set.intersection, ...) returned that page's own ngram set and picked its longest ngram. That ngram was then treated as a repeated header/footer and str.replace'd out of every page, wiping the middle page's content.

Fix: require at least two sequences before deriving a "common" ngram. A single sequence shares no ngram with any other, so there is nothing to remove. Genuine repeated headers/footers (documents with two or more comparable inner pages) are still detected and stripped exactly as before.

Before/after with the public component API:

from haystack import Document
from haystack.components.preprocessors import DocumentCleaner

cleaner = DocumentCleaner(
    remove_empty_lines=False,
    remove_extra_whitespaces=False,
    remove_repeated_substrings=True,
)
text = "PAGE ONE\fThe quick brown fox jumps high\fPAGE THREE"
out = cleaner.run(documents=[Document(content=text)])["documents"][0]
print(out.content)

# Before: "PAGE ONE\f\fPAGE THREE"          # middle page wiped
# After:  "PAGE ONE\fThe quick brown fox jumps high\fPAGE THREE"   # all pages preserved

Tests: adds a regression test that runs the 3-page document through DocumentCleaner and asserts the full content round-trips unchanged — all three pages preserved in order, not merely the middle one. Verified it fails on the pre-fix code and passes with the fix. Includes a release note.

[vercel]

@Otis0408 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/preprocessors
document_cleaner.py
Project Total

This report was generated by python-coverage-comment-action

[bogdankostic]

Thanks for the fix @Otis0408!

[otiscuilei]

Thanks for the review and merge, much appreciated!