fix: populate _split_overlap metadata for word/token units in RecursiveDocumentSplitter by Osamaali313 · Pull Request #11825 · deepset-ai/haystack

Osamaali313 · 2026-06-29T19:09:51Z

Related Issues

None — found while reviewing RecursiveDocumentSplitter overlap handling (a follow-up to the split_idx_start family, #11710/#11711, which fixed a different site).

Proposed Changes

_add_overlap_info computed the overlap length with self._chunk_length(prev_doc.content), which returns a word/token count for split_unit="word"/"token", while curr_pos and split_idx_start are character offsets. Mixing units made overlap_length negative, so the if overlap_length > 0 guard never fired and the _split_overlap metadata was silently left empty for word/token splitting with overlap (char units were unaffected, since there len == _chunk_length).

Fix: measure the overlap and ranges in characters via len(prev_doc.content).

# before (word unit, split_overlap=1): every chunk's _split_overlap == []
# after: overlap provenance is populated for all overlapping chunks

How did you test it?

Added test_run_split_by_dot_and_overlap_1_word_unit_split_overlap_metadata, which fails on main (empty metadata) and passes after the fix. Full test_recursive_splitter.py: 49 passed. Release note added under releasenotes/notes/.

Checklist

I have read the contributors guidelines and the code of conduct
I have updated the related issue with new insights and changes
I added unit tests and updated the docstrings
I've used one of the conventional commit types for my PR title
I documented my code
I ran pre-commit hooks and fixed any issue

Disclaimer: this PR was prepared with AI assistance (Claude). I reviewed the change and verified it RED→GREEN against the real component before submitting.

…veDocumentSplitter _add_overlap_info computed the overlap length with self._chunk_length(prev_doc.content), which returns a word/token count for word/token split units, while curr_pos and split_idx_start are character offsets. Mixing units made overlap_length negative, so the guard never fired and the _split_overlap metadata was silently left empty for word/token splitting with overlap (char units were unaffected). Measure the overlap and ranges in characters via len(prev_doc.content). Adds a regression test and a release note.

vercel · 2026-06-29T19:09:57Z

@Osamaali313 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

CLAassistant · 2026-06-29T19:10:09Z

All committers have signed the CLA.

sjrl · 2026-06-30T07:50:27Z

Hey @Osamaali313 thanks for opening the PR! Please address the failing CI issues.

sjrl · 2026-07-06T06:21:10Z

Hey @Osamaali313 are you able to work on this, this week?

- Shorten the _add_overlap_info comment (per review). - Rename test to test_word_unit_split_populates_split_overlap_metadata and drop the docstring paragraph referencing the old behavior. - Use double backticks for inline code in the release note (reST) so the release-notes RST check passes. - Run ruff format (collapse the _split_overlap dict literal) to fix the format check.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Osamaali313 · 2026-07-06T06:56:37Z

Thanks @sjrl — yes, done. Pushed a commit addressing all the review points and the CI:

Comment wording (recursive_splitter.py): shortened to your suggested two-liner.
Test name: renamed to test_word_unit_split_populates_split_overlap_metadata.
Test docstring: removed the paragraph referring to the old behavior; it now just states the expected behavior.
Release note: switched inline code to double backticks (reST), which clears the release-notes RST formatting check.
format check: ran ruff format (it collapsed the _split_overlap dict literal).

Locally: ruff check + ruff format --check are clean, reno lint . passes, and the full test_recursive_splitter.py suite passes (49 passed). The only remaining red check is the Vercel preview deploy, which needs a maintainer to authorize the external-fork deploy. Let me know if you'd like anything else.

github-actions · 2026-07-06T07:47:43Z

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/preprocessors
recursive_splitter.py
Project Total

_{This report was generated by python-coverage-comment-action}

Per review: replace the word-unit test's loose assertions with direct asserts on the exact chunk contents and _split_overlap ranges, and add a matching token-unit test. Values verified against the splitter output.

Osamaali313 · 2026-07-06T08:39:48Z

Done — thanks for the concrete suggestions. Both applied:

test_word_unit_split_populates_split_overlap_metadata now asserts the exact chunk contents and _split_overlap ranges for all 5 chunks.
Added test_token_unit_split_populates_split_overlap_metadata with the exact ranges for chunks 0, 1, and 7.

I ran the splitter against the actual output to confirm the expected values before committing (all of your suggested contents/ranges match). Both tests pass, the full test_recursive_splitter.py suite is green (50 passed), and ruff check + ruff format --check are clean.

sjrl

Thanks!

Osamaali313 requested a review from a team as a code owner June 29, 2026 19:09

Osamaali313 requested review from Copilot and sjrl and removed request for a team and Copilot June 29, 2026 19:09

github-actions Bot added the topic:tests label Jun 29, 2026

sjrl self-assigned this Jun 30, 2026

sjrl reviewed Jun 30, 2026

View reviewed changes

Comment thread releasenotes/notes/fix-recursive-splitter-word-token-overlap-metadata-9f3c1a7be0d24856.yaml

sjrl reviewed Jun 30, 2026

View reviewed changes

Comment thread test/components/preprocessors/test_recursive_splitter.py Outdated

sjrl reviewed Jun 30, 2026

View reviewed changes

Comment thread test/components/preprocessors/test_recursive_splitter.py Outdated

sjrl reviewed Jun 30, 2026

View reviewed changes

Comment thread haystack/components/preprocessors/recursive_splitter.py Outdated

Copilot AI review requested due to automatic review settings July 6, 2026 06:55

Copilot AI reviewed Jul 6, 2026

github-actions Bot added the type:documentation Improvements on the docs label Jul 6, 2026

sjrl reviewed Jul 6, 2026

View reviewed changes

Comment thread test/components/preprocessors/test_recursive_splitter.py

test: assert exact _split_overlap ranges + add token-unit test

89bd1c8

Per review: replace the word-unit test's loose assertions with direct asserts on the exact chunk contents and _split_overlap ranges, and add a matching token-unit test. Values verified against the splitter output.

sjrl reviewed Jul 6, 2026

View reviewed changes

Comment thread test/components/preprocessors/test_recursive_splitter.py

Update test/components/preprocessors/test_recursive_splitter.py

ec66eb1

sjrl approved these changes Jul 6, 2026

View reviewed changes

sjrl merged commit 1216c61 into deepset-ai:main Jul 6, 2026
23 of 24 checks passed

Uh oh!

Conversation

Osamaali313 commented Jun 29, 2026

Related Issues

Proposed Changes

How did you test it?

Checklist

Uh oh!

vercel Bot commented Jun 29, 2026

Uh oh!

CLAassistant commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sjrl commented Jun 30, 2026

Uh oh!

Uh oh!

Uh oh!

sjrl commented Jul 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Osamaali313 commented Jul 6, 2026

Uh oh!

github-actions Bot commented Jul 6, 2026

Coverage report

Uh oh!

Uh oh!

Osamaali313 commented Jul 6, 2026

Uh oh!

Uh oh!

sjrl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CLAassistant commented Jun 29, 2026 •

edited

Loading