Skip to content

fix: PythonCodeSplitter secondary-split pieces missing qualified name context - #11875

Merged
anakin87 merged 6 commits into
deepset-ai:mainfrom
GovindhKishore:fix/python-code-splitter-secondary-split-context
Jul 27, 2026
Merged

fix: PythonCodeSplitter secondary-split pieces missing qualified name context#11875
anakin87 merged 6 commits into
deepset-ai:mainfrom
GovindhKishore:fix/python-code-splitter-secondary-split-context

Conversation

@GovindhKishore

Copy link
Copy Markdown
Contributor

Related Issues

Proposed Changes:

PythonCodeSplitter._secondary_split() falls back to a plain line-based split for oversized functions/methods/classes. Only the first resulting piece naturally contains the def/class signature line, so every piece after that has no identifying text at all, which hurts retrieval ranking for those pieces.

Fixed by reusing the qualified_name value already computed in _secondary_split() (currently only used in the logger.warning(...) call) and prepending it as a # qualified_name comment line to every piece after the first.

How did you test it?

Manually verified locally with a reproduction script: created a Document with a single function exceeding the oversized threshold, ran it through PythonCodeSplitter().run(...), and confirmed:

  • Piece 0 still starts with the original def function_name(...): line (unchanged).
  • Pieces 1+ now start with # function_name before the body content (previously had no identifying text at all).

Notes for the reviewer

Open to feedback on the prepended format (comment style, placement) before I finalize with tests and a release note.

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/components/preprocessors
  python_code_splitter.py
Project Total  

This report was generated by python-coverage-comment-action

@GovindhKishore
GovindhKishore force-pushed the fix/python-code-splitter-secondary-split-context branch from dc9ac34 to ea9f6cb Compare July 6, 2026 18:40
@vercel

vercel Bot commented Jul 6, 2026

Copy link
Copy Markdown

@GovindhKishore is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

@GovindhKishore
GovindhKishore marked this pull request as ready for review July 6, 2026 18:51
@GovindhKishore
GovindhKishore requested a review from a team as a code owner July 6, 2026 18:52
@GovindhKishore
GovindhKishore requested review from anakin87 and removed request for a team July 6, 2026 18:52

@anakin87 anakin87 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #11874 (comment) and let's discuss there

@anakin87

Copy link
Copy Markdown
Member

@GovindhKishore also based on our discussions, I'd recommend to just write the qualified name in a meta field.

@GovindhKishore
GovindhKishore force-pushed the fix/python-code-splitter-secondary-split-context branch from ea9f6cb to 4cfd4f7 Compare July 25, 2026 10:28
@GovindhKishore

Copy link
Copy Markdown
Contributor Author

@GovindhKishore also based on our discussions, I'd recommend to just write the qualified name in a meta field.

Hi @anakin87, Done. Kindly let me know if any further changes are needed.

@anakin87 anakin87 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@anakin87
anakin87 enabled auto-merge (squash) July 27, 2026 08:18
@anakin87
anakin87 merged commit 2e32788 into deepset-ai:main Jul 27, 2026
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: PythonCodeSplitter secondary-split pieces lose function/method identity, hurting retrieval ranking

2 participants