Skip to content

Commit 2e32788

Browse files
fix: PythonCodeSplitter secondary-split pieces missing qualified name context (#11875)
Co-authored-by: Stefano Fiorucci <stefanofiorucci@gmail.com>
1 parent a66bb65 commit 2e32788

3 files changed

Lines changed: 17 additions & 0 deletions

File tree

haystack/components/preprocessors/python_code_splitter.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -556,7 +556,9 @@ def _secondary_split(self, unit: _CodeUnit, parent_doc: Document) -> list[Docume
556556
meta["secondary_split"] = True
557557
meta["secondary_split_index"] = idx
558558
meta["secondary_split_total"] = len(intermediate)
559+
meta["qualified_name"] = qualified_name
559560
results.append(Document(content=piece.content or "", meta=meta))
561+
560562
return results
561563

562564
@component.output_types(documents=list[Document])
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
fixes:
3+
- |
4+
Fixed ``PythonCodeSplitter`` losing identifying context for oversized
5+
functions, methods, or classes. When a unit is too large and falls back
6+
to line-based secondary splitting, only the first resulting piece
7+
naturally retains the source ``def``/``class`` line; every piece now
8+
includes a ``qualified_name`` field in ``meta`` identifying the function,
9+
method, or class it came from.

test/components/preprocessors/test_python_code_splitter.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -836,6 +836,12 @@ def longer():
836836
assert chunks_with_short and chunks_with_long
837837
assert chunks_with_short[0] is not chunks_with_long[0]
838838

839+
def test_qualified_name_in_meta_for_all_pieces(self, oversized_function_source):
840+
splitter = PythonCodeSplitter(min_effective_lines=2, max_effective_lines=5, oversized_factor=3)
841+
result = splitter.run(documents=[Document(content=oversized_function_source)])
842+
for piece in result["documents"]:
843+
assert piece.meta.get("qualified_name") == "giant"
844+
839845

840846
class TestEdgeCases:
841847
def test_module_with_only_docstring(self):

0 commit comments

Comments
 (0)