fix: raise a clear error for empty inputs in retrieval and answer evaluators - #11958
fix: raise a clear error for empty inputs in retrieval and answer evaluators#11958otiscuilei wants to merge 4 commits into
Conversation
AnswerExactMatchEvaluator, DocumentMAPEvaluator, DocumentMRREvaluator and DocumentRecallEvaluator averaged their per-query scores by dividing by the input length without guarding against an empty batch. Passing two empty lists (which satisfies the equal-length check) therefore raised an opaque ZeroDivisionError. They now raise a descriptive ValueError, matching the existing contract of DocumentNDCGEvaluator.
|
@Otis0408 is attempting to deploy a commit to the deepset Team on Vercel. A member of the Team first needs to authorize it. |
Coverage reportClick to see where and how coverage changed
This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||
bogdankostic
left a comment
There was a problem hiding this comment.
Looks already good in principle, let's just place the tests in the existing evaluator test files.
There was a problem hiding this comment.
Let's not add this test file containing only a single test function. Instead, let's add this test to the existing evaluator test files.
|
Done — dropped the standalone test file and moved the empty-input assertions into the existing evaluator test files ( |
Summary
AnswerExactMatchEvaluator,DocumentMAPEvaluator,DocumentMRREvaluatorandDocumentRecallEvaluatoraverage their per-query scores by dividing by the number of input queries, without guarding against an empty batch. Two empty lists satisfy the equal-length check these evaluators perform, so the code proceeds tosum(...) / len(...)and crashes with an opaqueZeroDivisionError: division by zero.DocumentNDCGEvaluatoralready handles this case: itsvalidate_inputsrejects empty input with a descriptiveValueError("ground_truth_documents and retrieved_documents must be provided."). This PR makes the other four evaluators consistent with that contract, so an empty batch surfaces a clear, actionable error instead of a bare arithmetic failure.Before
After
The same change applies to
AnswerExactMatchEvaluator(message:ground_truth_answers and predicted_answers must be provided.),DocumentMAPEvaluatorandDocumentMRREvaluator.Tests
Added
test/components/evaluators/test_evaluators_empty_input.py, a parametrized regression test asserting all four evaluators raiseValueErroron empty input. It fails on the pre-fix source (ZeroDivisionError) and passes with the fix. All existing evaluator test suites continue to pass.