feat: expose evaluator row status for LLM evaluators

[SeCuReDmE-main-dev]

Summary

This is Phase 1 for RFC #11332.

It adds a narrow top-level evaluation_statuses output to LLMEvaluator results so callers can distinguish rows that were successfully evaluated from rows that failed during generation or parsing when raise_on_failure=False is used.

Scope

Included:

• LLMEvaluator now returns evaluation_statuses alongside existing results and meta.
• Successful parsed rows are marked as evaluated.
• generation/parsing failures that continue under raise_on_failure=False are marked as error.
• ContextRelevanceEvaluator and FaithfulnessEvaluator pass through the new status list while preserving their current nan behavior.
• focused tests for successful rows, invalid JSON, and generation failures.
• release note added under releasenotes/notes.

Intentionally excluded:

• no EvaluationRunResult reporting changes;
• no retriever, agent, router, HITL, or governance changes;
• no indeterminate status yet;
• no public neutrosophic naming in code.

Phase 2 and Phase 3 remain dependent on maintainer feedback from the RFC and this PR.

Tests

C:\Users\jeans\.local\bin\uv.exe run pytest test/components/evaluators/test_llm_evaluator.py test/components/evaluators/test_context_relevance_evaluator.py test/components/evaluators/test_faithfulness_evaluator.py -q
C:\Users\jeans\.local\bin\uv.exe run ruff check haystack/components/evaluators test/components/evaluators
C:\Users\jeans\.local\bin\uv.exe run ruff format --check haystack/components/evaluators test/components/evaluators
C:\Users\jeans\.local\bin\uv.exe run reno lint .

Results:

• 38 passed, 2 skipped
• ruff check: passed
• ruff format --check: passed
• reno lint: passed

[vercel]

@SeCuReDmE-main-dev is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

All committers have signed the CLA.

[SeCuReDmE-main-dev]

Phase 2 base work is complete and stacked on top of this Phase 1 branch.

Downstream stacked PR: SeCuReDmE-main-dev#1
Outcome: BM25 now has an opt-in retrieval confidence metadata path via Document.meta.
Boundary: this stays BM25-only and does not redefine global retriever score semantics.

[SeCuReDmE-main-dev]

Phase 3 base work is complete and the stacked progression now reaches explicit HITL decision contracts.

Downstream stacked PR: SeCuReDmE-main-dev#2
Outcome: ToolExecutionDecision now carries explicit approved / modified / rejected status semantics.
Boundary: this stops at HITL contract enrichment and does not introduce Phase 4 runtime/governance changes.

[SeCuReDmE-main-dev]

Pause boundary — awaiting maintainer feedback

All three stacked phases are now validated and documented:

• Phase 1 (this PR): evaluation_statuses for LLMEvaluator — 16 tests passing
• Phase 2 (fork PR SeCuReDmE-main-dev/haystack_case_study#1): opt-in BM25 retrieval confidence metadata — 19 tests passing
• Phase 3 (fork PR SeCuReDmE-main-dev/haystack_case_study#2): explicit HITL decision statuses — 26 tests passing

All branches pass ruff check and ruff format. All new fields are opt-in or additive with no breaking changes.

No further work will be performed until maintainers respond. The next action depends entirely on your feedback — whether that is naming guidance, scope adjustment, acceptance, or rejection.

The corresponding RFC is #11332. Happy to answer any questions.

[SeCuReDmE-main-dev]

@coderabbitai review

[anakin87]

Sorry for the long delay and thanks for your work.

First of all, I'd not modify LLMEvaluator: a user can already just check results[i] is None.

I'd focus on inheriting evaluators instead, where evaluation status might get wrongly discarded.
There, I'd add status to the results dict. Something like this in _postprocess_results:

for idx, res in enumerate(result["results"]):
    if res is None:
        result["results"][idx] = {"statements": [], "statement_scores": [], "score": float("nan"), "status": "error"}
        continue
    res["status"] = "evaluated"
    ...

This would not require modifying @component.output_types.
Also, since _postprocess_results is shared by run and run_async, this will work on both paths.

Feel free to modify this PR or open another one.

[SeCuReDmE-main-dev]

This was part of the rfc #11332 can you please judge the entire thing over there and write me what you desire more clearly it will be a pleasure for me to help you guys

[anakin87]

This was part of the rfc #11332 can you please judge the entire thing over there and write me what you desire more clearly it will be a pleasure for me to help you guys

Yes, I skimmed that.

For the moment, I suggest just doing what I proposed in #11333 (review): it's a small contribution but already improves things.

[SeCuReDmE-main-dev]

Perfect im not home i will do when im home this will be done by the end of today thanks you

[SeCuReDmE-main-dev]

im on it

[SeCuReDmE-main-dev]

While I have your attention, this project is published on the Neutrosophic Science International Association website.

I also have an ongoing study related to Haystack, with a published white paper available here.

Finally, I created a cool tool that I think you may find useful: Neutrosophic Lab VM Public Guide.

Go check it out! ^_-

[SeCuReDmE-main-dev]

Updated the PR with the scoped change you proposed: status is now preserved inside the ContextRelevanceEvaluator and FaithfulnessEvaluator result dictionaries, with no LLMEvaluator output change. Sync/async and success, zero-score, generation/parsing, partial-failure, and all-failure cases are covered. The focused local evaluator suites pass; CI is running.

[anakin87]

Thank you!

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/evaluators
context_relevance.py
faithfulness.py
llm_evaluator.py
Project Total

This report was generated by python-coverage-comment-action