feat(evaluators): add native run_async support to LLMEvaluator

[GovindhKishore]

Related Issues

• fixes feat: Add asynchronous support (run_async) to LLMEvaluator #11579

Proposed Changes:

Added native asynchronous support (run_async) to LLMEvaluator, FaithfulnessEvaluator, and ContextRelevanceEvaluator.This allows evaluation pipelines to run concurrently inside asynchronous environments (like FastMCP or FastAPI) without stalling the main event loop during LLM network requests.

How it works:

• Runtime check for async support: Verifies whether the assigned chat generator natively supports async execution. If it only supports synchronous execution, it automatically drops down to a safe thread-pool fallback via asyncio.to_thread instead of blocking the event loop.
• Post-processing loop mirroring: FaithfulnessEvaluator and ContextRelevanceEvaluator explicitly mirror their synchronous counterparts, calling the parent async engine and running their unique metrics parsing loops over the concurrent batch results.
• Non-blocking progress bars: Processes inputs concurrently while using async_tqdm to keep tracking completely non-blocking.
• Monitored failure tracking: Strictly matches the existing sync error-handling configurations (raise_on_failure), safely caching individual row exceptions into NaN targets when disabled.

How did you test it?

Added complete async test suites across all three evaluator test modules (TestLLMEvaluatorAsync, TestFaithfulnessEvaluatorAsync, and TestContextRelevanceEvaluatorAsync). The tests cover:

• Standard async flow: Verifying that concurrent evaluation batches run successfully and parse the returned JSON data accurately.
• Thread-pool fallback execution: Confirming that when a sync-only chat generator is passed, the engine safely offloads the execution to a worker thread via asyncio.to_thread without stalling the event loop.
• Metric post-processing math: Validating that the custom scoring and statement parsing loops in FaithfulnessEvaluator and ContextRelevanceEvaluator aggregate data identically to their synchronous counterparts.
• Error isolation and NaN propagation: Testing that when raise_on_failure=False, individual API failures are safely skipped, logged as warnings, and tracked as NaN values without breaking the final average score calculation.

Notes for the reviewer

The implementation structure mirrors the synchronous run method identically to keep the component maintenance straightforward. The async test fixtures utilize standard monkeypatch strategies to mirror existing testing conventions throughout the file.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have updated the related issue with new insights and changes.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title: feat: add run_async support to LLMEvaluator
• I have documented my code.
• I have added a release note file, following the contributors guidelines.
• I have run pre-commit hooks and fixed any issue.

[vercel]

@GovindhKishore is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[GovindhKishore]

@sjrl The CI is failing because ContextRelevanceEvaluator and FaithfulnessEvaluator inherit from LLMEvaluator but override the sync run method with extra output fields. This clashes with the inherited base run_async schema and throws a ComponentError. Before touching those components, I want to confirm this is still within scope for this PR. Since most of the underlying generators have async support either directly or inherited from OpenAIChatGenerator, implementing run_async for these child components too makes sense and keeps things consistent.

Is this approach fine and within scope?

I plan to add matching run_async overrides to both child classes. They will call await super().run_async(**inputs) to use the new async engine, then handle their specific metrics to match their sync counterparts.

Let me know if this works or if you prefer a different pattern!

[sjrl]

I plan to add matching run_async overrides to both child classes. They will call await super().run_async(**inputs) to use the new async engine, then handle their specific metrics to match their sync counterparts.

Let me know if this works or if you prefer a different pattern!

Yes please do! Your approach sounds good and I'll double check it once you have it in

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/evaluators
context_relevance.py
faithfulness.py
llm_evaluator.py					306-307, 312
Project Total

This report was generated by python-coverage-comment-action

[GovindhKishore]

I plan to add matching run_async overrides to both child classes. They will call await super().run_async(**inputs) to use the new async engine, then handle their specific metrics to match their sync counterparts.
Let me know if this works or if you prefer a different pattern!

Yes please do! Your approach sounds good and I'll double check it once you have it in

@sjrl I've updated the implementation. Kindly take your time to leave a review.

[sjrl]

Thanks!

[GovindhKishore]

Thanks!

Thanks a lot for your guidance and merge @sjrl