feat: Revision third-party eval metrics adapter (DeepEval + Autoevals) by stone-coding · Pull Request #568 · aws/bedrock-agentcore-sdk-python

stone-coding · 2026-07-06T17:07:35Z

Summary

Add DeepEvalAdapter and AutoevalsAdapter that integrate third-party evaluation metrics with
AgentCore's code-based evaluator framework
Built-in StrandsSpanMapper (class-based) auto-extracts input/output from Strands agent
OTel spans — supports both inline events (unified ADOT) and span body (CloudWatch ADOT) formats
customer_mapper parameter lets customers bypass the built-in mapper and return the metric
library's native type directly:
- DeepEval: customer_mapper: Callable[[EvaluatorInput], LLMTestCase]
- Autoevals: customer_mapper: Callable[[EvaluatorInput], Dict[str, Any]]
Simplified SpanMapResult — only fields metrics actually consume (input, actual_output,
retrieval_context, system_prompt, expected_output)
Removed separate validate_fields() step — validation happens inside each adapter's _run()
method when building the test case
_filter_spans_by_target applied only for built-in mapper path (not for customer mapper)
Never raises unhandled exceptions — all error paths return valid EvaluatorOutput with
structured errorCode/errorMessage

Test plan

83 unit tests passing (pytest tests/.../third_party/ -v)
E2E on-demand: built-in mapper → Score 1.0 Pass (Strands healthcare agent)
E2E on-demand: custom mapper → Score 1.0 Pass (non-Strands spans)
E2E online eval: pipeline invokes Lambda, score 0.98 Pass (CloudWatch results)
Error handling: 4/4 (malformed, unsupported scope, empty attrs, success schema)

Introduces a new integrations/deepeval/ module that adapts AgentCore Lambda evaluation events into DeepEval LLMTestCase objects, runs any BaseMetric, and returns structured score/label/explanation responses.

…leTurnParams deprecation

…d EvaluatorInput support

… layer, simplify per TJ/Irene feedback

…ion, add validate_fields to DeepEvalAdapter

…xpected_response_text property

- Rename span_parsers → span_mappers, simplify to Strands-only - Rename field_mapper → customer_mapper across all adapters - customer_mapper now returns native types directly: - DeepEval: EvaluatorInput → LLMTestCase - Autoevals: EvaluatorInput → Dict[str, Any] (eval kwargs) - Refactor BaseAdapter: remove intermediate execute(), add _run() pattern - 83 unit tests passing

chianhw · 2026-07-08T18:56:48Z

+            customer_mapper=lambda ev: {
+                "input": ev.session_spans[0]["attributes"]["question"],
+                "output": ev.session_spans[0]["attributes"]["answer"],
+                "expected": "the expected answer",
+            },


Do we need to accept aws lambda?

Replaced lambda examples with named functions in docstrings

chianhw · 2026-07-08T20:16:27Z

+    def __init__(
+        self,
+        scorer: Any,
+        customer_mapper: Optional[Callable[[EvaluatorInput], Dict[str, Any]]] = None,


We have not updated this type Dict[str, Any]?

Regarding naming, "Custom mapper" would be a better choice for describing an arbitrary mapper provided by a customer?

1.The type Dict[str, Any] is correct — Autoevals has no typed
input class like DeepEval's LLMTestCase. Scorers take plain
kwargs (input, output, expected) and the dict gets unpacked
as metric.eval(**kwargs). Different scorers need different
keys, so a generic dict is the right contract.

chianhw · 2026-07-08T21:00:07Z

+        self,
+        scorer: Any,
+        customer_mapper: Optional[Callable[[EvaluatorInput], Dict[str, Any]]] = None,
+        threshold: float = 0.5,


label: Optional[str] = None label in EvaluatorOutput is optional.
We don't need to have a default value to generate a label when a metric doesn't have a label and the user doesn't provide any threshold. Can we set default as None?

Changed threshold default to None. When no threshold is set,
no Pass/Fail judgment is made. Note: EvaluatorOutput's
validator currently requires a label for success responses —
I'm returning the score as the label string when threshold is
None. Let me know if you'd rather relax the validator to
allow label=None.

chianhw · 2026-07-08T21:00:37Z

+                mapping when provided. Expected keys: input, output, expected (optional).
+            threshold: Score threshold for Pass/Fail determination. Defaults to 0.5.
+        """
+        self.scorer = scorer


this naming is not aligned with DeepEval adaptor. Looks like you haven't updated AutoevalsAdapter.
If you haven't completed end to end tests for AutoevalsAdapter, you should not include it in your PR.

Working on E2E test plan now. Considering how to strcuture parameterized Lambda handlers and how to obtains span from strands, openinference, opentelemetry, and customer mappers.

chianhw · 2026-07-08T21:00:43Z

+            customer_mapper=lambda ev: LLMTestCase(
+                input=ev.session_spans[0]["attributes"]["user_query"],
+                actual_output=ev.session_spans[0]["attributes"]["response"],
+            ),


using lambda? same as above

chianhw · 2026-07-08T21:00:49Z

+    def __init__(
+        self,
+        metric: BaseMetric,
+        customer_mapper: Optional[Callable[[EvaluatorInput], LLMTestCase]] = None,


naming? same as above

chianhw · 2026-07-08T21:00:52Z

+        self.metric = metric
+        self.customer_mapper = customer_mapper
+        if model is not None:
+            self.metric.model = model


Where do we use self.metric.model?

chianhw · 2026-07-08T21:16:32Z

+    Raises:
+        ValueError: If no mapper can extract data from the spans.
+    """
+    result = _strands_mapper.map(session_spans)


We can check the span scope name and then decide which mapper to use.

Applied to all three mappers (Strands, OpenInference LangChain, OpenTelemetry LangChain).

chianhw · 2026-07-13T23:38:29Z

+            attributes = span.get("attributes", {})
+            if attributes.get("gen_ai.operation.name") == "invoke_agent":
+                return span
+        return None


Does this mapper only take the first agent span? For strands, the OtelSpanMapper in OpenTelemetryMapper package use the last agent span. That is, the agent span with the latest end_time per trace is the one retained.

Fixed, now takes the last agent span to match server-side OtelSpanMapper behavior. Applied to all three mappers (Strands, OpenInference LangChain, OpenTelemetry LangChain).

chianhw · 2026-07-14T05:13:09Z

+                expected_output=result.expected_output,
+                context=result.context,
+                retrieval_context=result.retrieval_context,
+            )


why don't we have tools_called and expected_tools?
Do we want to support ToolCorrectnessMetric, ArgumentCorrectnessMetric in DeepEval?

Added both. tools_called is now extracted from execute_tool spans (name + input_parameters + output) and mapped to LLMTestCase.tools_called. expected_tools comes from reference_inputs[0].expectedTrajectory.toolNames → LLMTestCase.expected_tools. This enables ToolCorrectnessMetric and ArgumentCorrectnessMetric.

chianhw · 2026-07-14T06:07:38Z

+    actual_output: Optional[str] = None
+    retrieval_context: Optional[List[str]] = None
+    context: Optional[List[str]] = None
+    system_prompt: Optional[str] = None


Are we using system_prompt for any metric?

Currently no DeepEval or Autoevals metric consumes system_prompt, LLMTestCase doesn't have this field. I include it in SpanMapResult for two reasons: (1) alignment with the server-side AgentInvocationSpan which also extracts it, and (2) forward-compatibility, customers using custom_mapper can access it from the spans if a future metric needs it. Happy to remove if you feel it adds unnecessary surface area.

No, I agree with you. We should keep it.

chianhw · 2026-07-14T06:08:03Z

+            expected = getattr(ref, "expected_response_text", None)
+            if expected:
+                result.expected_output = expected
+        return result


We only take the data for expected_output here, but why don't we use the expected_trajectory field and assertions in reference_inputs?

updated. Now all three fields from reference_inputs are extracted:

expectedResponse.text → SpanMapResult.expected_output → LLMTestCase.expected_output

expectedTrajectory.toolNames → SpanMapResult.expected_tools → LLMTestCase.expected_tools (enables ToolCorrectnessMetric)

assertions[].text → SpanMapResult.assertions → LLMTestCase.context (enables HallucinationMetric to check agent response against ground-truth claims)
For the Autoevals adapter, assertions are joined as a fallback for the expected kwarg when no explicit expectedResponse is provided.

chianhw · 2026-07-14T06:08:29Z

+            input=input_text,
+            actual_output=actual_output,
+            retrieval_context=tool_outputs if tool_outputs else None,
+            context=tool_outputs if tool_outputs else None,


Why do we use tool_output as context?

DeepEval uses context and retrieval_context for different metrics: HallucinationMetric requires context (ground-truth facts to check against), while FaithfulnessMetric requires retrieval_context (retrieved documents to verify grounding). Tool outputs serve both purposes — they're what the agent retrieved AND the factual basis for its response. Setting both ensures both metrics work without requiring a custom_mapper. Note: when assertions from reference_inputs are present, they override context (explicit ground truth takes
precedence over tool outputs). This matches the server-side pattern — the AgentInvocationSpan exposes tool results, and the service's built-in metrics use them as both retrieval context and grounding truth.

haomiao037 added 14 commits June 30, 2026 11:59

Add DeepEvalHandler integration with unit tests

77d16a9

Introduces a new integrations/deepeval/ module that adapts AgentCore Lambda evaluation events into DeepEval LLMTestCase objects, runs any BaseMetric, and returns structured score/label/explanation responses.

Fix span extraction to use real AgentCore _eval_log_records structure

402ea78

Set context field from tool messages for HallucinationMetric support

e9ef47d

Use metric.success for label instead of manual threshold comparison

f97827e

Add model override and timeout enforcement to DeepEvalHandler

9d256ae

Add model override, timeout enforcement, use metric.success, fix Sing…

c142d50

…leTurnParams deprecation

Fix _get_required_params to handle GEval unmappable typing params

3ccc98c

Add .deepeval/ to gitignore

a884f91

Move model override to init to avoid per-call mutation

6ac198c

Refactor to BaseAdapter framework with DeepEval/Autoevals adapters an…

8d415e5

…d EvaluatorInput support

Major refactor: move to custom_code_based_evaluators, add span parser…

8627ab0

… layer, simplify per TJ/Irene feedback

Fix review items: add reference_inputs to model, tighten error detect…

9a9b6a7

…ion, add validate_fields to DeepEvalAdapter

Adapt to upstream ReferenceInput model: remove duplicate field, use e…

0499d4b

…xpected_response_text property

stone-coding requested a review from a team July 6, 2026 17:07

stone-coding requested a deployment to manual-approval July 6, 2026 17:07 — with GitHub Actions Waiting

chianhw reviewed Jul 8, 2026

View reviewed changes

chianhw reviewed Jul 13, 2026

View reviewed changes

chianhw reviewed Jul 14, 2026

View reviewed changes

Uh oh!

Conversation

stone-coding commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stone-coding commented Jul 6, 2026 •

edited

Loading