feat(python): add evaluation framework (DeepEval + Ragas) by RoyiAizenstain · Pull Request #1453 · i-am-bee/beeai-framework

RoyiAizenstain · 2026-05-15T11:48:02Z

Which issue(s) does this pull-request address?

Closes: #

Description

Adds a Python evaluation framework for systematically measuring BeeAI
agent quality on multi-hop QA, tool usage, and factual accuracy. Rather
than eyeballing outputs, users can run a fixed dataset of questions
through an agent and score the responses with pluggable metrics.

What's included:

Adapters (python/evaluation/adapters/) that bridge BeeAI's
ChatModel to two popular eval frameworks, so any of the 26+ BeeAI
providers can serve as the judge LLM:
- DeepEvalLLM → deepeval.DeepEvalBaseLLM
- InstructorRagasLLM → ragas.InstructorBaseRagasLLM
Reference experiments (python/examples/evaluation/):
- Shared agent.py (a RequirementAgent for multi-hop QA with
  Wikipedia / OpenMeteo / PythonTool) and dataset.json ground truth.
- deepeval/experiment.py — runs the agent against the dataset and
  scores it with built-in DeepEval metrics plus three custom ones
  (AnswerLLMJudgeMetric, ToolUsageMetric, FactsSimilarityMetric).
- ragas/experiment.py — async-script runner with built-in Ragas
  metrics plus a Pydantic-structured FactsSimilarityMetric.
Docs (docs/modules/evaluation.mdx) — adapter usage, env vars,
invocation, output file locations.

Design highlights:

Adapter pattern keeps eval frameworks pluggable behind BeeAI's
ChatModel.
Agent-under-test and judge-LLM are decoupled via AGENT_CHAT_MODEL_NAME
/ EVAL_CHAT_MODEL_NAME env vars to avoid self-bias.
Default fallback model is ollama:llama3.1:8b, matching BeeAI's
conventional default.
No module-load side effects: ragas/dataset.py exposes
build_dataset() callable, no pytest decorators in example scripts.

No breaking changes — purely additive; existing modules untouched
apart from new docs.

Checklist

General

I have read the appropriate contributor guide: Python
I have signed off on my commit: Python instructions
Commit messages and PR title follow conventional commits
Appropriate label(s) added to PR: Python

Code quality checks

Code quality checks pass: mise check (mise fix to auto-fix)

Testing

Unit tests pass: mise test:unit
E2E tests pass: mise test:e2e
Tests are included (for bug fixes or new features)

Documentation

Documentation is updated
Embedme embeds code examples in docs. To update after edits, run: Python mise docs:fix

- Add evaluation core concepts page (docs/modules/evaluation.mdx) - Add DeepEval and Ragas adapters bridging BeeAI ChatModel - Add evaluation examples with shared dataset, agent, and custom metrics Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Add missing license headers - Replace print with logging in ragas adapter - Fix deprecated asyncio.get_event_loop usage - Narrow exception handling, remove unused import - Pass kwargs through in from_name factory Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

- Replace print with logging in experiments and metrics - Remove duplicate LLMTestCase import in ToolUsageMetric - Remove unused Tool import in agent.py - Use collections.Counter instead of typing.Counter - Move ragas judge LLM creation outside experiment loop - Derive ragas reference tool calls from supporting_titles - Narrow exception handling in metric scoring - Use Hungarian-free variable names in deepeval experiment Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…consistency

gemini-code-assist

Code Review

This pull request introduces a new evaluation module for BeeAI agents, providing integration with DeepEval and Ragas frameworks via specialized adapters. It includes documentation, custom metrics for answer accuracy and tool usage, and example experiment scripts. Critical feedback identifies a type contract violation in the DeepEval adapter, a signature mismatch in the Ragas adapter, and flawed configuration merging for VertexAI. Further improvements are suggested to handle nested event loops during evaluation, implement robust regex-based JSON parsing, and prevent resource leaks from temporary directory creation.

gemini-code-assist · 2026-05-15T11:49:26Z

            )
        )
        text = response.get_text_content()
        return schema.model_validate_json(text) if schema else text  # type: ignore


The a_generate method is defined in the DeepEvalBaseLLM interface to return a str. Returning a Pydantic model instance (schema.model_validate_json(text)) violates this contract and will cause type errors in DeepEval metrics that expect a string response to parse. Since the BeeAI framework already validates the output against the schema when response_format is used, you should return the raw JSON string.

Suggested change

return schema.model_validate_json(text) if schema else text # type: ignore

return text

gemini-code-assist · 2026-05-15T11:49:27Z

+        if isinstance(name, str) and name.startswith("vertexai:"):
+            merged: dict = {"allow_prompt_caching": False}
+            if isinstance(options, dict):
+                merged.update(options)
+            options = merged or options


The logic for merging options when the provider is vertexai is flawed. If options is an instance of ChatModelParameters (rather than a dict), it will be completely overwritten by the merged dictionary, causing all original configuration to be lost. You should handle both dict and object types or use a more robust merging strategy.

Suggested change

if isinstance(name, str) and name.startswith("vertexai:"):

merged: dict = {"allow_prompt_caching": False}

if isinstance(options, dict):

merged.update(options)

options = merged or options

if isinstance(name, str) and name.startswith("vertexai:"):

if options is None:

options = {"allow_prompt_caching": False}

elif isinstance(options, dict):

options.setdefault("allow_prompt_caching", False)

gemini-code-assist · 2026-05-15T11:49:27Z

+    def __init__(self, model_name: str):
+        self.model = ChatModel.from_name(model_name)


There is a signature mismatch between the constructor and the from_name static method. from_name accepts **kwargs and passes them to __init__, but __init__ only accepts model_name. This will raise a TypeError if any keyword arguments are passed to from_name.

Suggested change

def __init__(self, model_name: str):

self.model = ChatModel.from_name(model_name)

def __init__(self, model_name: str, **kwargs: Any):

self.model = ChatModel.from_name(model_name, **kwargs)

gemini-code-assist · 2026-05-15T11:49:27Z

+        FactsSimilarityMetric(model=eval_model),
+    ]
+
+    eval_results = evaluate(test_cases=test_cases, metrics=metrics)


Calling deepeval.evaluate inside an already running asyncio event loop (triggered by asyncio.run(main()) on line 219) will likely raise a RuntimeError: asyncio.run() cannot be called from a running event loop. deepeval.evaluate is a synchronous function that manages its own event loop internally. You should execute the async data preparation first, then call evaluate from a synchronous context.

gemini-code-assist · 2026-05-15T11:49:27Z

+        clean_json = raw_text.strip()
+        if clean_json.startswith("```json"):
+            clean_json = clean_json[7:]
+        if clean_json.startswith("```"):
+            clean_json = clean_json[3:]
+        if clean_json.endswith("```"):
+            clean_json = clean_json[:-3]
+        clean_json = clean_json.strip()


The manual string slicing used to extract JSON from markdown blocks is fragile. It assumes the output starts exactly with the backticks and doesn't handle cases where the LLM might include preamble or postscript text. Using a regular expression is a more robust approach for extracting JSON content from LLM responses.

Suggested change

clean_json = raw_text.strip()

if clean_json.startswith("```json"):

clean_json = clean_json[7:]

if clean_json.startswith("```"):

clean_json = clean_json[3:]

if clean_json.endswith("```"):

clean_json = clean_json[:-3]

clean_json = clean_json.strip()

import re

match = re.search(r"```json\s*(.*?)\s*```", raw_text, re.DOTALL) or re.search(r"```\s*(.*?)\s*```", raw_text, re.DOTALL)

clean_json = match.group(1).strip() if match else raw_text.strip()

gemini-code-assist · 2026-05-15T11:49:27Z

+
+def create_calculator_tool() -> PythonTool:
+    storage = LocalPythonStorage(
+        local_working_dir=tempfile.mkdtemp("code_interpreter_source"),


tempfile.mkdtemp creates a directory on the file system that is not automatically deleted. In an evaluation context where many agents might be instantiated, this can lead to an accumulation of orphaned temporary directories. It is better to use tempfile.TemporaryDirectory as a context manager or ensure explicit cleanup.

RoyiAizenstain · 2026-06-06T19:31:17Z

can you review it @Tomas2D ?
Thanks :)

Tomas2D

Thanks so much for this contribution, @RoyiAizenstain! 🐝 The Ragas adapter and the runnable end-to-end examples are a genuinely valuable addition — having reference experiments people can copy and adapt is exactly what the evaluation story needs. I left inline comments throughout; here I'll summarize the structural points that need attention before we can merge:

Docs need a rebase onto the new docs system. While this PR was open, the docs migrated from Mintlify to Astro Starlight (#1462), so docs/docs.json and docs/modules/ no longer exist on main and the branch is currently in merge conflict. Could you rebase and move the new page to docs/src/content/docs/modules/evaluation.mdx using the Starlight frontmatter/components? Happy to help if the new setup is unclear.
PR description vs. what's new. DeepEvalLLM already exists at python/eval/model.py (added in #1352) — this PR moves it rather than adds it. The genuinely new pieces (the Ragas adapter and the examples) are great; it would just help reviewers if the description reflected that.
Stranded tutorial. The eval/ → evaluation/ rename moves every file except python/eval/deep_eval tutorial.mdx, which would be left behind as the only file in python/eval/ — and it overlaps with the new docs page. Could you move/merge or remove it as part of this PR?
Missing dependency declarations. The new code imports ragas, nest_asyncio, numpy, and multiprocess, but none of them are declared in python/pyproject.toml (only deepeval = "3.3.5" exists today). Could you add them with pins — ideally behind an optional evaluation group — and double-check that the DeepEval symbols used (ExactMatchMetric, ArgumentCorrectnessMetric, DEEPEVAL_PER_TASK_TIMEOUT_SECONDS_OVERRIDE) exist in the pinned version?
Checklist. DCO sign-off, mise check, and the test runs are still unchecked, and Closes: #<issue-number> is a leftover placeholder. A few of the new files are also missing trailing newlines (agent.py, dataset.py, dataset.json) and carry Copyright 2025 headers — mise fix should take care of most of this.

None of this diminishes the value of the work — once it's rebased and aligned with the existing harness, this will be a really nice addition. Thank you again! 🙏

Tomas2D · 2026-06-12T12:21:22Z

+
+The dataset is a JSON file where each item contains:
+
+| Field                  | Description                                            |


Small mismatch: this table documents expected_answer, supporting_sentences, and expected_tool_calls, but python/examples/evaluation/dataset.json actually uses answer, relevant_sentences, and wiki_times. Could you align the two? (Honestly, the names documented here are the clearer ones — renaming the JSON fields to match might be the nicer fix.)

Tomas2D · 2026-06-12T12:21:22Z

+Wraps a BeeAI `ChatModel` as a `DeepEvalBaseLLM`, enabling DeepEval metrics to call any BeeAI-supported model for LLM-as-judge evaluations. Supports structured output via Pydantic schemas.
+
+```py Python [expandable]
+from evaluation.adapters import DeepEvalLLM


Heads-up: this presents evaluation.adapters as framework API, but python/evaluation/ is a repo-level directory that isn't part of the published beeai_framework wheel — so users who pip install beeai-framework won't be able to run this import. If we want the adapters to be public API, they should live under beeai_framework/ (perhaps gated behind an optional extra); otherwise the docs should say explicitly that this requires a repo checkout.

Tomas2D · 2026-06-12T12:21:23Z

+            merged: dict = {"allow_prompt_caching": False}
+            if isinstance(options, dict):
+                merged.update(options)
+            options = merged or options


Two thoughts here:

merged is always non-empty at this point, so merged or options always evaluates to merged — the or options part is dead code.

A vertexai:-specific workaround hardcoded inside a generic adapter (and duplicated in examples/evaluation/agent.py) feels like the wrong layer. If prompt caching misbehaves on VertexAI, the fix probably belongs in the VertexAI backend adapter — or at minimum only in the example, with a comment explaining why.

Tomas2D · 2026-06-12T12:21:23Z

        return None

-    # pyrefly: ignore [bad-override]
    def generate(self, prompt: str, schema: BaseModel | None = None) -> str:


The removed # pyrefly: ignore [bad-override] comments were added during the pyrefly migration (#1347), and the overridden signatures haven't changed — so I'd expect mise check to fail without them. Could you run mise check and restore whatever it needs?

Tomas2D · 2026-06-12T12:21:23Z

+class InstructorRagasLLM(InstructorBaseRagasLLM):
+    """A class that bridges Ragas with BeeAI directly (without LangChain intermediary)."""
+
+    def __init__(self, model_name: str):


For consistency with DeepEvalLLM (which wraps an existing ChatModel), consider __init__(self, model: ChatModel) plus a from_name() factory. Taking only a name string means callers can't pass a configured model (options, parameters, middleware).

Relatedly, from_name(model_name, **kwargs) below forwards **kwargs into this constructor, which accepts none — so any kwarg raises TypeError.

Tomas2D · 2026-06-12T12:21:23Z

+        self.success = score >= self.threshold
+        return score
+
+    def measure(self, test_case: LLMTestCase) -> float:


asyncio.run() raises RuntimeError when called from a running event loop — which is the common case under deepeval.evaluate with async metrics. Since async_mode = True already routes everything through a_measure, consider having measure() raise NotImplementedError (like the adapter does) rather than appearing to work. Same applies to FactsSimilarityMetric.measure().

Tomas2D · 2026-06-12T12:21:23Z

+# Load dataset
+_script_dir = Path(__file__).parent
+_data_dir = _script_dir / "data"
+dataset = Dataset.load(name="my_evaluation", backend="local/csv", root_dir=str(_data_dir))


The PR description mentions "no module-load side effects", but this module loads the dataset and constructs the judge LLM at import time — so importing it fails outright if dataset.py hasn't been run yet. Moving both into main() would make the example match the description (and play nicer with tooling that imports modules to inspect them).

Tomas2D · 2026-06-12T12:21:23Z

+
+# Suppress multiprocess resource tracker warnings on Windows - must be before other imports
+# TODO: remove once multiprocess fixes ResourceTracker.__del__ on Windows
+if sys.platform == "win32":


This is quite a lot of workaround machinery for an example users will copy as a starting point. The sys.path.insert calls above can go away with proper package-relative imports, and for the ResourceTracker monkey-patch it would be good to link the upstream multiprocess issue it works around — or drop it if it only suppresses noise at interpreter shutdown.

Tomas2D · 2026-06-12T12:21:23Z

+]
+
+
+def create_calculator_tool() -> PythonTool:


The example agent unconditionally includes PythonTool, which needs a code interpreter running at http://127.0.0.1:50081 — but neither dataset question requires it, and the docs never mention starting one. Suggest dropping it from the default agent or making it opt-in via an env var, so the example works out of the box.

Tomas2D · 2026-06-12T12:21:23Z

+        )
+
+    options: dict = {"allow_parallel_tool_calls": True}
+    if model_name.startswith("vertexai:"):


Same vertexai: workaround as in DeepEvalLLM.from_name — see my comment there; it would be nice to have this live in one place (ideally the VertexAI backend) rather than duplicated.

Tomas2D · 2026-06-15T15:21:01Z

Hi @RoyiAizenstain — circling back on this one. Whenever you have time, the main open items from my review are:

Rebasing onto main (the branch currently conflicts after the docs migration from Mintlify to Astro Starlight), and moving the docs page under docs/src/content/docs/modules/.
Signing off the commits so DCO passes.
The code points in the inline comments.

No rush, and happy to help if any of it is unclear. Thanks again for the contribution!

RoyiAizenstain · 2026-06-16T14:46:04Z

Thanks! I will review and fix what you asked for :) @Tomas2D

RoyiAizenstain and others added 6 commits April 10, 2026 15:37

refactor: merge eval/ into evaluation/ and remove duplicate model.py

efd0834

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

chore(examples): delete unused ragas experiment CSV

972bbf3

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

refactor: update evaluation framework documentation and examples for …

8ddfb63

…consistency

RoyiAizenstain requested review from a team as code owners May 15, 2026 11:48

dosubot Bot added the size:XXL This PR changes 1000+ lines, ignoring generated files. label May 15, 2026

github-actions Bot added documentation Improvements or additions to documentation python Python related functionality labels May 15, 2026

gemini-code-assist Bot reviewed May 15, 2026

View reviewed changes

Tomas2D requested changes Jun 12, 2026

View reviewed changes

	return schema.model_validate_json(text) if schema else text # type: ignore
	return text

		def __init__(self, model_name: str):
		self.model = ChatModel.from_name(model_name)


		The dataset is a JSON file where each item contains:

		\| Field \| Description \|

Uh oh!

Conversation

RoyiAizenstain commented May 15, 2026

Which issue(s) does this pull-request address?

Description

Checklist

General

Code quality checks

Testing

Documentation

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 15, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 15, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 15, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 15, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 15, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 15, 2026

Choose a reason for hiding this comment

Uh oh!

RoyiAizenstain commented Jun 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Tomas2D left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Tomas2D commented Jun 15, 2026

Uh oh!

RoyiAizenstain commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

RoyiAizenstain commented Jun 6, 2026 •

edited

Loading

RoyiAizenstain commented Jun 16, 2026 •

edited

Loading