Skip to content

AgentEvaluator fails to execute if test_config.json is not found #3305

@juszzz

Description

@juszzz

Describe the bug
I believe commit 65554d6 introduced a regression that impacts eval sets that do not have an associated test_config.json.

Prior to this commit, when loading the evaluation criteria, the code would check to see if config_path existed before attempting to load it:

  def find_config_for_test_file(test_file: str):
    """Find the test_config.json file in the same folder as the test file."""
    test_folder = os.path.dirname(test_file)
    config_path = os.path.join(test_folder, "test_config.json")
    if os.path.exists(config_path):
      config_data = load_json(config_path)
      if "criteria" in config_data and isinstance(
          config_data["criteria"], dict
      ):
        return config_data["criteria"]
      else:
        raise ValueError(
            f"Invalid format for test_config.json at {config_path}. Expected a"
            " 'criteria' dictionary."
        )
    return DEFAULT_CRITERIA

If the path did not exist, a default criteria was returned.

After 65554d6, the check for the existence of config_path was removed, and the get_evaluation_criteria_or_default(config_path) does not perform this check. This lead to a regression when we did a version bump -- previously working tests started failing due to the following error:

../../opt/mambaforge/envs/adk/lib/python3.10/site-packages/google/adk/evaluation/eval_config.py:87: in get_evaluation_criteria_or_default
    with open(eval_config_file_path, "r", encoding="utf-8") as f:
E   FileNotFoundError: [Errno 2] No such file or directory: 'tests/test_config.json'
================================================= short test summary info ==================================================
ERROR tests/sample_test.py - FileNotFoundError: [Errno 2] No such file or directory: 'tests/test_config.json'

To Reproduce
Try running a .evalset.json file that lacks a test_config.json in the same directory.

Expected behavior
If a test_config.json file is not present in the test directory, a default configuration is used.

Metadata

Metadata

Assignees

Labels

eval[Component] This issue is related to evaluation

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions