Describe the bug
I believe commit 65554d6 introduced a regression that impacts eval sets that do not have an associated test_config.json.
Prior to this commit, when loading the evaluation criteria, the code would check to see if config_path existed before attempting to load it:
def find_config_for_test_file(test_file: str):
"""Find the test_config.json file in the same folder as the test file."""
test_folder = os.path.dirname(test_file)
config_path = os.path.join(test_folder, "test_config.json")
if os.path.exists(config_path):
config_data = load_json(config_path)
if "criteria" in config_data and isinstance(
config_data["criteria"], dict
):
return config_data["criteria"]
else:
raise ValueError(
f"Invalid format for test_config.json at {config_path}. Expected a"
" 'criteria' dictionary."
)
return DEFAULT_CRITERIA
If the path did not exist, a default criteria was returned.
After 65554d6, the check for the existence of config_path was removed, and the get_evaluation_criteria_or_default(config_path) does not perform this check. This lead to a regression when we did a version bump -- previously working tests started failing due to the following error:
../../opt/mambaforge/envs/adk/lib/python3.10/site-packages/google/adk/evaluation/eval_config.py:87: in get_evaluation_criteria_or_default
with open(eval_config_file_path, "r", encoding="utf-8") as f:
E FileNotFoundError: [Errno 2] No such file or directory: 'tests/test_config.json'
================================================= short test summary info ==================================================
ERROR tests/sample_test.py - FileNotFoundError: [Errno 2] No such file or directory: 'tests/test_config.json'
To Reproduce
Try running a .evalset.json file that lacks a test_config.json in the same directory.
Expected behavior
If a test_config.json file is not present in the test directory, a default configuration is used.
Describe the bug
I believe commit 65554d6 introduced a regression that impacts eval sets that do not have an associated
test_config.json.Prior to this commit, when loading the evaluation criteria, the code would check to see if
config_pathexisted before attempting to load it:If the path did not exist, a default criteria was returned.
After 65554d6, the check for the existence of
config_pathwas removed, and theget_evaluation_criteria_or_default(config_path)does not perform this check. This lead to a regression when we did a version bump -- previously working tests started failing due to the following error:To Reproduce
Try running a
.evalset.jsonfile that lacks atest_config.jsonin the same directory.Expected behavior
If a
test_config.jsonfile is not present in the test directory, a default configuration is used.