Skip to content

AOAI remote evals #4148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ebwinters merged 5 commits into from
May 7, 2025
Merged

AOAI remote evals #4148

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
199424f
add aoai changes
ebwinters Apr 25, 2025
4bc433e
upgrade
ebwinters May 6, 2025
e98ba9e
pin otel
ebwinters May 6, 2025
4694dfd
pin otel api
ebwinters May 6, 2025
bea046a
Merge branch 'main' into etwinter/aoai_remote
ebwinters May 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions assets/evaluation_on_cloud/environments/evaluations-built-in/context/evaluate_on_data.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,9 @@

from azure.ai.ml.identity import AzureMLOnBehalfOfCredential
from azure.ai.evaluation import evaluate
from azure.ai.evaluation._evaluate._evaluate_aoai import (
_convert_remote_eval_params_to_grader
)
from save_evaluation import load_evaluator
from model_target import ModelTarget

Expand Down Expand Up @@ -84,10 +87,14 @@ def initialize_evaluators(command_line_args):
for evaluator_name, evaluator in evaluators_o.items():
init_params = evaluator["InitParams"]
update_value_in_dict(init_params, "AZURE_OPENAI_API_KEY", lambda x: os.environ[x.upper()])
flow = load_evaluator("./" + evaluator_name)
if any(rai_eval in evaluator["Id"] for rai_eval in rai_evaluators):
init_params["credential"] = AzureMLOnBehalfOfCredential()
evaluators[evaluator_name] = flow(**init_params)
if evaluator["Id"].startswith("aoai://"):
grader = _convert_remote_eval_params_to_grader(evaluator["Id"], init_params)
evaluators[evaluator_name] = grader
else:
flow = load_evaluator("./" + evaluator_name)
if any(rai_eval in evaluator["Id"] for rai_eval in rai_evaluators):
init_params["credential"] = AzureMLOnBehalfOfCredential()
evaluators[evaluator_name] = flow(**init_params)
return evaluators


Expand Down
6 changes: 3 additions & 3 deletions assets/evaluation_on_cloud/environments/evaluations-built-in/context/requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
azure-ai-evaluation==1.2.0
azure-ai-evaluation==1.6.0
azureml-mlflow=={{latest-pypi-version}}
azure-ai-ml=={{latest-pypi-version}}

openai=={{latest-pypi-version}}
opentelemetry-api=={{latest-pypi-version}}
opentelemetry-sdk=={{latest-pypi-version}}
opentelemetry-api==1.31.0
opentelemetry-sdk==1.31.0
azure-monitor-query=={{latest-pypi-version}}
croniter=={{latest-pypi-version}}
azure-monitor-opentelemetry=={{latest-pypi-version}}
Expand Down
Loading