Skip to content

Commit 3b495aa

Browse files
ahmad-naderahmad-nader
authored
feat(azure-ai-projects): add endpoint evaluator samples for API Key auth and entra id auth (#47669)
1 parent 4cdae03 commit 3b495aa

4 files changed

Lines changed: 583 additions & 0 deletions

File tree

‎sdk/ai/azure-ai-projects/samples/evaluations/README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,13 @@ Located in the [agentic_evaluators](https://github.com/Azure/azure-sdk-for-pytho
114114
| [sample_generic_agentic_evaluator](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/ai/azure-ai-projects/samples/evaluations/agentic_evaluators/sample_generic_agentic_evaluator) | Generic agentic evaluator example |
115115

116116

117+
### Endpoint-Based (Custom) Evaluators
118+
119+
| Sample | Description | Requirements |
120+
|--------|-------------|--------------|
121+
| [sample_endpoint_evaluator_with_api_key.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/sample_endpoint_evaluator_with_api_key.py) | Bring your own HTTP endpoint as an evaluator, authenticated with an API key | Scoring endpoint, `azure-mgmt-cognitiveservices` |
122+
| [sample_endpoint_evaluator_with_entra_id.py](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/sample_endpoint_evaluator_with_entra_id.py) | Bring your own HTTP endpoint as an evaluator, authenticated with Entra ID managed identity | Scoring endpoint with Easy Auth, `azure-mgmt-cognitiveservices` |
123+
117124
### Evaluation Result Insights & Analysis
118125

119126
| Sample | Description |

‎sdk/ai/azure-ai-projects/samples/evaluations/sample_endpoint_evaluator_with_api_key.py‎

Lines changed: 285 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,285 @@
1+
# pylint: disable=line-too-long,useless-suppression
2+
# ------------------------------------
3+
# Copyright (c) Microsoft Corporation.
4+
# Licensed under the MIT License.
5+
# ------------------------------------
6+
7+
"""
8+
DESCRIPTION:
9+
Given an AIProjectClient, this sample demonstrates the end-to-end workflow for
10+
running an evaluation using an endpoint-based evaluator with API Key authentication:
11+
12+
1. Create a workspace connection with API Key credentials
13+
2. Register an endpoint evaluator backed by the connection
14+
3. Create an evaluation with the endpoint evaluator as testing criteria
15+
4. Run the evaluation with inline data
16+
5. Poll for results and display output
17+
18+
Endpoint evaluators allow you to bring your own HTTP endpoint for evaluation. The service
19+
POSTs each evaluation row to your endpoint and expects a JSON response with label/score
20+
and optional reasoning fields. Authentication is resolved server-side from a workspace
21+
connection.
22+
23+
Your endpoint must accept POST requests with a JSON body like:
24+
{
25+
"schema_version": "0.0.1",
26+
"evaluator_name": "...",
27+
"evaluator_version": "...",
28+
"evaluation_level": "turn",
29+
"data": {
30+
"item": {"query": "...", "context": "..."},
31+
"sample": {"response": "..."}
32+
}
33+
}
34+
And return a JSON response like:
35+
{
36+
"schema_version": "0.0.1",
37+
"score": 0.85,
38+
"reason": "Response is accurate and concise.",
39+
"status": "Completed", // Valid values: "Completed", "Error", "Skipped"
40+
"properties": {"model_used": "gpt-4o", "custom_flag": true},
41+
"threshold": null,
42+
"passed": true
43+
}
44+
45+
USAGE:
46+
python sample_endpoint_evaluator_with_api_key.py
47+
48+
Before running the sample:
49+
50+
pip install "azure-ai-projects>=2.3.0" azure-mgmt-cognitiveservices python-dotenv
51+
52+
Set these environment variables with your own values:
53+
1) FOUNDRY_PROJECT_ENDPOINT - Required. The Azure AI Project endpoint, as found in the overview page of your
54+
Microsoft Foundry project. It has the form: https://<account_name>.services.ai.azure.com/api/projects/<project_name>.
55+
2) AZURE_SUBSCRIPTION_ID - Required. The Azure subscription ID containing your project.
56+
3) AZURE_RESOURCE_GROUP - Required. The resource group containing your Azure AI account.
57+
4) ENDPOINT_URL - Required. The URL of your scoring endpoint
58+
(e.g., https://my-scoring-endpoint.azurewebsites.net/api/evaluate).
59+
5) ENDPOINT_API_KEY - Required. The API key for authenticating with your scoring endpoint.
60+
61+
PREREQUISITES:
62+
- A scoring endpoint that:
63+
* Accepts POST requests with a JSON body containing evaluation data
64+
* Authenticates requests using an API key (via the ``api-key`` header)
65+
* Returns a JSON response with label/score and optional reasoning fields
66+
"""
67+
68+
import os
69+
import time
70+
from urllib.parse import urlparse
71+
72+
from azure.identity import DefaultAzureCredential
73+
from azure.ai.projects import AIProjectClient
74+
from azure.ai.projects.models import (
75+
EndpointBasedEvaluatorDefinition,
76+
EvaluatorCategory,
77+
EvaluatorType,
78+
EvaluatorVersion,
79+
EvaluatorMetric,
80+
)
81+
from azure.mgmt.cognitiveservices import CognitiveServicesManagementClient
82+
from azure.mgmt.cognitiveservices.models import (
83+
ConnectionPropertiesV2BasicResource,
84+
ApiKeyAuthConnectionProperties,
85+
ConnectionApiKey,
86+
)
87+
88+
from openai.types.evals.create_eval_jsonl_run_data_source_param import (
89+
CreateEvalJSONLRunDataSourceParam,
90+
SourceFileContent,
91+
SourceFileContentContent,
92+
)
93+
from openai.types.eval_create_params import DataSourceConfigCustom
94+
95+
from pprint import pprint
96+
from dotenv import load_dotenv
97+
98+
load_dotenv()
99+
100+
endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
101+
subscription_id = os.environ["AZURE_SUBSCRIPTION_ID"]
102+
resource_group = os.environ["AZURE_RESOURCE_GROUP"]
103+
endpoint_url = os.environ["ENDPOINT_URL"]
104+
endpoint_api_key = os.environ["ENDPOINT_API_KEY"]
105+
106+
# Derive account name from the project endpoint URL
107+
# e.g., https://accountname.services.ai.azure.com/api/projects/default -> "accountname"
108+
hostname = urlparse(endpoint).hostname
109+
if not hostname:
110+
raise ValueError(f"Could not parse hostname from endpoint: {endpoint}")
111+
account_name = hostname.split(".")[0]
112+
connection_name = "my-endpoint-apikey-connection"
113+
114+
with (
115+
DefaultAzureCredential() as credential,
116+
AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
117+
project_client.get_openai_client() as client,
118+
):
119+
120+
# ── Step 1: Create a workspace connection with API Key auth ─────────
121+
# The connection stores the endpoint URL and API key credentials. At evaluation
122+
# time, the service resolves the connection and sends the API key in the request
123+
# header to your endpoint.
124+
# Note: create is idempotent — it creates or updates the connection if it already exists.
125+
print("[1/5] Creating workspace connection with API Key auth...")
126+
127+
mgmt_client = CognitiveServicesManagementClient(
128+
credential=credential,
129+
subscription_id=subscription_id,
130+
)
131+
connection = ConnectionPropertiesV2BasicResource(
132+
properties=ApiKeyAuthConnectionProperties(
133+
category="ApiKey",
134+
target=endpoint_url,
135+
credentials=ConnectionApiKey(key=endpoint_api_key),
136+
),
137+
)
138+
mgmt_client.account_connections.create(
139+
resource_group_name=resource_group,
140+
account_name=account_name,
141+
connection_name=connection_name,
142+
connection=connection,
143+
)
144+
print(f" Connection created: {connection_name}")
145+
146+
# ── Step 2: Register an endpoint-based evaluator ────────────────────
147+
# The evaluator references the workspace connection created above. The service
148+
# resolves the endpoint URL and API key credentials at evaluation time.
149+
print("[2/5] Registering endpoint-based evaluator with API Key auth...")
150+
151+
evaluator = project_client.beta.evaluators.create_version(
152+
name="my-endpoint-evaluator-apikey",
153+
evaluator_version=EvaluatorVersion(
154+
categories=[EvaluatorCategory.QUALITY],
155+
evaluator_type=EvaluatorType.CUSTOM,
156+
definition=EndpointBasedEvaluatorDefinition(
157+
connection_name=connection_name,
158+
metrics={
159+
"score": EvaluatorMetric(
160+
type="ordinal",
161+
min_value=1,
162+
max_value=5,
163+
)
164+
},
165+
),
166+
display_name="Endpoint Evaluator (API Key)",
167+
description="Custom scoring endpoint authenticated with an API key",
168+
),
169+
)
170+
171+
print(f" Created evaluator: name={evaluator.name}, version={evaluator.version}")
172+
173+
# ── Step 3: Create an evaluation ────────────────────────────────────
174+
print("[3/5] Creating evaluation...")
175+
176+
data_source_config = DataSourceConfigCustom(
177+
{
178+
"type": "custom",
179+
"item_schema": {
180+
"type": "object",
181+
"properties": {
182+
"query": {"type": "string"},
183+
"response": {"type": "string"},
184+
"context": {"type": "string"},
185+
},
186+
"required": ["query", "response"],
187+
},
188+
"include_sample_schema": True,
189+
}
190+
)
191+
192+
testing_criteria = [
193+
{
194+
"type": "azure_ai_evaluator",
195+
"name": "endpoint_eval_apikey",
196+
"evaluator_name": "my-endpoint-evaluator-apikey",
197+
"data_mapping": {
198+
"query": "{{item.query}}",
199+
"response": "{{item.response}}",
200+
"context": "{{item.context}}",
201+
},
202+
}
203+
]
204+
205+
eval_object = client.evals.create(
206+
name="endpoint-evaluator-apikey-test",
207+
data_source_config=data_source_config,
208+
testing_criteria=testing_criteria, # type: ignore
209+
)
210+
211+
print(f" Evaluation created: id={eval_object.id}")
212+
213+
# ── Step 4: Run the evaluation with inline data ─────────────────────
214+
print("[4/5] Running evaluation with inline data...")
215+
216+
eval_run = client.evals.runs.create(
217+
eval_id=eval_object.id,
218+
name="endpoint-apikey-run",
219+
data_source=CreateEvalJSONLRunDataSourceParam(
220+
type="jsonl",
221+
source=SourceFileContent(
222+
type="file_content",
223+
content=[
224+
SourceFileContentContent(
225+
item={
226+
"query": "What is machine learning?",
227+
"response": "Machine learning is a subset of AI that enables systems to learn from data.",
228+
"context": "AI and ML overview",
229+
}
230+
),
231+
SourceFileContentContent(
232+
item={
233+
"query": "What is the capital of France?",
234+
"response": "The capital of France is Paris.",
235+
"context": "Geography question about European capitals",
236+
}
237+
),
238+
SourceFileContentContent(
239+
item={
240+
"query": "Explain quantum computing",
241+
"response": "Quantum computing leverages quantum mechanical phenomena like superposition and entanglement to process information.",
242+
"context": "Complex scientific concept explanation",
243+
}
244+
),
245+
SourceFileContentContent(
246+
item={
247+
"query": "What are some tips for staying healthy?",
248+
"response": "To stay healthy, focus on regular exercise, a balanced diet, adequate sleep, and stress management.",
249+
"context": "Health and wellness advice",
250+
}
251+
),
252+
],
253+
),
254+
),
255+
)
256+
257+
print(f" Eval run created: id={eval_run.id}")
258+
259+
# ── Step 5: Poll for results ────────────────────────────────────────
260+
print("[5/5] Waiting for evaluation run to complete...")
261+
262+
while True:
263+
run = client.evals.runs.retrieve(run_id=eval_run.id, eval_id=eval_object.id)
264+
if run.status in ("completed", "failed"):
265+
print(f" Run status: {run.status}")
266+
output_items = list(client.evals.runs.output_items.list(run_id=run.id, eval_id=eval_object.id))
267+
pprint(output_items)
268+
print(f" Report URL: {run.report_url}")
269+
break
270+
time.sleep(5)
271+
print(f" Status: {run.status} — polling again...")
272+
273+
# ── Cleanup ────────────────────────────────────────────────────────
274+
# The connection, evaluator, and evaluation are left in place so you can
275+
# inspect results in the Foundry portal. Uncomment the following to delete them:
276+
# mgmt_client.account_connections.delete(
277+
# resource_group_name=resource_group,
278+
# account_name=account_name,
279+
# connection_name=connection_name,
280+
# )
281+
# print(f" Connection deleted: {connection_name}")
282+
# project_client.beta.evaluators.delete(name="my-endpoint-evaluator-apikey")
283+
# print(" Evaluator deleted: my-endpoint-evaluator-apikey")
284+
# client.evals.delete(eval_id=eval_object.id)
285+
# print(f" Evaluation deleted: {eval_object.id}")

0 commit comments

Comments
 (0)