Skip to content

Commit c922d8f

Browse files
fix(evaluation): initialize sample_input in prompty format_llm_response (#47005)
Agent-Logs-Url: https://github.com/Azure/azure-sdk-for-python/sessions/40ee70eb-f614-4ef1-a0a5-472639f541a8 Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: posaninagendra <2713981+posaninagendra@users.noreply.github.com> Co-authored-by: Nagendra Posani <naposani@microsoft.com>
1 parent 45b72db commit c922d8f

3 files changed

Lines changed: 54 additions & 0 deletions

File tree

sdk/evaluation/azure-ai-evaluation/CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
- Fixed `_get_metric_result` prefix matching where shorter metric names (e.g., `xpia`) could match before longer, more-specific ones (e.g., `xpia_manipulated_content`). Now sorts by length descending for correct longest-prefix matching.
2828
- Fixed non-dict `_properties` values from evaluators causing downstream issues. Values that are not dicts are now logged and dropped gracefully.
2929
- Fixed filename length error in `_inline_image` by catching OSError/ValueError during local path resolution and fall back to returning a text chunk instead of throwing.
30+
- Fixed `format_llm_response` raising `UnboundLocalError` when `inputs` was not provided by ensuring `sample_input` is always initialized.
3031

3132
### Other Changes
3233

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_legacy/prompty/_utils.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -609,6 +609,7 @@ async def format_stream(llm_response: AsyncStream[ChatCompletionChunk]) -> Async
609609
)
610610
sample_output = json.dumps(sample_output_list)
611611
input_str = f"{json.dumps(inputs)}" if inputs else ""
612+
sample_input = ""
612613
if inputs and len(inputs) > 0:
613614
sample_input_json = []
614615
msg = ChatCompletionUserMessageParam(
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
# ---------------------------------------------------------
2+
# Copyright (c) Microsoft Corporation. All rights reserved.
3+
# ---------------------------------------------------------
4+
5+
import asyncio
6+
from types import SimpleNamespace
7+
8+
import pytest
9+
10+
from azure.ai.evaluation._legacy.prompty._utils import format_llm_response
11+
12+
13+
class _FakeResponse:
14+
def __init__(self):
15+
self.usage = SimpleNamespace(prompt_tokens=1, completion_tokens=2, total_tokens=3)
16+
self.choices = [
17+
SimpleNamespace(
18+
finish_reason="stop",
19+
message=SimpleNamespace(role="assistant", content="test-output"),
20+
)
21+
]
22+
self.model = "test-model"
23+
24+
def model_dump(self):
25+
return {
26+
"choices": [
27+
{
28+
"message": {
29+
"role": "assistant",
30+
"content": "test-output",
31+
}
32+
}
33+
]
34+
}
35+
36+
37+
@pytest.mark.unittest
38+
def test_format_llm_response_with_no_inputs_sets_empty_sample_input():
39+
response = _FakeResponse()
40+
41+
result = asyncio.run(
42+
format_llm_response(
43+
response=response,
44+
is_first_choice=True,
45+
response_format={"type": "text"},
46+
outputs=None,
47+
inputs=None,
48+
)
49+
)
50+
51+
assert result["llm_output"] == "test-output"
52+
assert result["sample_input"] == ""

0 commit comments

Comments
 (0)