eval: set USER_ERROR blame when AOAI run fails with UserError code#46746
eval: set USER_ERROR blame when AOAI run fails with UserError code#46746posaninagendra merged 5 commits intomainfrom
Conversation
…s + unit tests Agent-Logs-Url: https://github.com/Azure/azure-sdk-for-python/sessions/85522f11-e665-4eb7-b1bc-05e48272e704 Co-authored-by: posaninagendra <2713981+posaninagendra@users.noreply.github.com>
…aluate_aoai Agent-Logs-Url: https://github.com/Azure/azure-sdk-for-python/sessions/f4a706f3-5992-4bba-8ca8-21d3af1b16cb Co-authored-by: posaninagendra <2713981+posaninagendra@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR improves error attribution for Azure OpenAI (AOAI) evaluation runs by setting EvaluationException.blame to USER_ERROR when the run failure’s error.code indicates a user-caused issue (case-insensitive "UserError"), instead of always reporting UNKNOWN.
Changes:
- Update
_get_single_run_resultsto deriveblamefromrun_results.error.code(nestedgetattr), mapping"usererror"→ErrorBlame.USER_ERROR, otherwiseErrorBlame.UNKNOWN. - Add unit tests validating blame selection across casing variations and missing/partial
errorstructures.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluate/_evaluate_aoai.py | Sets exception blame based on run_results.error.code when run status is non-completed. |
| sdk/evaluation/azure-ai-evaluation/tests/unittests/test_evaluate_aoai.py | Adds unit tests to ensure blame is USER_ERROR only for "UserError" code and UNKNOWN otherwise. |
…tion + changelog Agent-Logs-Url: https://github.com/Azure/azure-sdk-for-python/sessions/e699106e-c9a8-428c-ac53-4d7a6729bb71 Co-authored-by: posaninagendra <2713981+posaninagendra@users.noreply.github.com>
|
Hi @copilot. We've noticed that new changes have been pushed to this pull request. Because it is set to automatically merge, we've reset the approvals to allow the opportunity to review the updates. |
github-actions
Bot
dismissed stale reviews from posaninagendra, slister1001, and slister1001
Hi @posaninagendra. We've noticed that new changes have been pushed to this pull request. Because it is set to automatically merge, we've reset the approvals to allow the opportunity to review the updates.
|
Hi @copilot. We've noticed that new changes have been pushed to this pull request. Because it is set to automatically merge, we've reset the approvals to allow the opportunity to review the updates. |
github-actions
Bot
dismissed
slister1001’s stale review
Hi @slister1001. We've noticed that new changes have been pushed to this pull request. Because it is set to automatically merge, we've reset the approvals to allow the opportunity to review the updates.
_get_single_run_results