Add evaluation test case for adding optional body to ARM action#15282
Conversation
3719c84 to
2bc964f
Compare
Add new test case covering how to add an optional body parameter to an existing ARM TypeSpec action operation in a new API version. Includes evaluation results in typespec-test.json. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2bc964f to
9d03fce
Compare
|
Create this PR as a draft that capture the current step. this would be updated when the new agent framework get merged. |
|
Hi @lirenhe. Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days. |
github-actions
Bot
added
the
no-recent-activity
|
do not close this item. It needs to be migrated to the new framework. |
github-actions
Bot
removed
the
no-recent-activity
|
@copilot resolve the merge conflicts in this pull request |
Co-authored-by: lirenhe <9100546+lirenhe@users.noreply.github.com>
Resolved the merge conflicts in commit 0776952. The optional ARM action body testcase is now migrated into |
Rerun the case added in PR #15282 against the current agentic bot and refresh its results/typespec-test.json entry: similarity 5, groundedness 5, response_completeness 4 (all pass), overall_score 6.6, with real grader reasons. Previously the case graded N/A under the old infrastructure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 49e51257-81dd-47bf-aada-544a4b70416f
|
Azure Pipelines: 65 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
Adds a new TypeSpec Q&A evaluation testcase covering how to evolve an ARM action by introducing an optional request body in a new API version (via OptionalRequestBody = true), and updates the checked-in evaluation results to include the new testcase.
Changes:
- Added a new testcase entry to the TypeSpec evaluation dataset (
typespec.jsonl). - Added the corresponding testcase result block to
typespec-test.jsonand incrementedtotal_evals.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| tools/sdk-ai-bots/azure-sdk-qa-bot-evaluation/evaluation_datasets/basic/typespec.jsonl | Adds the new “optional body on existing ARM action” testcase to the curated TypeSpec dataset. |
| tools/sdk-ai-bots/azure-sdk-qa-bot-evaluation/results/typespec-test.json | Records evaluation output for the new testcase and updates aggregate counts. |
|
/check-enforcer override |
Add new Q&A evaluation test case to \ ypespec_core.jsonl\ covering how to add an optional body parameter to an existing ARM TypeSpec action operation in a new API version.
Changes:
esults/typespec-test.json: Evaluation results for the new test case
Test case summary:
Note: This test case does not work well in the current infrastructure but it works well in the new agentic framework.