Clean up retired Benchmarks references and the waza eval pipeline by helen229 · Pull Request #16431 · Azure/azure-sdk-tools

helen229 · 2026-07-21T16:58:02Z

Summary

Partial cleanup for #16422 (items 1, 3, and 4). Item 2 (delete the Gen-1 Azure.Sdk.Tools.Cli.Evaluations harness) is explicitly out of scope for this PR -- see "Not done" below for why.

Done

1. Gen-1.5 Benchmarks leftover references

tools/azsdk-cli/ci.yml -- removed the deprecated RunBenchmarks no-op parameter. Confirmed via repo-wide search that no other pipeline definition sets RunBenchmarks=true.
tools/azsdk-cli/Azure.Sdk.Tools.Cli/CONTRIBUTING.md -- removed the broken Azure.Sdk.Tools.Cli.Benchmarks/ table row (the project itself was already deleted in Remove Azure.Sdk.Tools.Cli.Benchmarks (superseded by Vally skill evals) #15697).
Left the historical mention in azsdk-evals/README.md ("This project supersedes the deleted Azure.Sdk.Tools.Cli.Benchmarks project (removed in Remove Azure.Sdk.Tools.Cli.Benchmarks (superseded by Vally skill evals) #15697)") as-is -- it's accurate provenance, not a stale/broken reference.

3. Old waza-based skill eval pipeline

Deleted plugins/azure-sdk-tools/skills/.waza.yaml.
Updated plugins/azure-sdk-tools/skills/README.md: removed the eval.yaml/tasks/ waza-era entries from the Skill Anatomy tree, replaced the waza "Tooling"/"Testing Skills" section with a pointer to the canonical .github/skills/ Vally process, and updated "Further Reading" to link Vally docs instead of waza docs.
.github/skills/skill-authoring/SKILL.md -- replaced stale waza check/waza CLI mentions with vally lint/vally CLI (matching what this repo's skill-eval pipeline actually runs).
.github/skills/markdown-token-optimizer/SKILL.md -- removed an inaccurate INVOKES: waza CLI clause; this skill's own "MCP Tools: None" / "CLI fallback" sections confirm it invokes no external tool at all.
Both edited SKILL.md files re-validated with vally lint --strict (3/3 checks passed each).

Left alone (explicitly out of scope): the individual skill copies under plugins/azure-sdk-tools/skills/*/ (e.g. sensei/tasks/sensei-basic-001.yaml, markdown-token-optimizer/SKILL.md, skill-authoring/SKILL.md) still contain eval.yaml/tasks/ files and stale waza mentions of their own. These are a separate plugin-marketplace distribution snapshot, not something #16422 called out by name -- flagging for whoever owns that sync process.

Not touched (confirmed out of scope for #3): the ADO pipeline definition 8165 retirement is a DevOps-project-level action, not something a code PR can do -- left for manual follow-up.

4. Current Vally lint/eval pipeline -- audited, not changed

Confirmed .github/workflows/skill-eval.yml + eng/skill-eval (vally lint) remains the single canonical skill-eval pipeline; no other doc points at a different one after the item-3 cleanup above.

Not done: item 2 (Gen-1 `Azure.Sdk.Tools.Cli.Evaluations` harness)

The issue calls this out as blocked pending confirmation, and investigating turned up a live dependency that makes blind deletion unsafe:

Azure.Sdk.Tools.Cli.Tests.csproj links ../Azure.Sdk.Tools.Cli.Evaluations/TestData/TestPrompts.json directly into the main test project.
Azure.Sdk.Tools.Cli.Tests/Tools/Core/ToolPromptCoverageTests.cs's AllToolsHaveTestPrompts test (a real, currently-passing PR-gate check, no MCP/OpenAI required) reads that same file to assert every MCP tool has at least one registered test prompt. This is a static completeness check, a different concern from the Gen-1 LLM-judged evaluation itself.
tools/azsdk-cli/ci.yml still wires ai-eval-job.yml into the internal-only CustomJobs block (${{ if eq(variables['System.TeamProject'], 'internal') }}), so the Gen-1 harness is still actively invoked by CI today, not dead code.

Deleting the project now would break the main Azure.Sdk.Tools.Cli.Tests build (missing linked file) and remove a still-valuable coverage check. Full deletion needs a follow-up that first migrates TestData/TestPrompts.json + the coverage test out of Azure.Sdk.Tools.Cli.Evaluations, then removes the ci.yml wiring and eng/common/pipelines/ai-evals-tests.yml / ai-eval-job.yml. Left eng/common/pipelines/ai-evals-tests.yml and Azure.Sdk.Tools.Cli.Evaluations/ untouched pending that.

Validation

Repo-wide search confirms no remaining RunBenchmarks callers.
Repo-wide search confirms no remaining waza mentions outside the explicitly-noted plugins/ skill copies.
vally lint .github/skills/skill-authoring --strict and vally lint .github/skills/markdown-token-optimizer --strict both pass 3/3.

github-actions · 2026-07-21T16:58:46Z

📊 GEPA Skill Quality Scores

Skill	Quality	Triggers	Tests
azsdk-common-pipeline-analysis	⚠️ 0.57	N/A	---
azsdk-common-pipeline-fixer	⚠️ 0.57	N/A	---
sensei	⚠️ 0.57	N/A	---
azsdk-common-prepare-release-plan	✅ 0.86	N/A	---
azsdk-common-apiview-feedback-resolution	✅ 1.00	N/A	---
azsdk-common-generate-sdk-locally	✅ 1.00	N/A	---
azsdk-common-generate-sdk-pipeline	✅ 1.00	N/A	---
azsdk-common-sdk-release	✅ 1.00	N/A	---
azure-typespec-author	✅ 1.00	N/A	---
markdown-token-optimizer	✅ 1.00	N/A	---
sdk-ai-bot-eval-dataset	✅ 1.00	N/A	---
sdk-ai-bot-run-evaluation	✅ 1.00	N/A	---
skill-authoring	✅ 1.00	N/A	---

10/13 skills at quality ≥ 0.80

How to improve

# Score a specific skill
python .github/skills/sensei/scripts/gepa/auto_evaluator.py score --skill <name> --skills-dir .github/skills --tests-dir tests

# Optimize a skill with GEPA
python .github/skills/sensei/scripts/gepa/auto_evaluator.py optimize --skill <name> --skills-dir .github/skills --tests-dir tests

azure-pipelines · 2026-07-21T16:58:51Z

Azure Pipelines: Successfully started running 1 pipeline(s). 64 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

Copilot

Pull request overview

Cleans up references to retired evaluation/benchmarking infrastructure in the azsdk-cli and skills plugin docs, aligning documentation and pipeline configuration with the current Vally-based skill evaluation workflow.

Changes:

Removed the deprecated RunBenchmarks parameter from tools/azsdk-cli/ci.yml.
Removed the deleted Benchmarks project from the azsdk-cli CONTRIBUTING repo layout table.
Retired the legacy waza project config (.waza.yaml) and updated skills docs/SKILL metadata to reference Vally instead of waza.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tools/azsdk-cli/ci.yml	Removes the deprecated no-op `RunBenchmarks` pipeline parameter.
tools/azsdk-cli/Azure.Sdk.Tools.Cli/CONTRIBUTING.md	Drops the stale Benchmarks directory reference from the repo layout table.
plugins/azure-sdk-tools/skills/README.md	Updates skill anatomy/tooling docs to reflect Vally and removes waza-era entries.
plugins/azure-sdk-tools/skills/.waza.yaml	Deletes the retired waza project configuration.
.github/skills/skill-authoring/SKILL.md	Replaces waza CLI references with Vally lint references in the skill definition text.
.github/skills/markdown-token-optimizer/SKILL.md	Removes an inaccurate “INVOKES: waza CLI” clause.

samvaity · 2026-07-21T17:52:21Z

 ---
 name: markdown-token-optimizer
-description: "Analyze markdown files for token efficiency and reduce context-window bloat. **UTILITY SKILL**. DO NOT USE FOR: code optimization, general file editing, non-markdown files. TRIGGERS: optimize markdown, reduce tokens, token count, token bloat, too many tokens, make concise, shrink file, file too large, optimize for AI, token efficiency, verbose markdown, reduce file size. INVOKES: waza CLI."
+description: "Analyze markdown files for token efficiency and reduce context-window bloat. **UTILITY SKILL**. DO NOT USE FOR: code optimization, general file editing, non-markdown files. TRIGGERS: optimize markdown, reduce tokens, token count, token bloat, too many tokens, make concise, shrink file, file too large, optimize for AI, token efficiency, verbose markdown, reduce file size."


nit: here we completely dropped and not replaced with invoke vally like in skill author. Just noting, do we want that?

Intentional, not an oversight: markdown-token-optimizer never actually invoked any CLI tool for its core function - its "Count" step is a rough in-context heuristic (~4 chars = 1 token), not a shell-out. So there is no vally equivalent to swap in the way skill-authoring genuinely runs "vally lint --strict" for validation. Dropping the INVOKES clause here is more accurate than replacing it with something that would not actually be true.

praveenkuttappan · 2026-07-22T16:14:55Z

@helen229 Can you please remove Eval job and references to it from this pipeline?
https://github.com/Azure/azure-sdk-tools/pull/16431/changes#diff-8ddd9d7e88871e11f160e0a8827aa0eb1790e0ee68a0f74628caa3b730bd1668L76

…skills/README.md Both READMEs describe the same .gitignore entry; .github/skills/README.md already says 'eval output directories' (generic), this one still said 'waza output directories', which reads as stale now that the doc states waza has been retired. Match the wording already used in the other copy.

AllToolsHaveTestPrompts -> AllToolsHaveEvalCoverage: reads coverage from evals/tools/*.eval.yaml's tool-calls.required[].name instead of the Gen-1 harness's TestData/TestPrompts.json. Keeps the same reflection-based tool enumeration and the same ExemptTools list; only the coverage data source changes. This severs Azure.Sdk.Tools.Cli.Tests.csproj's linked-file dependency on Azure.Sdk.Tools.Cli.Evaluations/TestData/TestPrompts.json (the live dependency this PR's own investigation flagged as blocking that project's deletion) -- TestPrompts.json itself is untouched, still used by the Gen-1 harness's PromptToToolMatchEvaluator. Verified via manual cross-reference (could not run dotnet test in this sandbox - github.copilot.sdk's build target fails downloading a native binary from registry.npmjs.org, unrelated to this change): of the 6 tool names covered in TestPrompts.json but not yet in evals/tools/*.eval.yaml, 5 no longer exist anywhere in the CLI source (stale entries for renamed/removed tools that the old one-directional check never caught) and the 6th (azsdk_cleanup_ai_agents) is already in ExemptTools. No regression expected; please confirm with a full test run.

This reverts commit aef4f7f.

Clean up retired Benchmarks references and the waza eval pipeline

74bfde7

Copilot AI review requested due to automatic review settings July 21, 2026 16:58

helen229 requested a review from a team as a code owner July 21, 2026 16:58

github-actions Bot added the azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli label Jul 21, 2026

Copilot started reviewing on behalf of helen229 July 21, 2026 16:58 View session

Copilot AI reviewed Jul 21, 2026

View reviewed changes

Comment thread plugins/azure-sdk-tools/skills/README.md Outdated

helen229 mentioned this pull request Jul 21, 2026

Clean up all pre-Vally eval infrastructure: Benchmarks, Gen-1 harness, and old waza skill-eval pipeline #16422

Open

4 tasks

samvaity approved these changes Jul 21, 2026

View reviewed changes

helen229 added 4 commits July 22, 2026 21:15

Merge origin/main into feat/cleanup-pre-vally-eval-infra

d51b6f6

Revert "Integrate tool-prompt coverage check with the Vally eval suite"

ebc504b

This reverts commit aef4f7f.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Clean up retired Benchmarks references and the waza eval pipeline#16431

Clean up retired Benchmarks references and the waza eval pipeline#16431
helen229 wants to merge 5 commits into
mainfrom
feat/cleanup-pre-vally-eval-infra

helen229 commented Jul 21, 2026

Uh oh!

github-actions Bot commented Jul 21, 2026

Uh oh!

azure-pipelines Bot commented Jul 21, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

samvaity Jul 21, 2026

Uh oh!

helen229 Jul 23, 2026

Uh oh!

Uh oh!

praveenkuttappan commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

helen229 commented Jul 21, 2026

Summary

Done

1. Gen-1.5 Benchmarks leftover references

3. Old waza-based skill eval pipeline

4. Current Vally lint/eval pipeline -- audited, not changed

Not done: item 2 (Gen-1 Azure.Sdk.Tools.Cli.Evaluations harness)

Validation

Uh oh!

github-actions Bot commented Jul 21, 2026

📊 GEPA Skill Quality Scores

Uh oh!

azure-pipelines Bot commented Jul 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

samvaity Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

helen229 Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

praveenkuttappan commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Not done: item 2 (Gen-1 `Azure.Sdk.Tools.Cli.Evaluations` harness)