feature: retire azure-hosted-copilot-sdk skill

[JasonYeMSFT]

Description

Remove azure-hosted-copilot-sdk skill. This will allow related prompts to be routed to microsoft-foundry skill or handled directly by the LLM, which now has better knowledge of Copilot SDK with the help of web search.

I also removed the disambiguation clause check. We have decided that we should avoid using DO NOT USE FOR clauses whenever possible. We shouldn't have a check to make it harder for us to remove the such remaining clauses. The source of truth of the skill routing accuracy should be the integration test results.

Checklist

• Tests pass locally (cd tests && npm test)
• If modifying skill descriptions: verified routing correctness with integration tests (npm run test:skills:integration -- <skill>)
• If modifying skill USE FOR / DO NOT USE FOR / PREFER OVER clauses: confirmed no routing regressions for competing skills

Related Issues

[Copilot]

Pull request overview

This PR retires the azure-hosted-copilot-sdk skill and removes its associated tests, eval suite, and spec documentation, aiming to let Copilot SDK-related prompts be handled by other skills (notably microsoft-foundry) or directly by the base model.

Changes:

• Removed plugin/skills/azure-hosted-copilot-sdk/ (SKILL.md, references, version config) and removed the skill from CODEOWNERS and test scheduling.
• Deleted all tests/azure-hosted-copilot-sdk/ trigger/integration tests and resources, plus removed the Vally eval spec under evals/azure-hosted-copilot-sdk/.
• Updated azure-prepare routing docs to remove references to the retired skill; updated microsoft-foundry trigger snapshots.

Show a summary per file

File	Description
tests/skills.json	Removed azure-hosted-copilot-sdk from the tested skill set and the integration schedule.
tests/microsoft-foundry/resource/create/snapshots/triggers.test.ts.snap	Updated microsoft-foundry trigger snapshot expectations (description/keywords).
tests/microsoft-foundry/models/deploy/deploy-model/snapshots/triggers.test.ts.snap	Updated microsoft-foundry trigger snapshot expectations (description/keywords).
tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/snapshots/triggers.test.ts.snap	Updated microsoft-foundry trigger snapshot expectations (description/keywords).
tests/microsoft-foundry/models/deploy/customize-deployment/snapshots/triggers.test.ts.snap	Updated microsoft-foundry trigger snapshot expectations (description/keywords).
tests/microsoft-foundry/models/deploy/capacity/snapshots/triggers.test.ts.snap	Updated microsoft-foundry trigger snapshot expectations (description/keywords).
tests/microsoft-foundry/foundry-agent/troubleshoot/snapshots/triggers.test.ts.snap	Updated microsoft-foundry trigger snapshot expectations (description/keywords).
tests/microsoft-foundry/foundry-agent/trace/snapshots/triggers.test.ts.snap	Updated microsoft-foundry trigger snapshot expectations (description/keywords).
tests/microsoft-foundry/foundry-agent/observe/snapshots/triggers.test.ts.snap	Updated microsoft-foundry trigger snapshot expectations (description/keywords).
tests/microsoft-foundry/foundry-agent/invoke/snapshots/triggers.test.ts.snap	Updated microsoft-foundry trigger snapshot expectations (description/keywords).
tests/microsoft-foundry/foundry-agent/eval-datasets/snapshots/triggers.test.ts.snap	Updated microsoft-foundry trigger snapshot expectations (description/keywords).
tests/microsoft-foundry/foundry-agent/deploy/snapshots/triggers.test.ts.snap	Updated microsoft-foundry trigger snapshot expectations (description/keywords).
tests/microsoft-foundry/foundry-agent/create/snapshots/triggers.test.ts.snap	Updated microsoft-foundry trigger snapshot expectations (description/keywords).
tests/microsoft-foundry/finetuning/snapshots/triggers.test.ts.snap	Updated microsoft-foundry trigger snapshot expectations (description/keywords).
tests/microsoft-foundry/snapshots/triggers.test.ts.snap	Updated microsoft-foundry trigger snapshot expectations (description/keywords).
tests/azure-hosted-copilot-sdk/util.ts	Deleted integration-test utilities specific to the retired skill.
tests/azure-hosted-copilot-sdk/triggers.test.ts	Deleted trigger tests for the retired skill.
tests/azure-hosted-copilot-sdk/resources/express-app-tsconfig.json	Deleted retired-skill test resource fixture.
tests/azure-hosted-copilot-sdk/resources/express-app-server.ts.txt	Deleted retired-skill test resource fixture.
tests/azure-hosted-copilot-sdk/resources/express-app-package.json	Deleted retired-skill test resource fixture.
tests/azure-hosted-copilot-sdk/resources/copilot-sdk-app-server.ts.txt	Deleted retired-skill test resource fixture.
tests/azure-hosted-copilot-sdk/resources/copilot-sdk-app-package.json	Deleted retired-skill test resource fixture.
tests/azure-hosted-copilot-sdk/integration.test.ts	Deleted integration tests for the retired skill.
tests/azure-hosted-copilot-sdk/snapshots/triggers.test.ts.snap	Deleted trigger snapshots for the retired skill.
plugin/skills/azure-prepare/SKILL.md	Updated azure-prepare frontmatter/routing content to remove retired-skill references and redirect Copilot SDK markers.
plugin/skills/azure-prepare/references/specialized-routing.md	Removed prompt-based routing row for the retired skill.
plugin/skills/azure-prepare/references/scan.md	Removed specialized Copilot SDK detection that previously delegated to the retired skill.
plugin/skills/azure-prepare/references/research.md	Removed guidance to invoke the retired skill during research.
plugin/skills/azure-prepare/references/analyze.md	Removed guidance to invoke the retired skill during planning analysis.
plugin/skills/azure-hosted-copilot-sdk/version.json	Deleted the retired skill’s version config.
plugin/skills/azure-hosted-copilot-sdk/SKILL.md	Deleted the retired skill definition.
plugin/skills/azure-hosted-copilot-sdk/references/existing-project-integration.md	Deleted retired skill reference doc.
plugin/skills/azure-hosted-copilot-sdk/references/deploy-existing.md	Deleted retired skill reference doc.
plugin/skills/azure-hosted-copilot-sdk/references/copilot-sdk.md	Deleted retired skill reference doc.
plugin/skills/azure-hosted-copilot-sdk/references/azure-model-config.md	Deleted retired skill reference doc.
plugin/skills/azure-hosted-copilot-sdk/references/auth-best-practices.md	Deleted retired skill reference doc.
evals/README.md	Replaced local run instructions with a pointer to the repo’s vally-eval skill doc.
evals/azure-hosted-copilot-sdk/eval.yaml	Deleted the retired skill’s Vally eval suite.
evals/_base/common-graders.yaml	Deleted shared grader reference doc used during prior migrations.
docs/spec/copilot-sdk-skill-comparison.md	Deleted spec doc comparing Copilot SDK skills.
docs/spec/azure-hosted-copilot-sdk.md	Deleted the retired skill’s spec/status doc.
.github/skills/skill-authoring/references/guidelines/frontmatter.md	Generalized wording to remove the retired skill from examples.
.github/skills/sensei/SKILL.md	Updated Sensei guidance examples to remove the retired skill mention.
.github/skills/sensei/references/SCORING.md	Updated scoring guidance examples to remove the retired skill mention.
.github/CODEOWNERS	Removed the retired skill path ownership entry.

Copilot's findings

• Files reviewed: 45/45 changed files
• Comments generated: 3

[wbreza]

Code Review

🤖 Automated review by Copilot CLI against commit 884b38e. Findings ranked by severity; reviewed by 2 models (consensus tagged where both agreed).

TL;DR: Skill retirement is mostly clean, but two real concerns: (1) a snapshot test will fail in CI, and (2) the "redirect to microsoft-foundry" claim is semantically wrong — microsoft-foundry covers Foundry-hosted agents, not GitHub Copilot Extension SDK apps. The PR description says these prompts should fall to the LLM, but the actual code in �zure-prepare actively redirects them to microsoft-foundry.

🔴 Critical

C1 — ests/azure-prepare/snapshots/triggers.test.ts.snap (lines 5, 13, 21, 75, 83) — likely CI failure
Snapshot still hardcodes �zure-hosted-copilot-sdk / copilot-sdk expectations. PR modifies plugin/skills/azure-prepare/* but does not update this snapshot — riggers.test.ts will fail. Fix: regenerate the snapshot (
pm test -- -u in ests/) and commit.

C2 — plugin/skills/azure-prepare/SKILL.md (line 3 frontmatter + codebase-marker table) — wrong redirect destination (✓ consensus)
The DO NOT USE FOR: copilot-sdk apps (use microsoft-foundry) clause and the four codebase-marker rows (@github/copilot-sdk / CopilotClient / createSession / sendAndWait) redirect to microsoft-foundry. But microsoft-foundry covers Foundry-hosted agents (Python/C# containers in AI Studio, ACR push, �gent_create) — not GitHub Copilot Extension SDK apps. The PR description says these prompts should "fall to the LLM (web search)", but the code actively redirects them elsewhere. Fix: either drop the four rows entirely (let them fall to LLM) or annotate them as (LLM handles directly). Remove (use microsoft-foundry) from the description.

C3 — plugin/skills/azure-prepare/references/scan.md vs SKILL.md — internal contradiction
scan.md's "Specialized SDK Detection — Check FIRST" block was deleted, but SKILL.md's codebase-marker table still references the same @github/copilot-sdk / CopilotClient markers. The two docs now contradict each other. Fix: align them — either restore minimal detection in scan.md with LLM-fallback instructions, or remove the marker rows from SKILL.md.

C4 — plugin/skills/azure-prepare/references/specialized-routing.md — stale rationale & broken Flow pointer
"Why This Step Exists" still cites Copilot SDK as the rationale for the specialized-routing step ("Some technologies (Copilot SDK) have dedicated skills with…"), and the Flow diagram points to a scan.md SDK-detection block that was deleted in this PR. Fix: rewrite both sections to reflect post-retirement reality.

🟠 High

H1 — microsoft-foundry does not absorb the retired skill's use cases (✓ consensus)
microsoft-foundry/SKILL.md and its sub-skills contain zero guidance for: �zd init --template azure-samples/copilot-sdk-service, CopilotClient / createSession / sendAndWait patterns, �earerToken + DefaultAzureCredential BYOM auth, OAuth token server scaffold, or integrating Copilot SDK into existing Express/Node apps. The only tangential match is oundry-agent/create/references/toolbox.md:60 — a sample link, not coverage. Routing copilot-sdk apps to microsoft-foundry will produce wrong specialized guidance rather than helpful LLM fallback. Fix: see C2 — route to LLM explicitly, or add a bridging note in microsoft-foundry pointing at the �zure-samples/copilot-sdk-service template.

H2 — ests/microsoft-foundry//snapshots/triggers.test.ts.snap — silent unrelated routing change**
"agent optimizer" was removed from microsoft-foundry's USE FOR keywords with no rationale or test. This is unrelated to the retirement and is a silent regression for any prompt containing that phrase. Fix: either revert this change or extract it into a separate PR with explicit routing validation.

🟡 Medium

• M1 — ests/README.md:368: coverage matrix still lists deleted �zure-hosted-copilot-sdk.
• M2 — ests/azure-quotas/unit.test.ts:104: comment still points to deleted ests/azure-hosted-copilot-sdk/unit.test.ts.
• M3 — �vals/README.md (−49/+1): 48 lines of eval-runner runbook docs collapsed to a single redirect to �ally-eval SKILL. Unrelated to retirement; should be its own PR. Developers looking under �vals/ will lose discoverability of
  px @microsoft/vally eval --suite pr and related commands.
• M4 — PR checklist: all three items unchecked (tests pass / routing verified / no regressions). Given the routing-table edits in 5 files, lack of explicit test evidence is notable — especially in light of C1.

🟢 Verified Clean

• .github/CODEOWNERS, ests/skills.json (slot removal + schedule re-arrangement), sensei/skill-authoring example cleanups — all correct
• �vals/_base/common-graders.yaml deletion: no remaining consumers
• No orphan refs in plugin manifests (.plugin, .cursor-plugin, .claude-plugin), gulpfile.ts, .token-limits.json, .vally.yaml
• Deleted spec files (docs/spec/azure-hosted-copilot-sdk.md, docs/spec/copilot-sdk-skill-comparison.md): no inbound links

[wbreza]

Code Review — Re-pass at d9a3200

🤖 Automated re-review by Copilot CLI against commits 44ea1a0, 9381e67, d9a3200. Comparing against prior review at commit 884b38e.

TL;DR: Strong progress — 5/10 prior findings fixed (C1, H2, M1, M2 fully; C2 mostly). Three prior findings remain unaddressed (C4, M3, M4) and two new concerns surfaced in the last commits.

Status of Prior Findings

ID	Status	Notes
🔴 C1 — snapshot CI failure	✅ FIXED	ests/azure-prepare/snapshots/triggers.test.ts.snap regenerated cleanly
🔴 C2 — wrong microsoft-foundry redirect	⚠️ MOSTLY FIXED	Frontmatter and codebase-marker rows cleaned; one residual at SKILL.md:89 — see N1 below
🔴 C3 — scan.md vs SKILL.md contradiction	⚠️ MOSTLY FIXED	Same residual at SKILL.md:89
🔴 C4 — specialized-routing.md stale rationale	❌ NOT FIXED	Lines 19-24 still cite "Some technologies (Copilot SDK) have dedicated skills"; Flow at lines 36-37, 43 still points to a scan.md SDK-detection block that no longer exists
🟠 H1 — microsoft-foundry coverage gap	⚠️ PARTIAL	Redirect removed (good); no bridging note added (acceptable if intent is LLM fallback, but worth documenting in PR description)
🟠 H2 — silent "agent optimizer" removal	✅ FIXED	Keyword retained in microsoft-foundry description and snapshots
🟡 M1 — ests/README.md stale coverage matrix	✅ FIXED
🟡 M2 — ests/azure-quotas/unit.test.ts stale comment	✅ FIXED
🟡 M3 — �vals/README.md collapsed to single redirect	❌ NOT ADDRESSED	Still −49/+1 — unrelated to retirement; recommend separating into its own PR or restoring the runbook content
�� M4 — PR checklist all unchecked	❌ NOT ADDRESSED	All three items still [ ]

New Concerns

🟠 HIGH — N1 — plugin/skills/azure-prepare/SKILL.md:89 orphan reference
Still mentions @github/copilot-sdk and instructs the LLM to "invoke that skill first" — but the concrete routing rows above were removed in this PR, leaving no target skill. Fix: either remove this line, or rewrite to clarify "no specialized skill exists; LLM handles directly with web search."

🟠 HIGH — N2 — Frontmatter validator weakening (commit 9381e67)
scripts/src/frontmatter/cli.ts (hunks @@ -465,51 +457,6 and @@ -790,14 +697,12) and scripts/src/frontmatter/tests/frontmatter.test.ts (hunk @@ -583,39 +581,6) remove the validator that blocked silent deletion of DO NOT USE FOR / PREFER OVER disambiguation clauses, along with its tests. This is a tooling regression that disables a safeguard against the exact failure mode flagged as H2 in the prior review (silent removal of routing-protection keywords). In a 24+ skill set, disambiguation clauses are load-bearing for routing correctness. The commit message frames it as cleanup, but consider whether the check should be relaxed (e.g., warn-only) rather than deleted outright — or whether removing the check should be its own PR with explicit rationale.

Recommendations

1. Address before merge: N1 (orphan ref), C4 (specialized-routing.md cleanup)
2. Justify or revert: N2 (frontmatter validator removal)
3. Recommend splitting: M3 (evals/README.md change is unrelated to skill retirement)
4. Optional polish: check the PR checklist boxes (M4); consider a short note in the PR description confirming the intentional LLM-fallback strategy for Copilot SDK prompts (closes H1)

[anchenyi]

The Foundry skill description already includes content to route prompts like "Build a Copilot SDK-based agent on Foundry", so I don’t think any changes to the Foundry skill are needed.

[jongio]

One question: evals/_base/common-graders.yaml is described as "shared patterns across eval suites" and documents grader patterns used by other eval suites (azure-prepare, azure-deploy, azure-enterprise-infra-planner), not just the retired skill. Was deleting it intentional, or should it stay as a reference for other eval specs?

The eval CI failure is unrelated (azure-compute lint check).