From f72915d55d29894bc6c379f3d232be4e1528e231 Mon Sep 17 00:00:00 2001 From: Sharadhi Srikanth Date: Thu, 18 Jun 2026 12:54:38 +0530 Subject: [PATCH 01/14] feat: add azure-advisor skill Add the azure-advisor product-area router skill with a read-only review capability that uses available advisor_* MCP tools. Includes shared capability-routing and subscription-discovery references, trigger tests, and registers the skill in tests/skills.json. --- plugin/skills/azure-advisor/README.md | 54 ++++++++ plugin/skills/azure-advisor/SKILL.md | 62 +++++++++ .../references/capability-routing.md | 23 ++++ .../references/subscription-discovery.md | 18 +++ plugin/skills/azure-advisor/review/review.md | 118 ++++++++++++++++++ plugin/skills/azure-advisor/version.json | 6 + .../__snapshots__/triggers.test.ts.snap | 117 +++++++++++++++++ tests/azure-advisor/triggers.test.ts | 92 ++++++++++++++ tests/skills.json | 3 +- 9 files changed, 492 insertions(+), 1 deletion(-) create mode 100644 plugin/skills/azure-advisor/README.md create mode 100644 plugin/skills/azure-advisor/SKILL.md create mode 100644 plugin/skills/azure-advisor/references/capability-routing.md create mode 100644 plugin/skills/azure-advisor/references/subscription-discovery.md create mode 100644 plugin/skills/azure-advisor/review/review.md create mode 100644 plugin/skills/azure-advisor/version.json create mode 100644 tests/azure-advisor/__snapshots__/triggers.test.ts.snap create mode 100644 tests/azure-advisor/triggers.test.ts diff --git a/plugin/skills/azure-advisor/README.md b/plugin/skills/azure-advisor/README.md new file mode 100644 index 000000000..8a040c23a --- /dev/null +++ b/plugin/skills/azure-advisor/README.md @@ -0,0 +1,54 @@ +# Azure Advisor Skill — Contributor Guide + +> This README is for **contributors** maintaining the `azure-advisor` skill. It is not +> loaded by the agent at runtime — the agent reads [SKILL.md](SKILL.md) and the +> capability files. Keep onboarding notes here. + +## What this skill is + +`azure-advisor` is a **product area**, not a single workflow. It groups multiple +Advisor-related capabilities behind one router so the agent can pick the right one by +user intent. Each capability is a self-contained folder; shared concerns live in the +[references](references/capability-routing.md) folder. + +## Folder map + +| Path | Purpose | +|------|---------| +| [SKILL.md](SKILL.md) | **Router.** Frontmatter (makes the skill discoverable) + a capability table that routes intent to a capability file. Keep it thin. | +| [references/capability-routing.md](references/capability-routing.md) | **Shared, capability-agnostic docs** reused by every capability. Don't duplicate these inside a capability. | +| [references/capability-routing.md](references/capability-routing.md) | How to resolve an `advisor_*` MCP tool by capability (catalog, recommendations, summary, IaaC fix). | +| [references/subscription-discovery.md](references/subscription-discovery.md) | How to resolve the target subscription from repo config / env without hardcoding. | +| [review/review.md](review/review.md) | **Capability:** holistic read-only Advisor sweep. | +| [review/review.md](review/review.md) | The review workflow (the only file the agent reads for this capability). | + +## Conventions (match the rest of the repo) + +- **One folder per capability**, named after the capability (e.g. `review/`, + `summarize/`). This mirrors `microsoft-foundry` (`quota/quota.md`, `rbac/rbac.md`). +- **Capability file is named after its folder**: `review/review.md`, + `summarize/summarize.md`. The agent loads this file when the capability matches. +- **Shared docs go in `references/`**, never inside a single capability folder — so the + next capability can reuse them without a refactor. +- **Capability-specific references** (used by only one capability) may live in a nested + `references/` inside that capability folder, e.g. `review/references/…`. +- **Frontmatter lives only in `SKILL.md`** (`name`, `description`, `license`, + `metadata.author`, `metadata.version`). Capability files are plain Markdown. + +## How to add a new capability + +1. **Create the folder + file:** `azure-advisor//.md`. +2. **Reuse shared references** instead of re-defining them — link to + [capability-routing.md](references/capability-routing.md) and + [subscription-discovery.md](references/subscription-discovery.md) where relevant. +3. **Register it in the router:** add a row to the **Capabilities** table in + [SKILL.md](SKILL.md) (and remove it from the Roadmap list there if listed). +4. **Keep it read-only by default.** Advisor workflows propose IaaC fixes; they don't + mutate cloud state. Document any exception explicitly. +5. **Do not edit `metadata.version`** in [SKILL.md](SKILL.md) — it stays + `0.0.0-placeholder`; NBGV stamps the real version at build time. + +## Where this ships from + +This is the upstream `microsoft/GitHub-Copilot-for-Azure` repository. Land changes here; +they are released from this repo and mirrored downstream automatically. diff --git a/plugin/skills/azure-advisor/SKILL.md b/plugin/skills/azure-advisor/SKILL.md new file mode 100644 index 000000000..d1ca5d473 --- /dev/null +++ b/plugin/skills/azure-advisor/SKILL.md @@ -0,0 +1,62 @@ +--- +name: azure-advisor +description: "Azure Advisor reviews and recommendation workflows using whichever advisor_* MCP tools are available. WHEN: \"run an advisor review\", \"check my Azure advisor recommendations\", \"summarize advisor findings\", \"what does Advisor say about my subscription\", \"give me an advisor health check\", \"audit my Azure resources with Advisor\". USE FOR: end-to-end Advisor sweeps that combine the recommendation catalog, active recommendations, summaries/aggregates, and IaaC fix suggestions into one chat summary. DO NOT USE FOR: applying changes to Azure resources directly (read-only review), cost analysis beyond Advisor's cost category (use azure-cost), or non-Advisor diagnostics (use azure-diagnostics)." +license: MIT +metadata: + author: Microsoft + version: "0.0.0-placeholder" +--- + +# Azure Advisor Skill + +Azure Advisor is a **product area** with multiple capabilities. This skill routes a +user's intent to the right capability and runs it using whichever `advisor_*` MCP tools +the connected Azure MCP server exposes. Routing is by *capability* (catalog, +recommendations, summary, IaaC fix), not by hard-coded tool names, so the skill stays +useful as new advisor tools land. + +> 🛠️ **Contributing a new capability?** See [README.md](README.md) for the folder map +> and a step-by-step recipe. + +## Pre-Execution Requirements + +Inspect the available `advisor_*` MCP tools and their parameters before running a +capability. Match tools by capability description, not by a fixed name list — see the +shared [Capability Routing](references/capability-routing.md) reference. + +## Shared References + +These product-area references are reused by **every** capability below. Read the +relevant one before acting: + +| Reference | Purpose | +|-----------|---------| +| [Capability Routing](references/capability-routing.md) | Resolve which `advisor_*` MCP tool to call for each capability (catalog, recommendations, summary, IaaC fix). | +| [Subscription Discovery](references/subscription-discovery.md) | Resolve the target subscription from repo config / env without hardcoding. | + +## Capabilities + +Route the user's request to the matching capability. **Use these instead of the main +skill when they match the task:** + +| Capability | When to Use | Reference | +|-----------|-------------|-----------| +| **review** | Run a holistic, read-only Advisor sweep across a subscription — probe the catalog, pull active recommendations, aggregate by category/impact, spotlight high-impact items, and propose IaaC fix snippets. | [review](review/review.md) | + +### Roadmap (not yet implemented) + +Planned capabilities will each be added as a sibling folder under `azure-advisor/`, +reusing the shared references above. Add a row to the table above when one ships: + +- **summarize** — focused recommendation summarization / aggregation views +- **resource-analysis** — per-resource Advisor analysis and drill-down +- **greenfield** — Advisor-informed guidance for new/empty subscriptions +- **cost** — Advisor cost-category optimization (coordinates with `azure-cost`) +- **reliability** — Advisor reliability-category reviews +- **governance** — Advisor operational-excellence / governance reviews + +## Constraints + +- ❌ **Never** hardcode a subscription id, tenant id, or resource group. +- ❌ **Never** modify Azure state — Advisor workflows in this skill are read + suggest only. +- ❌ **Never** call non-`advisor_*` tools as substitutes; if no capability matches, report it and skip. diff --git a/plugin/skills/azure-advisor/references/capability-routing.md b/plugin/skills/azure-advisor/references/capability-routing.md new file mode 100644 index 000000000..d6f722005 --- /dev/null +++ b/plugin/skills/azure-advisor/references/capability-routing.md @@ -0,0 +1,23 @@ +# Shared Reference — Advisor Capability Routing + +> **Shared across all `azure-advisor` capabilities.** Any capability that needs to +> pick an `advisor_*` MCP tool should link here instead of re-defining the table. +> Match tools by *capability description*, never by hard-coded name. + +The connected Azure MCP server may expose a changing set of `advisor_*` tools. Resolve +which tool to invoke at each step from this capability table. + +| Capability | Look for an `advisor_*` tool whose description says... | Required input | +|---|---|---| +| **Metadata / catalog** | "list recommendation types / categories / impact levels / supported values" | tenant context (no subscription needed) | +| **Active recommendations** | "list Advisor recommendations in a subscription" | subscription (resource group optional, filters optional) | +| **Aggregation / summary** | "summarize / group / aggregate Advisor recommendations" | subscription + group-by field | +| **IaaC remediation** | "apply Advisor recommendations to IaaC / ARM / Bicep / Terraform" | a resource type identifier | + +## Resolution rules + +- If a step's capability has **no matching tool**, skip it and note that in the chat + output (e.g. "no aggregation tool available, presenting raw list"). +- If **multiple tools** match, prefer the one with the more specific description. +- **Never** substitute a non-`advisor_*` tool for a missing capability — report the gap + and skip instead. diff --git a/plugin/skills/azure-advisor/references/subscription-discovery.md b/plugin/skills/azure-advisor/references/subscription-discovery.md new file mode 100644 index 000000000..b6b914bce --- /dev/null +++ b/plugin/skills/azure-advisor/references/subscription-discovery.md @@ -0,0 +1,18 @@ +# Shared Reference — Subscription Discovery + +> **Shared across all `azure-advisor` capabilities.** Any capability that operates on a +> subscription should link here instead of re-defining discovery logic. + +Subscription **must never be hardcoded**. Resolve in this order and stop at the first hit: + +1. **Repository config files** — scan the workspace for any of: + - `azure.yaml` → `subscriptionId:` key + - `.azure/*/config.json` → `subscriptionId` field + - `*.bicepparam` / `*.parameters.json` → `subscriptionId` / `subscription` parameter + - `.env*` files → `AZURE_SUBSCRIPTION_ID` line + - `infra/**/main.parameters.json` +2. **Environment variable** `AZURE_SUBSCRIPTION_ID`. +3. **Ask the user.** Do not guess. Say which files were scanned and what was missing. + +Always mention the *source* of the resolved subscription in the final chat summary +(e.g. "Subscription pulled from `infra/main.parameters.json`"). diff --git a/plugin/skills/azure-advisor/review/review.md b/plugin/skills/azure-advisor/review/review.md new file mode 100644 index 000000000..8a0b69931 --- /dev/null +++ b/plugin/skills/azure-advisor/review/review.md @@ -0,0 +1,118 @@ +# Azure Advisor Review + +Drive an end-to-end Azure Advisor sweep using whichever `advisor_*` MCP tools the +connected Azure MCP server exposes. Designed to stay useful as new advisor tools +land — it routes by *capability* (catalog, recommendations, summary, IaaC fix), +not by hard-coded tool name lists. + +## When to Use This Sub-Skill + +Use this sub-skill when the user wants to: + +- Get an overall picture of Advisor findings for an Azure subscription +- See "what categories / impact levels does Advisor surface here?" before drilling in +- Combine the recommendation catalog (metadata) with the active recommendation list +- Produce a single grouped/aggregated view ("top N by impact", "by resource type") +- Quickly check whether their tenant even has Advisor data yet (new/empty subs) + +This is a **read-only** review. Even if an `apply`-style advisor tool exists, this +sub-skill only proposes IaaC fix snippets — it never modifies cloud state. + +## Shared References + +This capability builds on two product-area references shared by every `azure-advisor` +capability — read them before running the workflow: + +- **[Capability Routing](../references/capability-routing.md)** — how to pick the right + `advisor_*` MCP tool by capability (catalog, recommendations, summary, IaaC fix). +- **[Subscription Discovery](../references/subscription-discovery.md)** — how to resolve + the target subscription from repo config / env without hardcoding. + +## Workflow + +### Step 1 — Resolve subscription + +Run [Subscription Discovery](../references/subscription-discovery.md). Save the resolved +value; mention its *source* in the final chat summary ("Subscription pulled from +`infra/main.parameters.json`"). + +### Step 2 — Probe the catalog (metadata) + +Invoke the **metadata / catalog** capability (see +[Capability Routing](../references/capability-routing.md)) with no filter to learn what +recommendation categories, impacts, and resource types Advisor knows about for this +tenant. Cache the result — later steps can reference category/impact values from it. + +If this call returns an empty list, the tenant likely lacks Advisor coverage — report +that and stop early. + +### Step 3 — Pull active recommendations + +Invoke the **active recommendations** capability with the resolved subscription and +**no filters first** to get raw counts. Limit output to a manageable page if the tool +supports it. + +### Step 4 — Aggregate (if available) + +If a **summary / aggregation** capability exists, call it twice: + +- Once grouped by **category** (Cost, Security, Reliability, etc.) +- Once grouped by **impact** (High, Medium, Low) + +If no aggregation tool exists, derive the same two breakdowns locally from the Step 3 +list. + +### Step 5 — Spotlight high-impact items + +From the Step 3 results, pick up to **5 distinct High-impact recommendations** across +different resource types. For each, if an **IaaC remediation** capability exists and +the recommendation's resource type matches a supported type, fetch a fix snippet. + +### Step 6 — Compose chat summary + +Reply in chat with this structure (no files written): + +``` +## Azure Advisor Review + +**Subscription:** (from ) +**Tenant catalog:** N categories, M impact levels, K resource types + +### Snapshot +- Total active recommendations: X +- By category: Cost A | Security B | Reliability C | Performance D | Operational E +- By impact: High H | Medium M | Low L + +### High-impact spotlight (up to 5) +1. — — + +... + +### Notes +- Tools used: +- Tools skipped (no capability match): +``` + +Do **not** write any file to disk. The summary lives only in the chat response. + +## Constraints + +- ✅ **Always** discover the subscription from repo files / env first; ask only as last resort. +- ✅ **Always** include every advisor_* tool name actually invoked in the "Tools used" line so the user can see what the skill chose. +- ✅ **Always** mention which steps were skipped because no matching capability was found. +- ❌ **Never** hardcode a subscription id, tenant id, or resource group. +- ❌ **Never** modify Azure state — this sub-skill is read + suggest only. +- ❌ **Never** call non-`advisor_*` tools as substitutes; if no capability matches, report it and skip. +- ❌ **Never** write the summary to a file unless the user explicitly asks for one in their prompt. +- ❌ **Never** write helper scripts, scratch files, or parsing utilities to disk (no `.tmp/*.js`, no `.tmp/*.json`, no temp files of any kind). Reason over MCP tool responses directly in-context. Prefer aggregating via the `advisor_recommendation_summary` tool over computing counts yourself. +- ❌ **Never** shell out to `node`, `python`, `pwsh`, `powershell`, `jq`, or any other interpreter to read, parse, group, or count MCP tool responses. The tool responses are already in your context — reason over them directly. If you need server-side aggregation, use the `advisor_recommendation_summary` tool. + +## Error Handling + +| Symptom | Probable cause | Action | +|---|---|---| +| `advisor_*` tools missing entirely | MCP server not configured / not running | Tell user to check `.vscode/mcp.json` and that `azmcp.exe` is reachable. | +| Catalog call returns empty | Tenant has no Advisor coverage yet | Stop after Step 2; report empty tenant. | +| Recommendation list 401/403 | Auth not scoped to subscription | Tell user to run `az login` and verify subscription access. | +| Aggregation tool errors on `group-by` | Field name not in supported list | Re-call with a value drawn from Step 2's catalog. | +| IaaC tool says "unknown resource" | Resource type not in its supported list | Skip the fix for that recommendation; keep the rest. | diff --git a/plugin/skills/azure-advisor/version.json b/plugin/skills/azure-advisor/version.json new file mode 100644 index 000000000..af73f6419 --- /dev/null +++ b/plugin/skills/azure-advisor/version.json @@ -0,0 +1,6 @@ +{ + "version": "1.1", + "pathFilters": [ + "." + ] +} diff --git a/tests/azure-advisor/__snapshots__/triggers.test.ts.snap b/tests/azure-advisor/__snapshots__/triggers.test.ts.snap new file mode 100644 index 000000000..59a8e123e --- /dev/null +++ b/tests/azure-advisor/__snapshots__/triggers.test.ts.snap @@ -0,0 +1,117 @@ +// Jest Snapshot v1, https://jestjs.io/docs/snapshot-testing + +exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` +{ + "description": "Azure Advisor reviews and recommendation workflows using whichever advisor_* MCP tools are available. WHEN: "run an advisor review", "check my Azure advisor recommendations", "summarize advisor findings", "what does Advisor say about my subscription", "give me an advisor health check", "audit my Azure resources with Advisor". USE FOR: end-to-end Advisor sweeps that combine the recommendation catalog, active recommendations, summaries/aggregates, and IaaC fix suggestions into one chat summary. DO NOT USE FOR: applying changes to Azure resources directly (read-only review), cost analysis beyond Advisor's cost category (use azure-cost), or non-Advisor diagnostics (use azure-diagnostics).", + "extractedKeywords": [ + "about", + "active", + "advisor", + "advisor_", + "aggregates", + "analysis", + "applying", + "audit", + "available", + "azure", + "azure-cost", + "azure-diagnostics", + "beyond", + "catalog", + "category", + "changes", + "chat", + "check", + "combine", + "cost", + "diagnostics", + "directly", + "does", + "end-to-end", + "findings", + "give", + "health", + "iaac", + "into", + "mcp", + "non-advisor", + "read-only", + "recommendation", + "recommendations", + "resources", + "review", + "reviews", + "subscription", + "suggestions", + "summaries", + "summarize", + "summary", + "sweeps", + "that", + "tools", + "using", + "what", + "when", + "whichever", + "with", + "workflows", + ], + "name": "azure-advisor", +} +`; + +exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill keywords match snapshot 1`] = ` +[ + "about", + "active", + "advisor", + "advisor_", + "aggregates", + "analysis", + "applying", + "audit", + "available", + "azure", + "azure-cost", + "azure-diagnostics", + "beyond", + "catalog", + "category", + "changes", + "chat", + "check", + "combine", + "cost", + "diagnostics", + "directly", + "does", + "end-to-end", + "findings", + "give", + "health", + "iaac", + "into", + "mcp", + "non-advisor", + "read-only", + "recommendation", + "recommendations", + "resources", + "review", + "reviews", + "subscription", + "suggestions", + "summaries", + "summarize", + "summary", + "sweeps", + "that", + "tools", + "using", + "what", + "when", + "whichever", + "with", + "workflows", +] +`; diff --git a/tests/azure-advisor/triggers.test.ts b/tests/azure-advisor/triggers.test.ts new file mode 100644 index 000000000..30ecaaf68 --- /dev/null +++ b/tests/azure-advisor/triggers.test.ts @@ -0,0 +1,92 @@ +/** + * Trigger Tests for azure-advisor + * + * Tests that verify the skill triggers on appropriate prompts + * and does NOT trigger on unrelated prompts. + * + */ + +import { TriggerMatcher } from "../utils/trigger-matcher"; +import { loadSkill, LoadedSkill } from "../utils/skill-loader"; + +const SKILL_NAME = "azure-advisor"; + +describe(`${SKILL_NAME} - Trigger Tests`, () => { + let triggerMatcher: TriggerMatcher; + let skill: LoadedSkill; + + beforeAll(async () => { + skill = await loadSkill(SKILL_NAME); + triggerMatcher = new TriggerMatcher(skill); + }); + + describe("Should Trigger - Advisor Review", () => { + const advisorPrompts: string[] = [ + "Run an Azure Advisor review of my subscription", + "Check my Azure Advisor recommendations", + "Summarize the Advisor findings for my subscription", + "What does Advisor say about my Azure subscription?", + "Give me an Azure Advisor health check", + "Audit my Azure resources with Advisor", + "Show me the top high-impact Azure Advisor recommendations", + "Aggregate my Advisor recommendations by category", + ]; + + test.each(advisorPrompts)( + 'triggers on: "%s"', + (prompt) => { + const result = triggerMatcher.shouldTrigger(prompt); + expect(result.triggered).toBe(true); + expect(result.matchedKeywords.length).toBeGreaterThanOrEqual(2); + } + ); + }); + + describe("Should NOT Trigger", () => { + const shouldNotTriggerPrompts: string[] = [ + "What is the weather today?", + "Help me write a poem", + "Explain quantum computing", + "How do I write a Python web scraper?", + ]; + + test.each(shouldNotTriggerPrompts)( + 'does not trigger on: "%s"', + (prompt) => { + const result = triggerMatcher.shouldTrigger(prompt); + expect(result.triggered).toBe(false); + } + ); + }); + + describe("Trigger Keywords Snapshot", () => { + test("skill keywords match snapshot", () => { + expect(triggerMatcher.getKeywords()).toMatchSnapshot(); + }); + + test("skill description triggers match snapshot", () => { + expect({ + name: skill.metadata.name, + description: skill.metadata.description, + extractedKeywords: triggerMatcher.getKeywords(), + }).toMatchSnapshot(); + }); + }); + + describe("Edge Cases", () => { + test("handles empty prompt", () => { + const result = triggerMatcher.shouldTrigger(""); + expect(result.triggered).toBe(false); + }); + + test("is case insensitive", () => { + const result1 = triggerMatcher.shouldTrigger( + "RUN AN AZURE ADVISOR REVIEW" + ); + const result2 = triggerMatcher.shouldTrigger( + "run an azure advisor review" + ); + expect(result1.triggered).toBe(result2.triggered); + }); + }); +}); diff --git a/tests/skills.json b/tests/skills.json index d20b46697..3cd0843c5 100644 --- a/tests/skills.json +++ b/tests/skills.json @@ -2,6 +2,7 @@ "skills": [ "airunway-aks-setup", "appinsights-instrumentation", + "azure-advisor", "azure-ai", "azure-aigateway", "azure-cloud-migrate", @@ -32,6 +33,6 @@ "integrationTestSchedule": { "0 5 * * 2-6": "microsoft-foundry", "0 8 * * 2-6": "azure-deploy", - "0 12 * * 2-6": "airunway-aks-setup,appinsights-instrumentation,azure-ai,azure-aigateway,azure-cloud-migrate,azure-compliance,azure-compute,azure-cost,azure-diagnostics,azure-enterprise-infra-planner,azure-hosted-copilot-sdk,azure-kubernetes,azure-kusto,azure-messaging,azure-prepare,azure-quotas,azure-rbac,azure-resource-lookup,azure-resource-visualizer,azure-storage,azure-upgrade,azure-validate,entra-agent-id,entra-app-registration,azure-reliability,python-appservice-deploy" + "0 12 * * 2-6": "airunway-aks-setup,appinsights-instrumentation,azure-advisor,azure-ai,azure-aigateway,azure-cloud-migrate,azure-compliance,azure-compute,azure-cost,azure-diagnostics,azure-enterprise-infra-planner,azure-hosted-copilot-sdk,azure-kubernetes,azure-kusto,azure-messaging,azure-prepare,azure-quotas,azure-rbac,azure-resource-lookup,azure-resource-visualizer,azure-storage,azure-upgrade,azure-validate,entra-agent-id,entra-app-registration,azure-reliability,python-appservice-deploy" } } \ No newline at end of file From cba3be51d5a8b2a9c88201ecec946d1e8346f1c1 Mon Sep 17 00:00:00 2001 From: Sharadhi Srikanth Date: Thu, 18 Jun 2026 14:19:13 +0530 Subject: [PATCH 02/14] fix: trim azure-advisor description to fit Copilot CLI char budget --- plugin/skills/azure-advisor/SKILL.md | 2 +- .../__snapshots__/triggers.test.ts.snap | 42 +++---------------- 2 files changed, 6 insertions(+), 38 deletions(-) diff --git a/plugin/skills/azure-advisor/SKILL.md b/plugin/skills/azure-advisor/SKILL.md index d1ca5d473..d706bfc10 100644 --- a/plugin/skills/azure-advisor/SKILL.md +++ b/plugin/skills/azure-advisor/SKILL.md @@ -1,6 +1,6 @@ --- name: azure-advisor -description: "Azure Advisor reviews and recommendation workflows using whichever advisor_* MCP tools are available. WHEN: \"run an advisor review\", \"check my Azure advisor recommendations\", \"summarize advisor findings\", \"what does Advisor say about my subscription\", \"give me an advisor health check\", \"audit my Azure resources with Advisor\". USE FOR: end-to-end Advisor sweeps that combine the recommendation catalog, active recommendations, summaries/aggregates, and IaaC fix suggestions into one chat summary. DO NOT USE FOR: applying changes to Azure resources directly (read-only review), cost analysis beyond Advisor's cost category (use azure-cost), or non-Advisor diagnostics (use azure-diagnostics)." +description: "Azure Advisor reviews and recommendations using available advisor_* MCP tools. WHEN: \"run an advisor review\", \"check my Azure advisor recommendations\", \"summarize advisor findings\", \"what does Advisor say about my subscription\", \"give me an advisor health check\", \"audit my Azure resources with Advisor\". USE FOR: read-only Advisor sweeps with catalog, recommendations, and IaaC fixes. DO NOT USE FOR: changing resources, cost analysis (use azure-cost), or non-Advisor diagnostics (use azure-diagnostics)." license: MIT metadata: author: Microsoft diff --git a/tests/azure-advisor/__snapshots__/triggers.test.ts.snap b/tests/azure-advisor/__snapshots__/triggers.test.ts.snap index 59a8e123e..adbe2dafd 100644 --- a/tests/azure-advisor/__snapshots__/triggers.test.ts.snap +++ b/tests/azure-advisor/__snapshots__/triggers.test.ts.snap @@ -2,59 +2,43 @@ exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Azure Advisor reviews and recommendation workflows using whichever advisor_* MCP tools are available. WHEN: "run an advisor review", "check my Azure advisor recommendations", "summarize advisor findings", "what does Advisor say about my subscription", "give me an advisor health check", "audit my Azure resources with Advisor". USE FOR: end-to-end Advisor sweeps that combine the recommendation catalog, active recommendations, summaries/aggregates, and IaaC fix suggestions into one chat summary. DO NOT USE FOR: applying changes to Azure resources directly (read-only review), cost analysis beyond Advisor's cost category (use azure-cost), or non-Advisor diagnostics (use azure-diagnostics).", + "description": "Azure Advisor reviews and recommendations using available advisor_* MCP tools. WHEN: "run an advisor review", "check my Azure advisor recommendations", "summarize advisor findings", "what does Advisor say about my subscription", "give me an advisor health check", "audit my Azure resources with Advisor". USE FOR: read-only Advisor sweeps with catalog, recommendations, and IaaC fixes. DO NOT USE FOR: changing resources, cost analysis (use azure-cost), or non-Advisor diagnostics (use azure-diagnostics).", "extractedKeywords": [ "about", - "active", "advisor", "advisor_", - "aggregates", "analysis", - "applying", "audit", "available", "azure", "azure-cost", "azure-diagnostics", - "beyond", "catalog", - "category", - "changes", - "chat", + "changing", "check", - "combine", "cost", "diagnostics", - "directly", "does", - "end-to-end", "findings", + "fixes", "give", "health", "iaac", - "into", "mcp", "non-advisor", "read-only", - "recommendation", "recommendations", "resources", "review", "reviews", "subscription", - "suggestions", - "summaries", "summarize", - "summary", "sweeps", - "that", "tools", "using", "what", "when", - "whichever", "with", - "workflows", ], "name": "azure-advisor", } @@ -63,55 +47,39 @@ exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill descripti exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill keywords match snapshot 1`] = ` [ "about", - "active", "advisor", "advisor_", - "aggregates", "analysis", - "applying", "audit", "available", "azure", "azure-cost", "azure-diagnostics", - "beyond", "catalog", - "category", - "changes", - "chat", + "changing", "check", - "combine", "cost", "diagnostics", - "directly", "does", - "end-to-end", "findings", + "fixes", "give", "health", "iaac", - "into", "mcp", "non-advisor", "read-only", - "recommendation", "recommendations", "resources", "review", "reviews", "subscription", - "suggestions", - "summaries", "summarize", - "summary", "sweeps", - "that", "tools", "using", "what", "when", - "whichever", "with", - "workflows", ] `; From c1f900dc6981031025f30eb5e735f02b3789f773 Mon Sep 17 00:00:00 2001 From: Sharadhi Srikanth Date: Thu, 25 Jun 2026 13:51:37 +0530 Subject: [PATCH 03/14] fix(azure-advisor): match advisor_ tools by substring, harden triggers, IaaC->IaC - Match MCP tools whose name contains advisor_ (e.g. azure-mcp-advisor_*) instead of requiring an advisor_ prefix, so Copilot CLI server-prefixed tools resolve correctly - Reword description to stop cost/diagnostics keyword bleed and expand shouldNotTriggerPrompts with adjacent-Azure-service boundaries (4 -> 9) - Regenerate triggers snapshot for updated description - Correct IaaC -> IaC across SKILL.md, README, references, and review docs - Harden .env* handling to read only AZURE_SUBSCRIPTION_ID line --- plugin/skills/azure-advisor/README.md | 4 ++-- plugin/skills/azure-advisor/SKILL.md | 16 ++++++++------- .../references/capability-routing.md | 20 ++++++++++++++++--- .../references/subscription-discovery.md | 2 +- plugin/skills/azure-advisor/review/review.md | 16 +++++++-------- .../__snapshots__/triggers.test.ts.snap | 14 ++++++------- tests/azure-advisor/triggers.test.ts | 5 +++++ 7 files changed, 49 insertions(+), 28 deletions(-) diff --git a/plugin/skills/azure-advisor/README.md b/plugin/skills/azure-advisor/README.md index 8a040c23a..75737b52f 100644 --- a/plugin/skills/azure-advisor/README.md +++ b/plugin/skills/azure-advisor/README.md @@ -17,7 +17,7 @@ user intent. Each capability is a self-contained folder; shared concerns live in |------|---------| | [SKILL.md](SKILL.md) | **Router.** Frontmatter (makes the skill discoverable) + a capability table that routes intent to a capability file. Keep it thin. | | [references/capability-routing.md](references/capability-routing.md) | **Shared, capability-agnostic docs** reused by every capability. Don't duplicate these inside a capability. | -| [references/capability-routing.md](references/capability-routing.md) | How to resolve an `advisor_*` MCP tool by capability (catalog, recommendations, summary, IaaC fix). | +| [references/capability-routing.md](references/capability-routing.md) | How to resolve an `advisor_*` MCP tool by capability (catalog, recommendations, summary, IaC fix). | | [references/subscription-discovery.md](references/subscription-discovery.md) | How to resolve the target subscription from repo config / env without hardcoding. | | [review/review.md](review/review.md) | **Capability:** holistic read-only Advisor sweep. | | [review/review.md](review/review.md) | The review workflow (the only file the agent reads for this capability). | @@ -43,7 +43,7 @@ user intent. Each capability is a self-contained folder; shared concerns live in [subscription-discovery.md](references/subscription-discovery.md) where relevant. 3. **Register it in the router:** add a row to the **Capabilities** table in [SKILL.md](SKILL.md) (and remove it from the Roadmap list there if listed). -4. **Keep it read-only by default.** Advisor workflows propose IaaC fixes; they don't +4. **Keep it read-only by default.** Advisor workflows propose IaC fixes; they don't mutate cloud state. Document any exception explicitly. 5. **Do not edit `metadata.version`** in [SKILL.md](SKILL.md) — it stays `0.0.0-placeholder`; NBGV stamps the real version at build time. diff --git a/plugin/skills/azure-advisor/SKILL.md b/plugin/skills/azure-advisor/SKILL.md index d706bfc10..d96d7e164 100644 --- a/plugin/skills/azure-advisor/SKILL.md +++ b/plugin/skills/azure-advisor/SKILL.md @@ -1,6 +1,6 @@ --- name: azure-advisor -description: "Azure Advisor reviews and recommendations using available advisor_* MCP tools. WHEN: \"run an advisor review\", \"check my Azure advisor recommendations\", \"summarize advisor findings\", \"what does Advisor say about my subscription\", \"give me an advisor health check\", \"audit my Azure resources with Advisor\". USE FOR: read-only Advisor sweeps with catalog, recommendations, and IaaC fixes. DO NOT USE FOR: changing resources, cost analysis (use azure-cost), or non-Advisor diagnostics (use azure-diagnostics)." +description: "Azure Advisor reviews and recommendations using available advisor_* MCP tools. WHEN: \"run an advisor review\", \"check my Azure advisor recommendations\", \"summarize advisor findings\", \"what does Advisor say about my subscription\", \"give me an advisor health check\", \"audit my Azure resources with Advisor\". USE FOR: read-only Advisor sweeps with catalog, recommendations, and IaC fixes. DO NOT USE FOR: changing resources, billing analysis (use azure-cost), or non-Advisor troubleshooting (use azure-diagnostics)." license: MIT metadata: author: Microsoft @@ -12,7 +12,7 @@ metadata: Azure Advisor is a **product area** with multiple capabilities. This skill routes a user's intent to the right capability and runs it using whichever `advisor_*` MCP tools the connected Azure MCP server exposes. Routing is by *capability* (catalog, -recommendations, summary, IaaC fix), not by hard-coded tool names, so the skill stays +recommendations, summary, IaC fix), not by hard-coded tool names, so the skill stays useful as new advisor tools land. > 🛠️ **Contributing a new capability?** See [README.md](README.md) for the folder map @@ -21,8 +21,10 @@ useful as new advisor tools land. ## Pre-Execution Requirements Inspect the available `advisor_*` MCP tools and their parameters before running a -capability. Match tools by capability description, not by a fixed name list — see the -shared [Capability Routing](references/capability-routing.md) reference. +capability. Match a tool when its name **contains** `advisor_` (i.e. `*advisor_*`), not +only when it *starts with* it — MCP clients prepend a server-name prefix (e.g. +`azure-mcp-advisor_recommendation_list`). Match by capability description, not by a fixed +name list — see the shared [Capability Routing](references/capability-routing.md) reference. ## Shared References @@ -31,7 +33,7 @@ relevant one before acting: | Reference | Purpose | |-----------|---------| -| [Capability Routing](references/capability-routing.md) | Resolve which `advisor_*` MCP tool to call for each capability (catalog, recommendations, summary, IaaC fix). | +| [Capability Routing](references/capability-routing.md) | Resolve which `advisor_*` MCP tool to call for each capability (catalog, recommendations, summary, IaC fix). | | [Subscription Discovery](references/subscription-discovery.md) | Resolve the target subscription from repo config / env without hardcoding. | ## Capabilities @@ -41,7 +43,7 @@ skill when they match the task:** | Capability | When to Use | Reference | |-----------|-------------|-----------| -| **review** | Run a holistic, read-only Advisor sweep across a subscription — probe the catalog, pull active recommendations, aggregate by category/impact, spotlight high-impact items, and propose IaaC fix snippets. | [review](review/review.md) | +| **review** | Run a holistic, read-only Advisor sweep across a subscription — probe the catalog, pull active recommendations, aggregate by category/impact, spotlight high-impact items, and propose IaC fix snippets. | [review](review/review.md) | ### Roadmap (not yet implemented) @@ -59,4 +61,4 @@ reusing the shared references above. Add a row to the table above when one ships - ❌ **Never** hardcode a subscription id, tenant id, or resource group. - ❌ **Never** modify Azure state — Advisor workflows in this skill are read + suggest only. -- ❌ **Never** call non-`advisor_*` tools as substitutes; if no capability matches, report it and skip. +- ❌ **Never** call a tool whose name does not contain `advisor_` as a substitute; if no capability matches, report it and skip. diff --git a/plugin/skills/azure-advisor/references/capability-routing.md b/plugin/skills/azure-advisor/references/capability-routing.md index d6f722005..096da3fbc 100644 --- a/plugin/skills/azure-advisor/references/capability-routing.md +++ b/plugin/skills/azure-advisor/references/capability-routing.md @@ -7,17 +7,31 @@ The connected Azure MCP server may expose a changing set of `advisor_*` tools. Resolve which tool to invoke at each step from this capability table. +## Tool name matching + +Match a tool when its name **contains** `advisor_` (i.e. `*advisor_*`) — **not** only +when it *starts with* `advisor_`. MCP clients prepend the **server name** to every tool, +so the same Advisor tool surfaces under a different prefix depending on the host: + +| Client | Example tool name | +|---|---| +| Copilot CLI | `azure-mcp-advisor_recommendation_list` | +| Other hosts | `advisor_recommendation_list` | + +A strict *starts-with* check would reject `azure-mcp-advisor_recommendation_list` and the +skill would wrongly report "no Advisor tools found". Always substring-match on `advisor_`. + | Capability | Look for an `advisor_*` tool whose description says... | Required input | |---|---|---| | **Metadata / catalog** | "list recommendation types / categories / impact levels / supported values" | tenant context (no subscription needed) | | **Active recommendations** | "list Advisor recommendations in a subscription" | subscription (resource group optional, filters optional) | | **Aggregation / summary** | "summarize / group / aggregate Advisor recommendations" | subscription + group-by field | -| **IaaC remediation** | "apply Advisor recommendations to IaaC / ARM / Bicep / Terraform" | a resource type identifier | +| **IaC remediation** | "apply Advisor recommendations to IaC / ARM / Bicep / Terraform" | a resource type identifier | ## Resolution rules - If a step's capability has **no matching tool**, skip it and note that in the chat output (e.g. "no aggregation tool available, presenting raw list"). - If **multiple tools** match, prefer the one with the more specific description. -- **Never** substitute a non-`advisor_*` tool for a missing capability — report the gap - and skip instead. +- **Never** substitute a tool whose name does **not** contain `advisor_` for a missing + capability — report the gap and skip instead. diff --git a/plugin/skills/azure-advisor/references/subscription-discovery.md b/plugin/skills/azure-advisor/references/subscription-discovery.md index b6b914bce..b38aeaed9 100644 --- a/plugin/skills/azure-advisor/references/subscription-discovery.md +++ b/plugin/skills/azure-advisor/references/subscription-discovery.md @@ -9,7 +9,7 @@ Subscription **must never be hardcoded**. Resolve in this order and stop at the - `azure.yaml` → `subscriptionId:` key - `.azure/*/config.json` → `subscriptionId` field - `*.bicepparam` / `*.parameters.json` → `subscriptionId` / `subscription` parameter - - `.env*` files → `AZURE_SUBSCRIPTION_ID` line + - `.env*` files → read **only** the `AZURE_SUBSCRIPTION_ID` line; never load, echo, or summarize other `.env*` contents (they routinely hold secrets) - `infra/**/main.parameters.json` 2. **Environment variable** `AZURE_SUBSCRIPTION_ID`. 3. **Ask the user.** Do not guess. Say which files were scanned and what was missing. diff --git a/plugin/skills/azure-advisor/review/review.md b/plugin/skills/azure-advisor/review/review.md index 8a0b69931..0aaf17569 100644 --- a/plugin/skills/azure-advisor/review/review.md +++ b/plugin/skills/azure-advisor/review/review.md @@ -2,7 +2,7 @@ Drive an end-to-end Azure Advisor sweep using whichever `advisor_*` MCP tools the connected Azure MCP server exposes. Designed to stay useful as new advisor tools -land — it routes by *capability* (catalog, recommendations, summary, IaaC fix), +land — it routes by *capability* (catalog, recommendations, summary, IaC fix), not by hard-coded tool name lists. ## When to Use This Sub-Skill @@ -16,7 +16,7 @@ Use this sub-skill when the user wants to: - Quickly check whether their tenant even has Advisor data yet (new/empty subs) This is a **read-only** review. Even if an `apply`-style advisor tool exists, this -sub-skill only proposes IaaC fix snippets — it never modifies cloud state. +sub-skill only proposes IaC fix snippets — it never modifies cloud state. ## Shared References @@ -24,7 +24,7 @@ This capability builds on two product-area references shared by every `azure-adv capability — read them before running the workflow: - **[Capability Routing](../references/capability-routing.md)** — how to pick the right - `advisor_*` MCP tool by capability (catalog, recommendations, summary, IaaC fix). + `advisor_*` MCP tool by capability (catalog, recommendations, summary, IaC fix). - **[Subscription Discovery](../references/subscription-discovery.md)** — how to resolve the target subscription from repo config / env without hardcoding. @@ -65,7 +65,7 @@ list. ### Step 5 — Spotlight high-impact items From the Step 3 results, pick up to **5 distinct High-impact recommendations** across -different resource types. For each, if an **IaaC remediation** capability exists and +different resource types. For each, if an **IaC remediation** capability exists and the recommendation's resource type matches a supported type, fetch a fix snippet. ### Step 6 — Compose chat summary @@ -85,7 +85,7 @@ Reply in chat with this structure (no files written): ### High-impact spotlight (up to 5) 1. — — - + ... ### Notes @@ -102,7 +102,7 @@ Do **not** write any file to disk. The summary lives only in the chat response. - ✅ **Always** mention which steps were skipped because no matching capability was found. - ❌ **Never** hardcode a subscription id, tenant id, or resource group. - ❌ **Never** modify Azure state — this sub-skill is read + suggest only. -- ❌ **Never** call non-`advisor_*` tools as substitutes; if no capability matches, report it and skip. +- ❌ **Never** call a tool whose name does not contain `advisor_` as a substitute; if no capability matches, report it and skip. (Match on the substring `advisor_` — clients prepend a server-name prefix like `azure-mcp-`.) - ❌ **Never** write the summary to a file unless the user explicitly asks for one in their prompt. - ❌ **Never** write helper scripts, scratch files, or parsing utilities to disk (no `.tmp/*.js`, no `.tmp/*.json`, no temp files of any kind). Reason over MCP tool responses directly in-context. Prefer aggregating via the `advisor_recommendation_summary` tool over computing counts yourself. - ❌ **Never** shell out to `node`, `python`, `pwsh`, `powershell`, `jq`, or any other interpreter to read, parse, group, or count MCP tool responses. The tool responses are already in your context — reason over them directly. If you need server-side aggregation, use the `advisor_recommendation_summary` tool. @@ -111,8 +111,8 @@ Do **not** write any file to disk. The summary lives only in the chat response. | Symptom | Probable cause | Action | |---|---|---| -| `advisor_*` tools missing entirely | MCP server not configured / not running | Tell user to check `.vscode/mcp.json` and that `azmcp.exe` is reachable. | +| No tool name contains `advisor_` | MCP server not configured / not running, **or** a strict starts-with match rejected prefixed names | Substring-match on `advisor_` (names look like `azure-mcp-advisor_*`). If still none, tell user to check `.vscode/mcp.json`, that `azmcp.exe` is reachable, and that tools are exposed individually (`--mode all`). | | Catalog call returns empty | Tenant has no Advisor coverage yet | Stop after Step 2; report empty tenant. | | Recommendation list 401/403 | Auth not scoped to subscription | Tell user to run `az login` and verify subscription access. | | Aggregation tool errors on `group-by` | Field name not in supported list | Re-call with a value drawn from Step 2's catalog. | -| IaaC tool says "unknown resource" | Resource type not in its supported list | Skip the fix for that recommendation; keep the rest. | +| IaC tool says "unknown resource" | Resource type not in its supported list | Skip the fix for that recommendation; keep the rest. | diff --git a/tests/azure-advisor/__snapshots__/triggers.test.ts.snap b/tests/azure-advisor/__snapshots__/triggers.test.ts.snap index adbe2dafd..6c334ce1e 100644 --- a/tests/azure-advisor/__snapshots__/triggers.test.ts.snap +++ b/tests/azure-advisor/__snapshots__/triggers.test.ts.snap @@ -2,7 +2,7 @@ exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Azure Advisor reviews and recommendations using available advisor_* MCP tools. WHEN: "run an advisor review", "check my Azure advisor recommendations", "summarize advisor findings", "what does Advisor say about my subscription", "give me an advisor health check", "audit my Azure resources with Advisor". USE FOR: read-only Advisor sweeps with catalog, recommendations, and IaaC fixes. DO NOT USE FOR: changing resources, cost analysis (use azure-cost), or non-Advisor diagnostics (use azure-diagnostics).", + "description": "Azure Advisor reviews and recommendations using available advisor_* MCP tools. WHEN: "run an advisor review", "check my Azure advisor recommendations", "summarize advisor findings", "what does Advisor say about my subscription", "give me an advisor health check", "audit my Azure resources with Advisor". USE FOR: read-only Advisor sweeps with catalog, recommendations, and IaC fixes. DO NOT USE FOR: changing resources, billing analysis (use azure-cost), or non-Advisor troubleshooting (use azure-diagnostics).", "extractedKeywords": [ "about", "advisor", @@ -13,17 +13,16 @@ exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill descripti "azure", "azure-cost", "azure-diagnostics", + "billing", "catalog", "changing", "check", - "cost", - "diagnostics", + "cli", "does", "findings", "fixes", "give", "health", - "iaac", "mcp", "non-advisor", "read-only", @@ -35,6 +34,7 @@ exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill descripti "summarize", "sweeps", "tools", + "troubleshooting", "using", "what", "when", @@ -55,17 +55,16 @@ exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill keywords "azure", "azure-cost", "azure-diagnostics", + "billing", "catalog", "changing", "check", - "cost", - "diagnostics", + "cli", "does", "findings", "fixes", "give", "health", - "iaac", "mcp", "non-advisor", "read-only", @@ -77,6 +76,7 @@ exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill keywords "summarize", "sweeps", "tools", + "troubleshooting", "using", "what", "when", diff --git a/tests/azure-advisor/triggers.test.ts b/tests/azure-advisor/triggers.test.ts index 30ecaaf68..962b64c64 100644 --- a/tests/azure-advisor/triggers.test.ts +++ b/tests/azure-advisor/triggers.test.ts @@ -48,6 +48,11 @@ describe(`${SKILL_NAME} - Trigger Tests`, () => { "Help me write a poem", "Explain quantum computing", "How do I write a Python web scraper?", + "Analyze my Azure costs and spending trends", + "Troubleshoot my Azure App Service deployment failure", + "Help me set up Azure Key Vault for secrets management", + "Deploy my web app to Azure App Service", + "Set up Azure RBAC role assignments for my team", ]; test.each(shouldNotTriggerPrompts)( From 058ce50aea582eff91156aefa553c5557553db1a Mon Sep 17 00:00:00 2001 From: Sharadhi Srikanth Date: Thu, 25 Jun 2026 14:40:46 +0530 Subject: [PATCH 04/14] docs(azure-advisor): collapse duplicate folder-map rows in README --- plugin/skills/azure-advisor/README.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/plugin/skills/azure-advisor/README.md b/plugin/skills/azure-advisor/README.md index 75737b52f..421c09a06 100644 --- a/plugin/skills/azure-advisor/README.md +++ b/plugin/skills/azure-advisor/README.md @@ -16,11 +16,9 @@ user intent. Each capability is a self-contained folder; shared concerns live in | Path | Purpose | |------|---------| | [SKILL.md](SKILL.md) | **Router.** Frontmatter (makes the skill discoverable) + a capability table that routes intent to a capability file. Keep it thin. | -| [references/capability-routing.md](references/capability-routing.md) | **Shared, capability-agnostic docs** reused by every capability. Don't duplicate these inside a capability. | -| [references/capability-routing.md](references/capability-routing.md) | How to resolve an `advisor_*` MCP tool by capability (catalog, recommendations, summary, IaC fix). | +| [references/capability-routing.md](references/capability-routing.md) | **Shared.** Capability-agnostic routing docs reused by every capability: resolve which `advisor_*` MCP tool to call for each capability (catalog, recommendations, summary, IaC fix). | | [references/subscription-discovery.md](references/subscription-discovery.md) | How to resolve the target subscription from repo config / env without hardcoding. | -| [review/review.md](review/review.md) | **Capability:** holistic read-only Advisor sweep. | -| [review/review.md](review/review.md) | The review workflow (the only file the agent reads for this capability). | +| [review/review.md](review/review.md) | **Capability:** holistic read-only Advisor sweep (the only file the agent reads for this capability). | ## Conventions (match the rest of the repo) From d407be6e73a3da354a3b1f3bcda37e86c1811132 Mon Sep 17 00:00:00 2001 From: Sharadhi Srikanth Date: Thu, 25 Jun 2026 15:38:12 +0530 Subject: [PATCH 05/14] feat: auto-discover and classify all subscriptions in azure-advisor review Auto-discover every subscription the repo references (per-environment param/azd configs) and classify each into prod/staging/test/dev/other, running the Advisor review per subscription with results grouped by environment. Tenant-wide subscription-list enumeration becomes a widen-on-request/fallback path. Also route summary via the Aggregation/summary capability instead of a hard-coded tool name, and strengthen the case-insensitive trigger test. --- plugin/skills/azure-advisor/SKILL.md | 4 +- .../references/subscription-discovery.md | 70 +++++++++++++++---- plugin/skills/azure-advisor/review/review.md | 59 +++++++++++++--- tests/azure-advisor/triggers.test.ts | 1 + 4 files changed, 110 insertions(+), 24 deletions(-) diff --git a/plugin/skills/azure-advisor/SKILL.md b/plugin/skills/azure-advisor/SKILL.md index d96d7e164..3f32eaace 100644 --- a/plugin/skills/azure-advisor/SKILL.md +++ b/plugin/skills/azure-advisor/SKILL.md @@ -34,7 +34,7 @@ relevant one before acting: | Reference | Purpose | |-----------|---------| | [Capability Routing](references/capability-routing.md) | Resolve which `advisor_*` MCP tool to call for each capability (catalog, recommendations, summary, IaC fix). | -| [Subscription Discovery](references/subscription-discovery.md) | Resolve the target subscription from repo config / env without hardcoding. | +| [Subscription Discovery](references/subscription-discovery.md) | Resolve a single target subscription, or enumerate and classify all subscriptions by environment, without hardcoding. | ## Capabilities @@ -43,7 +43,7 @@ skill when they match the task:** | Capability | When to Use | Reference | |-----------|-------------|-----------| -| **review** | Run a holistic, read-only Advisor sweep across a subscription — probe the catalog, pull active recommendations, aggregate by category/impact, spotlight high-impact items, and propose IaC fix snippets. | [review](review/review.md) | +| **review** | Run a holistic, read-only Advisor sweep across one subscription — or **all** subscriptions classified by environment (dev/staging/prod) — probing the catalog, pulling active recommendations, aggregating by category/impact, spotlighting high-impact items, and proposing IaC fix snippets. | [review](review/review.md) | ### Roadmap (not yet implemented) diff --git a/plugin/skills/azure-advisor/references/subscription-discovery.md b/plugin/skills/azure-advisor/references/subscription-discovery.md index b38aeaed9..8555cca58 100644 --- a/plugin/skills/azure-advisor/references/subscription-discovery.md +++ b/plugin/skills/azure-advisor/references/subscription-discovery.md @@ -3,16 +3,60 @@ > **Shared across all `azure-advisor` capabilities.** Any capability that operates on a > subscription should link here instead of re-defining discovery logic. -Subscription **must never be hardcoded**. Resolve in this order and stop at the first hit: - -1. **Repository config files** — scan the workspace for any of: - - `azure.yaml` → `subscriptionId:` key - - `.azure/*/config.json` → `subscriptionId` field - - `*.bicepparam` / `*.parameters.json` → `subscriptionId` / `subscription` parameter - - `.env*` files → read **only** the `AZURE_SUBSCRIPTION_ID` line; never load, echo, or summarize other `.env*` contents (they routinely hold secrets) - - `infra/**/main.parameters.json` -2. **Environment variable** `AZURE_SUBSCRIPTION_ID`. -3. **Ask the user.** Do not guess. Say which files were scanned and what was missing. - -Always mention the *source* of the resolved subscription in the final chat summary -(e.g. "Subscription pulled from `infra/main.parameters.json`"). +## Default — discover every subscription the repo references + +Auto-discovery is the point of this skill: **do not make the user name the scope**. Scan +the workspace and collect **every distinct** subscription id you find — a repo commonly +pins a different subscription per environment, so **don't stop at the first hit**. +Subscriptions **must never be hardcoded**. Scan all of: + +- `azure.yaml` → `subscriptionId:` key, plus any per-environment azd configs +- `.azure/*/config.json` → `subscriptionId` field (one per azd environment) +- `*.bicepparam` / `*.parameters.json`, incl. `infra/**/main.parameters.json` → + `subscriptionId` / `subscription` parameter +- `.env*` files → read **only** the `AZURE_SUBSCRIPTION_ID` line; never load, echo, or + summarize other `.env*` contents (they routinely hold secrets) +- environment variable `AZURE_SUBSCRIPTION_ID` + +Then **classify each** discovered subscription by environment (see table below) and carry +the `(id, name, environment, source)` tuple forward so every per-subscription result can +be attributed and grouped in the summary. If only one subscription is found, the review +simply runs once. Always mention each *source* in the final chat summary +(e.g. "dev sub from `infra/main.parameters.json`, prod sub from `.azure/prod/config.json`"). + +## Widen to the whole tenant — on request or as fallback + +Enumerate **all** subscriptions the signed-in identity can access when either: + +- the user explicitly asks for a tenant-wide / org-wide sweep ("all my subscriptions", + "every subscription in the tenant"), **or** +- repo discovery above found nothing. + +Invoke the Azure MCP **subscription-list** tool (a tool whose name contains `subscription` +and whose description says "list Azure subscriptions"). Do **not** hardcode ids. Classify +each result with the same table below. If no subscription-list tool is available and repo +discovery also found nothing, fall back to **Ask the user**. + +## Ask the user — last resort + +Only if repo discovery yields nothing **and** tenant enumeration is unavailable. Do not +guess. Say which files were scanned and what was missing. + +## Classify each subscription by environment + +Use the first signal that matches: + +- **Tags** — an `Environment` / `env` tag value (e.g. `Production`, `Staging`, `Dev`). +- **Name / config keywords** (case-insensitive substring) when no tag exists — the + subscription name, or the azd environment / param-file name it came from: + + | Bucket | Match any of | + |---|---| + | **prod** | `prod`, `production`, `prd`, `live` | + | **staging** | `stag`, `staging`, `stg`, `uat`, `preprod`, `pre-prod` | + | **test** | `test`, `qa`, `sit` | + | **dev** | `dev`, `development`, `sandbox`, `sbx` | + | **other** | nothing above matched — list under "Unclassified" | + +Never invent an environment; if ambiguous, place it in **other** and say so. + diff --git a/plugin/skills/azure-advisor/review/review.md b/plugin/skills/azure-advisor/review/review.md index 0aaf17569..2adf41fe7 100644 --- a/plugin/skills/azure-advisor/review/review.md +++ b/plugin/skills/azure-advisor/review/review.md @@ -10,6 +10,7 @@ not by hard-coded tool name lists. Use this sub-skill when the user wants to: - Get an overall picture of Advisor findings for an Azure subscription +- Sweep **all** subscriptions (dev, staging, prod, …), classify them by environment, and get recommendations grouped per environment - See "what categories / impact levels does Advisor surface here?" before drilling in - Combine the recommendation catalog (metadata) with the active recommendation list - Produce a single grouped/aggregated view ("top N by impact", "by resource type") @@ -30,11 +31,20 @@ capability — read them before running the workflow: ## Workflow -### Step 1 — Resolve subscription +### Step 1 — Discover and classify subscription(s) -Run [Subscription Discovery](../references/subscription-discovery.md). Save the resolved -value; mention its *source* in the final chat summary ("Subscription pulled from -`infra/main.parameters.json`"). +Run [Subscription Discovery](../references/subscription-discovery.md). By **default**, +auto-discover **every** subscription the repo references (a repo often pins a different +subscription per environment) and classify each into an environment bucket (prod / +staging / test / dev / other) — do **not** wait for the user to name the scope. Then run +Steps 2–5 **once per discovered subscription**. + +- If discovery finds only one subscription, the review simply runs once. +- Widen to a tenant-wide enumeration (subscription-list tool) only when the user explicitly + asks for it, or when repo discovery finds nothing. + +Mention each subscription's *source* and the environment breakdown in the final chat +summary. Never hardcode ids. ### Step 2 — Probe the catalog (metadata) @@ -70,7 +80,9 @@ the recommendation's resource type matches a supported type, fetch a fix snippet ### Step 6 — Compose chat summary -Reply in chat with this structure (no files written): +Reply in chat with this structure (no files written). + +**Single subscription:** ``` ## Azure Advisor Review @@ -93,26 +105,55 @@ Reply in chat with this structure (no files written): - Tools skipped (no capability match): ``` +**All subscriptions** — group the report by environment bucket (prod first, then staging, +test, dev, other), with a roll-up at the top: + +``` +## Azure Advisor Review — All Subscriptions + +**Scope:** S subscriptions across E environments +**Totals:** High H | Medium M | Low L · Cost A | Security B | Reliability C | Performance D | Operational E + +### prod (n subscriptions) +- (): High h | Medium m | Low l — top category + - High-impact spotlight (up to 5): — — + +... + +### staging (n subscriptions) +... + +### Unclassified (n subscriptions) +... + +### Notes +- Subscriptions enumerated via: +- Tools used: +- Tools skipped (no capability match): +- Subscriptions with no Advisor coverage: +``` + Do **not** write any file to disk. The summary lives only in the chat response. ## Constraints - ✅ **Always** discover the subscription from repo files / env first; ask only as last resort. +- ✅ **Always** auto-discover **every** subscription the repo references and classify each by environment before reviewing; never make the user name the scope and never hardcode the list. - ✅ **Always** include every advisor_* tool name actually invoked in the "Tools used" line so the user can see what the skill chose. - ✅ **Always** mention which steps were skipped because no matching capability was found. - ❌ **Never** hardcode a subscription id, tenant id, or resource group. - ❌ **Never** modify Azure state — this sub-skill is read + suggest only. - ❌ **Never** call a tool whose name does not contain `advisor_` as a substitute; if no capability matches, report it and skip. (Match on the substring `advisor_` — clients prepend a server-name prefix like `azure-mcp-`.) - ❌ **Never** write the summary to a file unless the user explicitly asks for one in their prompt. -- ❌ **Never** write helper scripts, scratch files, or parsing utilities to disk (no `.tmp/*.js`, no `.tmp/*.json`, no temp files of any kind). Reason over MCP tool responses directly in-context. Prefer aggregating via the `advisor_recommendation_summary` tool over computing counts yourself. -- ❌ **Never** shell out to `node`, `python`, `pwsh`, `powershell`, `jq`, or any other interpreter to read, parse, group, or count MCP tool responses. The tool responses are already in your context — reason over them directly. If you need server-side aggregation, use the `advisor_recommendation_summary` tool. +- ❌ **Never** write helper scripts, scratch files, or parsing utilities to disk (no `.tmp/*.js`, no `.tmp/*.json`, no temp files of any kind). Reason over MCP tool responses directly in-context. Prefer aggregating via the Aggregation / summary capability (see [capability-routing.md](../references/capability-routing.md)) over computing counts yourself. +- ❌ **Never** shell out to `node`, `python`, `pwsh`, `powershell`, `jq`, or any other interpreter to read, parse, group, or count MCP tool responses. The tool responses are already in your context — reason over them directly. If you need server-side aggregation, use the Aggregation / summary capability from [capability-routing.md](../references/capability-routing.md). ## Error Handling | Symptom | Probable cause | Action | |---|---|---| | No tool name contains `advisor_` | MCP server not configured / not running, **or** a strict starts-with match rejected prefixed names | Substring-match on `advisor_` (names look like `azure-mcp-advisor_*`). If still none, tell user to check `.vscode/mcp.json`, that `azmcp.exe` is reachable, and that tools are exposed individually (`--mode all`). | -| Catalog call returns empty | Tenant has no Advisor coverage yet | Stop after Step 2; report empty tenant. | -| Recommendation list 401/403 | Auth not scoped to subscription | Tell user to run `az login` and verify subscription access. | +| Tenant-wide enumeration requested but no subscription-list tool | Azure MCP subscription tool not exposed | Fall back to repo-discovered subscriptions; if none, ask the user. | +| Catalog call returns empty | Tenant has no Advisor coverage yet | Stop after Step 2; report empty tenant. || Recommendation list 401/403 | Auth not scoped to subscription | Tell user to run `az login` and verify subscription access. | | Aggregation tool errors on `group-by` | Field name not in supported list | Re-call with a value drawn from Step 2's catalog. | | IaC tool says "unknown resource" | Resource type not in its supported list | Skip the fix for that recommendation; keep the rest. | diff --git a/tests/azure-advisor/triggers.test.ts b/tests/azure-advisor/triggers.test.ts index 962b64c64..aa8f84996 100644 --- a/tests/azure-advisor/triggers.test.ts +++ b/tests/azure-advisor/triggers.test.ts @@ -91,6 +91,7 @@ describe(`${SKILL_NAME} - Trigger Tests`, () => { const result2 = triggerMatcher.shouldTrigger( "run an azure advisor review" ); + expect(result1.triggered).toBe(true); expect(result1.triggered).toBe(result2.triggered); }); }); From 106786aafebd069277318271db55584deb4d8bac Mon Sep 17 00:00:00 2001 From: Sharadhi Srikanth Date: Thu, 25 Jun 2026 16:42:50 +0530 Subject: [PATCH 06/14] feat: scope advisor review to repo resource types and tidy review docs - Add resource-discovery reference and mandatory Step 1b to collect every repo resource type/group/id; apply scope in Steps 3-4 (filter or post-filter). - Reword SKILL.md description opener and add resource-scope + security/performance roadmap rows. - Add resource-discovery row to README folder map. - Drop whole-tenant catalog line from the chat summary (scope to customer resources, remove Tenant keyword). - Split merged error-handling table rows onto separate lines. - Update trigger snapshots. --- plugin/skills/azure-advisor/README.md | 1 + plugin/skills/azure-advisor/SKILL.md | 5 +- .../references/resource-discovery.md | 61 +++++++++++++++++++ plugin/skills/azure-advisor/review/review.md | 29 +++++++-- .../__snapshots__/triggers.test.ts.snap | 8 ++- 5 files changed, 94 insertions(+), 10 deletions(-) create mode 100644 plugin/skills/azure-advisor/references/resource-discovery.md diff --git a/plugin/skills/azure-advisor/README.md b/plugin/skills/azure-advisor/README.md index 421c09a06..73c8e6720 100644 --- a/plugin/skills/azure-advisor/README.md +++ b/plugin/skills/azure-advisor/README.md @@ -18,6 +18,7 @@ user intent. Each capability is a self-contained folder; shared concerns live in | [SKILL.md](SKILL.md) | **Router.** Frontmatter (makes the skill discoverable) + a capability table that routes intent to a capability file. Keep it thin. | | [references/capability-routing.md](references/capability-routing.md) | **Shared.** Capability-agnostic routing docs reused by every capability: resolve which `advisor_*` MCP tool to call for each capability (catalog, recommendations, summary, IaC fix). | | [references/subscription-discovery.md](references/subscription-discovery.md) | How to resolve the target subscription from repo config / env without hardcoding. | +| [references/resource-discovery.md](references/resource-discovery.md) | How to narrow a review to the resources defined in this repo (resource group / type / id) without hardcoding. | | [review/review.md](review/review.md) | **Capability:** holistic read-only Advisor sweep (the only file the agent reads for this capability). | ## Conventions (match the rest of the repo) diff --git a/plugin/skills/azure-advisor/SKILL.md b/plugin/skills/azure-advisor/SKILL.md index 3f32eaace..28bb7cbf5 100644 --- a/plugin/skills/azure-advisor/SKILL.md +++ b/plugin/skills/azure-advisor/SKILL.md @@ -1,6 +1,6 @@ --- name: azure-advisor -description: "Azure Advisor reviews and recommendations using available advisor_* MCP tools. WHEN: \"run an advisor review\", \"check my Azure advisor recommendations\", \"summarize advisor findings\", \"what does Advisor say about my subscription\", \"give me an advisor health check\", \"audit my Azure resources with Advisor\". USE FOR: read-only Advisor sweeps with catalog, recommendations, and IaC fixes. DO NOT USE FOR: changing resources, billing analysis (use azure-cost), or non-Advisor troubleshooting (use azure-diagnostics)." +description: "Azure Advisor reviews resources and provides recommendations using advisor_* MCP tools. WHEN: \"run an advisor review\", \"check my Azure advisor recommendations\", \"summarize advisor findings\", \"what does Advisor say about my subscription\", \"give me an advisor health check\", \"audit my Azure resources with Advisor\". USE FOR: read-only sweeps with catalog, recommendations, and IaC fixes. DO NOT USE FOR: changing resources, billing analysis (use azure-cost), or non-Advisor troubleshooting (use azure-diagnostics)." license: MIT metadata: author: Microsoft @@ -35,6 +35,7 @@ relevant one before acting: |-----------|---------| | [Capability Routing](references/capability-routing.md) | Resolve which `advisor_*` MCP tool to call for each capability (catalog, recommendations, summary, IaC fix). | | [Subscription Discovery](references/subscription-discovery.md) | Resolve a single target subscription, or enumerate and classify all subscriptions by environment, without hardcoding. | +| [Resource Scope Discovery](references/resource-discovery.md) | Narrow a review to the resources defined in this repo (resource group / type / id) without hardcoding. | ## Capabilities @@ -56,6 +57,8 @@ reusing the shared references above. Add a row to the table above when one ships - **cost** — Advisor cost-category optimization (coordinates with `azure-cost`) - **reliability** — Advisor reliability-category reviews - **governance** — Advisor operational-excellence / governance reviews +- **security** — Advisor security-category reviews +- **performance** — Advisor performance-category reviews ## Constraints diff --git a/plugin/skills/azure-advisor/references/resource-discovery.md b/plugin/skills/azure-advisor/references/resource-discovery.md new file mode 100644 index 000000000..e9e12609b --- /dev/null +++ b/plugin/skills/azure-advisor/references/resource-discovery.md @@ -0,0 +1,61 @@ +# Shared Reference — Repo Resource Scope Discovery + +> **Shared across all `azure-advisor` capabilities.** Any capability that wants to +> narrow Advisor results to the resources this repository actually deploys should link +> here instead of re-defining discovery logic. + +Advisor tools query a **whole subscription** by default. This skill's repo-scoped review +defaults to narrowing the query to the resources the repo defines, rather than dumping the +entire subscription. + +A repo almost always defines **several** resources of **multiple** types — collect them +**all**, never stop after the first. Only when the repo defines no Azure resources at all +should you fall back to a full-subscription review and say so. **Never** hardcode a +resource group, resource type, or resource id. + +## What to extract + +Scan the workspace's Infrastructure-as-Code and config files for any of these scoping +signals, in this order. Collect every distinct hit (a repo may define several): + +1. **Resource group(s)** — the strongest scope signal: + - `azure.yaml` → `resourceGroup:` / `resourceGroupName:` key + - `.azure/*/config.json` → `resourceGroup` field + - `*.bicepparam` / `*.parameters.json` → a `resourceGroup` / `resourceGroupName` parameter + - `.env*` files → read **only** the `AZURE_RESOURCE_GROUP` / `AZURE_RESOURCEGROUP` line; never load, echo, or summarize other `.env*` contents (they routinely hold secrets) + - `main.tf` / `*.tf` → `azurerm_resource_group` `name =` value (or a `resource_group_name` local/variable default) +2. **Resource type(s)** — derive from the IaC resource declarations: + - Bicep `resource x 'Microsoft./@...'` → `Microsoft./` + - ARM `"type": "Microsoft./"` + - Terraform `resource "azurerm_"` → map to its `Microsoft.*` provider type +3. **Specific resource id(s)** — only when a fully-qualified ARM id is present in + params/env (`/subscriptions/.../resourceGroups/.../providers/...`). + +## How to apply the scope + +Pass the discovered signal(s) to the **active recommendations** and **summary** +capabilities as filters (see [Capability Routing](capability-routing.md)): + +| Discovered signal | Filter to pass | +|---|---| +| Resource group | the tool's resource-group input | +| Resource type(s) | the tool's resource-type filter — review **once per type** when the tool takes a single type, so **every** discovered type is covered | +| Specific resource id | the tool's resource (resource-id) filter | + +Apply **all** discovered signals — the review covers the **union** of every resource group +and resource type the repo defines. Where signals overlap, prefer the narrowest available +for a given resource (a specific resource id beats its type, which beats its group), but +never drop a resource type just because a broader signal also exists. + +If the recommendations tool has **no** matching scope filter, pull the subscription list +unfiltered and **post-filter** in-context to the discovered resource groups / types / ids +before aggregating or spotlighting. + +## Resolution rules + +- If **no** scope signal is found, run the **full-subscription** review and note + "no repo resource scope found — reviewed the whole subscription" in the summary. +- Always mention the *source* of the resolved scope in the final chat summary + (e.g. "Scoped to resource group `rg-shop-prod` from `infra/main.parameters.json`"). +- Scope is a **filter**, not a guarantee — Advisor only returns findings for resources + that are actually deployed, so a repo resource with no Advisor data simply won't appear. diff --git a/plugin/skills/azure-advisor/review/review.md b/plugin/skills/azure-advisor/review/review.md index 2adf41fe7..13f92d342 100644 --- a/plugin/skills/azure-advisor/review/review.md +++ b/plugin/skills/azure-advisor/review/review.md @@ -28,6 +28,9 @@ capability — read them before running the workflow: `advisor_*` MCP tool by capability (catalog, recommendations, summary, IaC fix). - **[Subscription Discovery](../references/subscription-discovery.md)** — how to resolve the target subscription from repo config / env without hardcoding. +- **[Resource Scope Discovery](../references/resource-discovery.md)** — how to narrow the + review to the resources this repo deploys (resource group / type / id) without + hardcoding. ## Workflow @@ -46,6 +49,14 @@ Steps 2–5 **once per discovered subscription**. Mention each subscription's *source* and the environment breakdown in the final chat summary. Never hardcode ids. +### Step 1b — Resolve repo resource scope + +Run [Resource Scope Discovery](../references/resource-discovery.md) to find the resource +group(s), resource **type(s)**, and resource id(s) this repo defines. Collect **every** +distinct signal — a repo usually defines **multiple** resource types, so do **not** stop +after the first. Save them all with their *source(s)*. Only if the repo defines no Azure +resources at all, fall back to a full-subscription review and note that in the summary. + ### Step 2 — Probe the catalog (metadata) Invoke the **metadata / catalog** capability (see @@ -58,13 +69,17 @@ that and stop early. ### Step 3 — Pull active recommendations -Invoke the **active recommendations** capability with the resolved subscription and -**no filters first** to get raw counts. Limit output to a manageable page if the tool -supports it. +Invoke the **active recommendations** capability with the resolved subscription. If +Step 1b found a repo scope, pass it as the matching filter (resource group / resource +type / resource id) per [Capability Routing](../references/capability-routing.md); if the +tool has no such filter, pull unfiltered and **post-filter** the list to the discovered +scope in-context. Otherwise pull the whole subscription. Limit output to a manageable +page if the tool supports it. ### Step 4 — Aggregate (if available) -If a **summary / aggregation** capability exists, call it twice: +If a **summary / aggregation** capability exists, call it twice (applying the same Step 1b +scope filter, if any): - Once grouped by **category** (Cost, Security, Reliability, etc.) - Once grouped by **impact** (High, Medium, Low) @@ -88,7 +103,7 @@ Reply in chat with this structure (no files written). ## Azure Advisor Review **Subscription:** (from ) -**Tenant catalog:** N categories, M impact levels, K resource types +**Scope:** ### Snapshot - Total active recommendations: X @@ -154,6 +169,8 @@ Do **not** write any file to disk. The summary lives only in the chat response. |---|---|---| | No tool name contains `advisor_` | MCP server not configured / not running, **or** a strict starts-with match rejected prefixed names | Substring-match on `advisor_` (names look like `azure-mcp-advisor_*`). If still none, tell user to check `.vscode/mcp.json`, that `azmcp.exe` is reachable, and that tools are exposed individually (`--mode all`). | | Tenant-wide enumeration requested but no subscription-list tool | Azure MCP subscription tool not exposed | Fall back to repo-discovered subscriptions; if none, ask the user. | -| Catalog call returns empty | Tenant has no Advisor coverage yet | Stop after Step 2; report empty tenant. || Recommendation list 401/403 | Auth not scoped to subscription | Tell user to run `az login` and verify subscription access. | +| Recommendations tool has no resource-scope filter | Tool only accepts a subscription | Pull unfiltered and post-filter the list to the Step 1b scope in-context. | +| Catalog call returns empty | Tenant has no Advisor coverage yet | Stop after Step 2; report empty tenant. | +| Recommendation list 401/403 | Auth not scoped to subscription | Tell user to run `az login` and verify subscription access. | | Aggregation tool errors on `group-by` | Field name not in supported list | Re-call with a value drawn from Step 2's catalog. | | IaC tool says "unknown resource" | Resource type not in its supported list | Skip the fix for that recommendation; keep the rest. | diff --git a/tests/azure-advisor/__snapshots__/triggers.test.ts.snap b/tests/azure-advisor/__snapshots__/triggers.test.ts.snap index 6c334ce1e..d190741f1 100644 --- a/tests/azure-advisor/__snapshots__/triggers.test.ts.snap +++ b/tests/azure-advisor/__snapshots__/triggers.test.ts.snap @@ -2,14 +2,13 @@ exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` { - "description": "Azure Advisor reviews and recommendations using available advisor_* MCP tools. WHEN: "run an advisor review", "check my Azure advisor recommendations", "summarize advisor findings", "what does Advisor say about my subscription", "give me an advisor health check", "audit my Azure resources with Advisor". USE FOR: read-only Advisor sweeps with catalog, recommendations, and IaC fixes. DO NOT USE FOR: changing resources, billing analysis (use azure-cost), or non-Advisor troubleshooting (use azure-diagnostics).", + "description": "Azure Advisor reviews resources and provides recommendations using advisor_* MCP tools. WHEN: "run an advisor review", "check my Azure advisor recommendations", "summarize advisor findings", "what does Advisor say about my subscription", "give me an advisor health check", "audit my Azure resources with Advisor". USE FOR: read-only sweeps with catalog, recommendations, and IaC fixes. DO NOT USE FOR: changing resources, billing analysis (use azure-cost), or non-Advisor troubleshooting (use azure-diagnostics).", "extractedKeywords": [ "about", "advisor", "advisor_", "analysis", "audit", - "available", "azure", "azure-cost", "azure-diagnostics", @@ -25,11 +24,13 @@ exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill descripti "health", "mcp", "non-advisor", + "provides", "read-only", "recommendations", "resources", "review", "reviews", + "security", "subscription", "summarize", "sweeps", @@ -51,7 +52,6 @@ exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill keywords "advisor_", "analysis", "audit", - "available", "azure", "azure-cost", "azure-diagnostics", @@ -67,11 +67,13 @@ exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill keywords "health", "mcp", "non-advisor", + "provides", "read-only", "recommendations", "resources", "review", "reviews", + "security", "subscription", "summarize", "sweeps", From 59e0776ae93d25a3f0b0a9af126a07d80164c94e Mon Sep 17 00:00:00 2001 From: Sharadhi Srikanth Date: Thu, 25 Jun 2026 17:00:59 +0530 Subject: [PATCH 07/14] docs: add Terraform type-mapping fallback note to resource-discovery Bicep/ARM provider types are literal extractions; the Terraform azurerm_ -> Microsoft.* mapping must be derived and can be non-obvious. When uncertain, fall back to the resource group filter rather than risk a wrong type filter that hides recommendations. --- plugin/skills/azure-advisor/references/resource-discovery.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/plugin/skills/azure-advisor/references/resource-discovery.md b/plugin/skills/azure-advisor/references/resource-discovery.md index e9e12609b..66fe8fef9 100644 --- a/plugin/skills/azure-advisor/references/resource-discovery.md +++ b/plugin/skills/azure-advisor/references/resource-discovery.md @@ -28,6 +28,11 @@ signals, in this order. Collect every distinct hit (a repo may define several): - Bicep `resource x 'Microsoft./@...'` → `Microsoft./` - ARM `"type": "Microsoft./"` - Terraform `resource "azurerm_"` → map to its `Microsoft.*` provider type + - Bicep and ARM give the provider type **literally** (extract it as-is). The Terraform + `azurerm_` → `Microsoft.*` mapping must be **derived** and is not always obvious + (e.g. `azurerm_linux_function_app` → `Microsoft.Web/sites`). If a Terraform type mapping + is uncertain, **do not guess** — fall back to the resource group filter for that resource + rather than risk a wrong type filter that hides real recommendations. 3. **Specific resource id(s)** — only when a fully-qualified ARM id is present in params/env (`/subscriptions/.../resourceGroups/.../providers/...`). From 3c79fb89ce35e8888c515e767780cabac037b8ad Mon Sep 17 00:00:00 2001 From: Sharadhi Srikanth Date: Tue, 30 Jun 2026 10:46:31 +0530 Subject: [PATCH 08/14] test: migrate azure-advisor to Vally eval suite Add evals/azure-advisor/eval.yaml covering review/recommendation/audit routing and response quality. Remove legacy Jest triggers test and snapshot. Drop README.md and reference Azure MCP server in SKILL.md description. --- evals/azure-advisor/eval.yaml | 154 ++++++++++++++++++ plugin/skills/azure-advisor/README.md | 53 ------ plugin/skills/azure-advisor/SKILL.md | 5 +- plugin/skills/azure-advisor/review/review.md | 4 +- .../__snapshots__/triggers.test.ts.snap | 87 ---------- tests/azure-advisor/triggers.test.ts | 98 ----------- 6 files changed, 157 insertions(+), 244 deletions(-) create mode 100644 evals/azure-advisor/eval.yaml delete mode 100644 plugin/skills/azure-advisor/README.md delete mode 100644 tests/azure-advisor/__snapshots__/triggers.test.ts.snap delete mode 100644 tests/azure-advisor/triggers.test.ts diff --git a/evals/azure-advisor/eval.yaml b/evals/azure-advisor/eval.yaml new file mode 100644 index 000000000..3173446db --- /dev/null +++ b/evals/azure-advisor/eval.yaml @@ -0,0 +1,154 @@ +# Vally eval config for the azure-advisor skill. +# +# Replaces the legacy Jest trigger tests (tests/azure-advisor/triggers.test.ts), +# which are deprecated per tests/README.md. This suite expresses the skill's +# scenarios as Vally stimuli graded against a real LLM agent run. +# +# Coverage (per tests/AGENTS.md): +# - Routing stimuli measure skill-invocation rate across the documented WHEN +# trigger phrases (5 runs each, 80% threshold), with early termination once the +# skill is invoked to keep runs cheap. +# - Response-quality stimuli (runs: 1) check the assistant output mentions Advisor +# recommendations and does not surface fatal errors. + +name: azure-advisor-integration-eval +description: | + Integration evaluation for the azure-advisor skill. Verifies routing for Advisor + review, recommendation-check, health-check, and audit prompts via skill-invocation + rate (5 runs, 80% threshold), plus response-quality checks for Advisor recommendation + output. + +tags: + type: integration + skill: azure-advisor + +environment: + skills: + - ../../plugin/skills/azure-advisor + +config: + runs: 5 + timeout: "10m" + executor: integration-test-agent-runner + model: claude-sonnet-4.6 + +scoring: + threshold: 0.8 + +stimuli: + # ═══════════════════════════════════════════ + # Skill routing prompts + # ═══════════════════════════════════════════ + + # ── run-advisor-review (smoke) ── + - name: "Run an Advisor review" + prompt: "Run an Azure Advisor review of my subscription" + tags: + type: integration + tier: smoke + cost: llm + area: routing + earlyTerminate: '[{"type":"skill-call","skill":"azure-advisor"},{"type":"tool-call-count","count":3}]' + graders: + - type: skill-invocation + config: + required: + - azure-advisor + - type: output-not-matches + config: + pattern: "(?i)fatal error|unhandled exception|stack trace" + + # ── check-recommendations ── + - name: "Check Advisor recommendations" + prompt: "Check my Azure Advisor recommendations" + tags: + type: integration + tier: full + cost: llm + area: routing + earlyTerminate: '[{"type":"skill-call","skill":"azure-advisor"},{"type":"tool-call-count","count":3}]' + graders: + - type: skill-invocation + config: + required: + - azure-advisor + - type: output-not-matches + config: + pattern: "(?i)fatal error|unhandled exception|stack trace" + + # ── what-does-advisor-say ── + - name: "What does Advisor say about my subscription" + prompt: "What does Advisor say about my Azure subscription?" + tags: + type: integration + tier: full + cost: llm + area: routing + earlyTerminate: '[{"type":"skill-call","skill":"azure-advisor"},{"type":"tool-call-count","count":3}]' + graders: + - type: skill-invocation + config: + required: + - azure-advisor + - type: output-not-matches + config: + pattern: "(?i)fatal error|unhandled exception|stack trace" + + # ── advisor-health-check ── + - name: "Azure Advisor health check" + prompt: "Give me an Azure Advisor health check" + tags: + type: integration + tier: full + cost: llm + area: routing + earlyTerminate: '[{"type":"skill-call","skill":"azure-advisor"},{"type":"tool-call-count","count":3}]' + graders: + - type: skill-invocation + config: + required: + - azure-advisor + - type: output-not-matches + config: + pattern: "(?i)fatal error|unhandled exception|stack trace" + + # ── audit-with-advisor ── + - name: "Audit resources with Advisor" + prompt: "Audit my Azure resources with Advisor" + tags: + type: integration + tier: full + cost: llm + area: routing + earlyTerminate: '[{"type":"skill-call","skill":"azure-advisor"},{"type":"tool-call-count","count":3}]' + graders: + - type: skill-invocation + config: + required: + - azure-advisor + - type: output-not-matches + config: + pattern: "(?i)fatal error|unhandled exception|stack trace" + + # ═══════════════════════════════════════════ + # Response quality tests + # ═══════════════════════════════════════════ + + # ── review-mentions-recommendations ── + - name: "Advisor review mentions recommendations" + prompt: "Run an Azure Advisor review of my subscription" + config: + runs: 1 + tags: + type: integration + tier: full + cost: llm + area: response-quality + graders: + - type: output-matches + config: + pattern: "(?i)recommendation|advisor" + - type: completed + - type: output-not-matches + config: + pattern: "(?i)fatal error|unhandled exception|stack trace" diff --git a/plugin/skills/azure-advisor/README.md b/plugin/skills/azure-advisor/README.md deleted file mode 100644 index 73c8e6720..000000000 --- a/plugin/skills/azure-advisor/README.md +++ /dev/null @@ -1,53 +0,0 @@ -# Azure Advisor Skill — Contributor Guide - -> This README is for **contributors** maintaining the `azure-advisor` skill. It is not -> loaded by the agent at runtime — the agent reads [SKILL.md](SKILL.md) and the -> capability files. Keep onboarding notes here. - -## What this skill is - -`azure-advisor` is a **product area**, not a single workflow. It groups multiple -Advisor-related capabilities behind one router so the agent can pick the right one by -user intent. Each capability is a self-contained folder; shared concerns live in the -[references](references/capability-routing.md) folder. - -## Folder map - -| Path | Purpose | -|------|---------| -| [SKILL.md](SKILL.md) | **Router.** Frontmatter (makes the skill discoverable) + a capability table that routes intent to a capability file. Keep it thin. | -| [references/capability-routing.md](references/capability-routing.md) | **Shared.** Capability-agnostic routing docs reused by every capability: resolve which `advisor_*` MCP tool to call for each capability (catalog, recommendations, summary, IaC fix). | -| [references/subscription-discovery.md](references/subscription-discovery.md) | How to resolve the target subscription from repo config / env without hardcoding. | -| [references/resource-discovery.md](references/resource-discovery.md) | How to narrow a review to the resources defined in this repo (resource group / type / id) without hardcoding. | -| [review/review.md](review/review.md) | **Capability:** holistic read-only Advisor sweep (the only file the agent reads for this capability). | - -## Conventions (match the rest of the repo) - -- **One folder per capability**, named after the capability (e.g. `review/`, - `summarize/`). This mirrors `microsoft-foundry` (`quota/quota.md`, `rbac/rbac.md`). -- **Capability file is named after its folder**: `review/review.md`, - `summarize/summarize.md`. The agent loads this file when the capability matches. -- **Shared docs go in `references/`**, never inside a single capability folder — so the - next capability can reuse them without a refactor. -- **Capability-specific references** (used by only one capability) may live in a nested - `references/` inside that capability folder, e.g. `review/references/…`. -- **Frontmatter lives only in `SKILL.md`** (`name`, `description`, `license`, - `metadata.author`, `metadata.version`). Capability files are plain Markdown. - -## How to add a new capability - -1. **Create the folder + file:** `azure-advisor//.md`. -2. **Reuse shared references** instead of re-defining them — link to - [capability-routing.md](references/capability-routing.md) and - [subscription-discovery.md](references/subscription-discovery.md) where relevant. -3. **Register it in the router:** add a row to the **Capabilities** table in - [SKILL.md](SKILL.md) (and remove it from the Roadmap list there if listed). -4. **Keep it read-only by default.** Advisor workflows propose IaC fixes; they don't - mutate cloud state. Document any exception explicitly. -5. **Do not edit `metadata.version`** in [SKILL.md](SKILL.md) — it stays - `0.0.0-placeholder`; NBGV stamps the real version at build time. - -## Where this ships from - -This is the upstream `microsoft/GitHub-Copilot-for-Azure` repository. Land changes here; -they are released from this repo and mirrored downstream automatically. diff --git a/plugin/skills/azure-advisor/SKILL.md b/plugin/skills/azure-advisor/SKILL.md index 28bb7cbf5..81e7d489d 100644 --- a/plugin/skills/azure-advisor/SKILL.md +++ b/plugin/skills/azure-advisor/SKILL.md @@ -1,6 +1,6 @@ --- name: azure-advisor -description: "Azure Advisor reviews resources and provides recommendations using advisor_* MCP tools. WHEN: \"run an advisor review\", \"check my Azure advisor recommendations\", \"summarize advisor findings\", \"what does Advisor say about my subscription\", \"give me an advisor health check\", \"audit my Azure resources with Advisor\". USE FOR: read-only sweeps with catalog, recommendations, and IaC fixes. DO NOT USE FOR: changing resources, billing analysis (use azure-cost), or non-Advisor troubleshooting (use azure-diagnostics)." +description: "Azure Advisor reviews resources and provides recommendations using the Azure MCP server's advisor_* tools. WHEN: \"run an advisor review\", \"check my Azure advisor recommendations\", \"summarize advisor findings\", \"what does Advisor say about my subscription\", \"give me an advisor health check\", \"audit my Azure resources with Advisor\". USE FOR: read-only sweeps with catalog, recommendations, and IaC fixes. DO NOT USE FOR: changing resources, billing analysis (use azure-cost), or non-Advisor troubleshooting (use azure-diagnostics)." license: MIT metadata: author: Microsoft @@ -15,9 +15,6 @@ the connected Azure MCP server exposes. Routing is by *capability* (catalog, recommendations, summary, IaC fix), not by hard-coded tool names, so the skill stays useful as new advisor tools land. -> 🛠️ **Contributing a new capability?** See [README.md](README.md) for the folder map -> and a step-by-step recipe. - ## Pre-Execution Requirements Inspect the available `advisor_*` MCP tools and their parameters before running a diff --git a/plugin/skills/azure-advisor/review/review.md b/plugin/skills/azure-advisor/review/review.md index 13f92d342..c09c73745 100644 --- a/plugin/skills/azure-advisor/review/review.md +++ b/plugin/skills/azure-advisor/review/review.md @@ -99,7 +99,7 @@ Reply in chat with this structure (no files written). **Single subscription:** -``` +```text ## Azure Advisor Review **Subscription:** (from ) @@ -123,7 +123,7 @@ Reply in chat with this structure (no files written). **All subscriptions** — group the report by environment bucket (prod first, then staging, test, dev, other), with a roll-up at the top: -``` +```text ## Azure Advisor Review — All Subscriptions **Scope:** S subscriptions across E environments diff --git a/tests/azure-advisor/__snapshots__/triggers.test.ts.snap b/tests/azure-advisor/__snapshots__/triggers.test.ts.snap deleted file mode 100644 index d190741f1..000000000 --- a/tests/azure-advisor/__snapshots__/triggers.test.ts.snap +++ /dev/null @@ -1,87 +0,0 @@ -// Jest Snapshot v1, https://jestjs.io/docs/snapshot-testing - -exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill description triggers match snapshot 1`] = ` -{ - "description": "Azure Advisor reviews resources and provides recommendations using advisor_* MCP tools. WHEN: "run an advisor review", "check my Azure advisor recommendations", "summarize advisor findings", "what does Advisor say about my subscription", "give me an advisor health check", "audit my Azure resources with Advisor". USE FOR: read-only sweeps with catalog, recommendations, and IaC fixes. DO NOT USE FOR: changing resources, billing analysis (use azure-cost), or non-Advisor troubleshooting (use azure-diagnostics).", - "extractedKeywords": [ - "about", - "advisor", - "advisor_", - "analysis", - "audit", - "azure", - "azure-cost", - "azure-diagnostics", - "billing", - "catalog", - "changing", - "check", - "cli", - "does", - "findings", - "fixes", - "give", - "health", - "mcp", - "non-advisor", - "provides", - "read-only", - "recommendations", - "resources", - "review", - "reviews", - "security", - "subscription", - "summarize", - "sweeps", - "tools", - "troubleshooting", - "using", - "what", - "when", - "with", - ], - "name": "azure-advisor", -} -`; - -exports[`azure-advisor - Trigger Tests Trigger Keywords Snapshot skill keywords match snapshot 1`] = ` -[ - "about", - "advisor", - "advisor_", - "analysis", - "audit", - "azure", - "azure-cost", - "azure-diagnostics", - "billing", - "catalog", - "changing", - "check", - "cli", - "does", - "findings", - "fixes", - "give", - "health", - "mcp", - "non-advisor", - "provides", - "read-only", - "recommendations", - "resources", - "review", - "reviews", - "security", - "subscription", - "summarize", - "sweeps", - "tools", - "troubleshooting", - "using", - "what", - "when", - "with", -] -`; diff --git a/tests/azure-advisor/triggers.test.ts b/tests/azure-advisor/triggers.test.ts deleted file mode 100644 index aa8f84996..000000000 --- a/tests/azure-advisor/triggers.test.ts +++ /dev/null @@ -1,98 +0,0 @@ -/** - * Trigger Tests for azure-advisor - * - * Tests that verify the skill triggers on appropriate prompts - * and does NOT trigger on unrelated prompts. - * - */ - -import { TriggerMatcher } from "../utils/trigger-matcher"; -import { loadSkill, LoadedSkill } from "../utils/skill-loader"; - -const SKILL_NAME = "azure-advisor"; - -describe(`${SKILL_NAME} - Trigger Tests`, () => { - let triggerMatcher: TriggerMatcher; - let skill: LoadedSkill; - - beforeAll(async () => { - skill = await loadSkill(SKILL_NAME); - triggerMatcher = new TriggerMatcher(skill); - }); - - describe("Should Trigger - Advisor Review", () => { - const advisorPrompts: string[] = [ - "Run an Azure Advisor review of my subscription", - "Check my Azure Advisor recommendations", - "Summarize the Advisor findings for my subscription", - "What does Advisor say about my Azure subscription?", - "Give me an Azure Advisor health check", - "Audit my Azure resources with Advisor", - "Show me the top high-impact Azure Advisor recommendations", - "Aggregate my Advisor recommendations by category", - ]; - - test.each(advisorPrompts)( - 'triggers on: "%s"', - (prompt) => { - const result = triggerMatcher.shouldTrigger(prompt); - expect(result.triggered).toBe(true); - expect(result.matchedKeywords.length).toBeGreaterThanOrEqual(2); - } - ); - }); - - describe("Should NOT Trigger", () => { - const shouldNotTriggerPrompts: string[] = [ - "What is the weather today?", - "Help me write a poem", - "Explain quantum computing", - "How do I write a Python web scraper?", - "Analyze my Azure costs and spending trends", - "Troubleshoot my Azure App Service deployment failure", - "Help me set up Azure Key Vault for secrets management", - "Deploy my web app to Azure App Service", - "Set up Azure RBAC role assignments for my team", - ]; - - test.each(shouldNotTriggerPrompts)( - 'does not trigger on: "%s"', - (prompt) => { - const result = triggerMatcher.shouldTrigger(prompt); - expect(result.triggered).toBe(false); - } - ); - }); - - describe("Trigger Keywords Snapshot", () => { - test("skill keywords match snapshot", () => { - expect(triggerMatcher.getKeywords()).toMatchSnapshot(); - }); - - test("skill description triggers match snapshot", () => { - expect({ - name: skill.metadata.name, - description: skill.metadata.description, - extractedKeywords: triggerMatcher.getKeywords(), - }).toMatchSnapshot(); - }); - }); - - describe("Edge Cases", () => { - test("handles empty prompt", () => { - const result = triggerMatcher.shouldTrigger(""); - expect(result.triggered).toBe(false); - }); - - test("is case insensitive", () => { - const result1 = triggerMatcher.shouldTrigger( - "RUN AN AZURE ADVISOR REVIEW" - ); - const result2 = triggerMatcher.shouldTrigger( - "run an azure advisor review" - ); - expect(result1.triggered).toBe(true); - expect(result1.triggered).toBe(result2.triggered); - }); - }); -}); From f8ee26d4c9ab77647d620d4e3da27edf87062187 Mon Sep 17 00:00:00 2001 From: Sharadhi Srikanth Date: Tue, 30 Jun 2026 10:49:57 +0530 Subject: [PATCH 09/14] fix: trim azure-advisor description under Copilot CLI char budget --- plugin/skills/azure-advisor/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugin/skills/azure-advisor/SKILL.md b/plugin/skills/azure-advisor/SKILL.md index 81e7d489d..e30420b59 100644 --- a/plugin/skills/azure-advisor/SKILL.md +++ b/plugin/skills/azure-advisor/SKILL.md @@ -1,6 +1,6 @@ --- name: azure-advisor -description: "Azure Advisor reviews resources and provides recommendations using the Azure MCP server's advisor_* tools. WHEN: \"run an advisor review\", \"check my Azure advisor recommendations\", \"summarize advisor findings\", \"what does Advisor say about my subscription\", \"give me an advisor health check\", \"audit my Azure resources with Advisor\". USE FOR: read-only sweeps with catalog, recommendations, and IaC fixes. DO NOT USE FOR: changing resources, billing analysis (use azure-cost), or non-Advisor troubleshooting (use azure-diagnostics)." +description: "Azure Advisor reviews resources and provides recommendations using Azure MCP advisor_* tools. WHEN: \"run an advisor review\", \"check my Azure advisor recommendations\", \"summarize advisor findings\", \"what does Advisor say about my subscription\", \"give me an advisor health check\", \"audit my Azure resources with Advisor\". USE FOR: read-only catalog, recommendations, and IaC fixes. DO NOT USE FOR: changing resources, billing (use azure-cost), or non-Advisor issues (use azure-diagnostics)." license: MIT metadata: author: Microsoft From 6fadac7ae68ed34abc2f5a34d9cd36cb01e85f4e Mon Sep 17 00:00:00 2001 From: Sharadhi Srikanth Date: Tue, 30 Jun 2026 11:02:38 +0530 Subject: [PATCH 10/14] test: add boundary stimuli for azure-advisor adjacent-service routing --- evals/azure-advisor/eval.yaml | 57 +++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/evals/azure-advisor/eval.yaml b/evals/azure-advisor/eval.yaml index 3173446db..e299cdcd4 100644 --- a/evals/azure-advisor/eval.yaml +++ b/evals/azure-advisor/eval.yaml @@ -130,6 +130,63 @@ stimuli: config: pattern: "(?i)fatal error|unhandled exception|stack trace" + # ═══════════════════════════════════════════ + # Boundary (negative) routing — adjacent services from + # the skill's DO NOT USE FOR section must NOT route to azure-advisor. + # `disallowed` asserts the skill stays out of these requests. + # ═══════════════════════════════════════════ + + # ── boundary-cost ── + - name: "Cost query does not route to advisor" + prompt: "Analyze my Azure subscription costs and spending trends" + tags: + type: integration + tier: smoke + cost: llm + area: routing + graders: + - type: skill-invocation + config: + disallowed: + - azure-advisor + - type: output-not-matches + config: + pattern: "(?i)fatal error|unhandled exception|stack trace" + + # ── boundary-diagnostics ── + - name: "Diagnostics query does not route to advisor" + prompt: "My App Service keeps returning 500 errors, help me troubleshoot it" + tags: + type: integration + tier: full + cost: llm + area: routing + graders: + - type: skill-invocation + config: + disallowed: + - azure-advisor + - type: output-not-matches + config: + pattern: "(?i)fatal error|unhandled exception|stack trace" + + # ── boundary-rbac ── + - name: "RBAC query does not route to advisor" + prompt: "Grant my team Reader access to the production resource group" + tags: + type: integration + tier: full + cost: llm + area: routing + graders: + - type: skill-invocation + config: + disallowed: + - azure-advisor + - type: output-not-matches + config: + pattern: "(?i)fatal error|unhandled exception|stack trace" + # ═══════════════════════════════════════════ # Response quality tests # ═══════════════════════════════════════════ From 96efad2954988b8a671e2f9dafbcef20a4bce927 Mon Sep 17 00:00:00 2001 From: Sharadhi Srikanth Date: Tue, 30 Jun 2026 11:22:20 +0530 Subject: [PATCH 11/14] test: add Key Vault and App Service boundary stimuli for azure-advisor --- evals/azure-advisor/eval.yaml | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/evals/azure-advisor/eval.yaml b/evals/azure-advisor/eval.yaml index e299cdcd4..9c275517f 100644 --- a/evals/azure-advisor/eval.yaml +++ b/evals/azure-advisor/eval.yaml @@ -187,6 +187,40 @@ stimuli: config: pattern: "(?i)fatal error|unhandled exception|stack trace" + # ── boundary-keyvault ── + - name: "Key Vault query does not route to advisor" + prompt: "Store a new secret in my Azure Key Vault" + tags: + type: integration + tier: full + cost: llm + area: routing + graders: + - type: skill-invocation + config: + disallowed: + - azure-advisor + - type: output-not-matches + config: + pattern: "(?i)fatal error|unhandled exception|stack trace" + + # ── boundary-appservice ── + - name: "App Service deploy does not route to advisor" + prompt: "Deploy my web app to Azure App Service" + tags: + type: integration + tier: full + cost: llm + area: routing + graders: + - type: skill-invocation + config: + disallowed: + - azure-advisor + - type: output-not-matches + config: + pattern: "(?i)fatal error|unhandled exception|stack trace" + # ═══════════════════════════════════════════ # Response quality tests # ═══════════════════════════════════════════ From 515a56f27b5cf474c1ea4d94881c9358d107d19e Mon Sep 17 00:00:00 2001 From: Sharadhi Srikanth Date: Wed, 1 Jul 2026 12:36:51 +0530 Subject: [PATCH 12/14] test: drop dead environment block and add per-stimulus skill tag to azure-advisor eval --- evals/azure-advisor/eval.yaml | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/evals/azure-advisor/eval.yaml b/evals/azure-advisor/eval.yaml index 9c275517f..8f37fc660 100644 --- a/evals/azure-advisor/eval.yaml +++ b/evals/azure-advisor/eval.yaml @@ -22,10 +22,6 @@ tags: type: integration skill: azure-advisor -environment: - skills: - - ../../plugin/skills/azure-advisor - config: runs: 5 timeout: "10m" @@ -45,6 +41,7 @@ stimuli: prompt: "Run an Azure Advisor review of my subscription" tags: type: integration + skill: azure-advisor tier: smoke cost: llm area: routing @@ -63,6 +60,7 @@ stimuli: prompt: "Check my Azure Advisor recommendations" tags: type: integration + skill: azure-advisor tier: full cost: llm area: routing @@ -81,6 +79,7 @@ stimuli: prompt: "What does Advisor say about my Azure subscription?" tags: type: integration + skill: azure-advisor tier: full cost: llm area: routing @@ -99,6 +98,7 @@ stimuli: prompt: "Give me an Azure Advisor health check" tags: type: integration + skill: azure-advisor tier: full cost: llm area: routing @@ -117,6 +117,7 @@ stimuli: prompt: "Audit my Azure resources with Advisor" tags: type: integration + skill: azure-advisor tier: full cost: llm area: routing @@ -141,6 +142,7 @@ stimuli: prompt: "Analyze my Azure subscription costs and spending trends" tags: type: integration + skill: azure-advisor tier: smoke cost: llm area: routing @@ -158,6 +160,7 @@ stimuli: prompt: "My App Service keeps returning 500 errors, help me troubleshoot it" tags: type: integration + skill: azure-advisor tier: full cost: llm area: routing @@ -175,6 +178,7 @@ stimuli: prompt: "Grant my team Reader access to the production resource group" tags: type: integration + skill: azure-advisor tier: full cost: llm area: routing @@ -192,6 +196,7 @@ stimuli: prompt: "Store a new secret in my Azure Key Vault" tags: type: integration + skill: azure-advisor tier: full cost: llm area: routing @@ -209,6 +214,7 @@ stimuli: prompt: "Deploy my web app to Azure App Service" tags: type: integration + skill: azure-advisor tier: full cost: llm area: routing @@ -232,6 +238,7 @@ stimuli: runs: 1 tags: type: integration + skill: azure-advisor tier: full cost: llm area: response-quality From b512a45300d67a750a6a6b9a80f48d08c572df6e Mon Sep 17 00:00:00 2001 From: Sharadhi Srikanth Date: Wed, 1 Jul 2026 12:51:57 +0530 Subject: [PATCH 13/14] test: add advisor tool-call trajectory tests; docs: drop duplicated Constraints from SKILL.md --- evals/azure-advisor/eval.yaml | 53 ++++++++++++++++++++++++++++ plugin/skills/azure-advisor/SKILL.md | 6 ---- 2 files changed, 53 insertions(+), 6 deletions(-) diff --git a/evals/azure-advisor/eval.yaml b/evals/azure-advisor/eval.yaml index 8f37fc660..91710ab7a 100644 --- a/evals/azure-advisor/eval.yaml +++ b/evals/azure-advisor/eval.yaml @@ -227,6 +227,59 @@ stimuli: config: pattern: "(?i)fatal error|unhandled exception|stack trace" + # ═══════════════════════════════════════════ + # Tool-call trajectory — validate behavior up to the point the agent + # invokes an Advisor MCP tool. Early-terminates as soon as an `advisor_` + # tool is called (or after a small tool-call cap as a safety net), then the + # `tool-calls` grader asserts an `advisor_` tool was actually executed. + # ═══════════════════════════════════════════ + + # ── review-calls-advisor-tool ── + - name: "Advisor review reaches an advisor tool call" + prompt: "Run an Azure Advisor review of my subscription" + tags: + type: integration + skill: azure-advisor + tier: smoke + cost: llm + area: behavior + earlyTerminate: '[{"type":"tool-call-match","toolPattern":"advisor_","argsPattern":".*"},{"type":"tool-call-count","count":8}]' + graders: + - type: skill-invocation + config: + required: + - azure-advisor + - type: tool-calls + config: + required: + - name: "advisor_" + - type: output-not-matches + config: + pattern: "(?i)fatal error|unhandled exception|stack trace" + + # ── check-recommendations-calls-advisor-tool ── + - name: "Recommendation check reaches an advisor tool call" + prompt: "Check my Azure Advisor recommendations" + tags: + type: integration + skill: azure-advisor + tier: full + cost: llm + area: behavior + earlyTerminate: '[{"type":"tool-call-match","toolPattern":"advisor_","argsPattern":".*"},{"type":"tool-call-count","count":8}]' + graders: + - type: skill-invocation + config: + required: + - azure-advisor + - type: tool-calls + config: + required: + - name: "advisor_" + - type: output-not-matches + config: + pattern: "(?i)fatal error|unhandled exception|stack trace" + # ═══════════════════════════════════════════ # Response quality tests # ═══════════════════════════════════════════ diff --git a/plugin/skills/azure-advisor/SKILL.md b/plugin/skills/azure-advisor/SKILL.md index e30420b59..32f2e5f4a 100644 --- a/plugin/skills/azure-advisor/SKILL.md +++ b/plugin/skills/azure-advisor/SKILL.md @@ -56,9 +56,3 @@ reusing the shared references above. Add a row to the table above when one ships - **governance** — Advisor operational-excellence / governance reviews - **security** — Advisor security-category reviews - **performance** — Advisor performance-category reviews - -## Constraints - -- ❌ **Never** hardcode a subscription id, tenant id, or resource group. -- ❌ **Never** modify Azure state — Advisor workflows in this skill are read + suggest only. -- ❌ **Never** call a tool whose name does not contain `advisor_` as a substitute; if no capability matches, report it and skip. From b649cddf40c00ab579e26d968d6c2cf434a1163e Mon Sep 17 00:00:00 2001 From: Sharadhi Srikanth Date: Fri, 3 Jul 2026 00:04:21 +0530 Subject: [PATCH 14/14] docs: remove not-yet-implemented roadmap section from azure-advisor skill --- plugin/skills/azure-advisor/SKILL.md | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/plugin/skills/azure-advisor/SKILL.md b/plugin/skills/azure-advisor/SKILL.md index 32f2e5f4a..cefee34ca 100644 --- a/plugin/skills/azure-advisor/SKILL.md +++ b/plugin/skills/azure-advisor/SKILL.md @@ -42,17 +42,3 @@ skill when they match the task:** | Capability | When to Use | Reference | |-----------|-------------|-----------| | **review** | Run a holistic, read-only Advisor sweep across one subscription — or **all** subscriptions classified by environment (dev/staging/prod) — probing the catalog, pulling active recommendations, aggregating by category/impact, spotlighting high-impact items, and proposing IaC fix snippets. | [review](review/review.md) | - -### Roadmap (not yet implemented) - -Planned capabilities will each be added as a sibling folder under `azure-advisor/`, -reusing the shared references above. Add a row to the table above when one ships: - -- **summarize** — focused recommendation summarization / aggregation views -- **resource-analysis** — per-resource Advisor analysis and drill-down -- **greenfield** — Advisor-informed guidance for new/empty subscriptions -- **cost** — Advisor cost-category optimization (coordinates with `azure-cost`) -- **reliability** — Advisor reliability-category reviews -- **governance** — Advisor operational-excellence / governance reviews -- **security** — Advisor security-category reviews -- **performance** — Advisor performance-category reviews