Add --no-prompt to azd env new commands to prevent interactive blocking#2
Closed
tmeschter wants to merge 580 commits into
Closed
Add --no-prompt to azd env new commands to prevent interactive blocking#2tmeschter wants to merge 580 commits into
tmeschter wants to merge 580 commits into
Conversation
…icrosoft#1231) * Initial plan * Remove foundry-mcp from plugin/.mcp.json and update observe references Co-authored-by: JasonYeMSFT <39359541+JasonYeMSFT@users.noreply.github.com> * Bump microsoft-foundry skill version to 1.0.4 Co-authored-by: JasonYeMSFT <39359541+JasonYeMSFT@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: JasonYeMSFT <39359541+JasonYeMSFT@users.noreply.github.com>
* Update analyze-test-run Update the analyze-test-run agentic workflow: 1. Update the workflow to run automatically when scheduled runs of "Integration Tests - all" complete. Manual runs are still supported. 2. Update the analyze-test-run SKILL.md to try and avoid creating duplicate issues when the same failure is seen multiple times. Note that actions-lock.json and analyze-test-run.lock.yml are produced by compiling analyze-test-run.md with `gh aw compile analyze-test-run.md`; these are not meant to be manually edited. Documentation on Agentic Workflows can be found at https://github.github.com/gh-aw. * Fix label consistency between agentic workflow and skill Update analyze-test-run.md workflow safe-outputs labels from [bug, test-failure] to [bug, integration-test] to match the analyze-test-run skill and issue template. Bump analyze-test-run SKILL.md version from 1.0.0 to 1.0.1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…icrosoft#1237) * Initial plan * Add MCP Server Authorization Failure section to Troubleshooting.md Co-authored-by: saikoumudi <22682497+saikoumudi@users.noreply.github.com> * Rename section to Foundry MCP server authorization failure, remove delete commands Co-authored-by: saikoumudi <22682497+saikoumudi@users.noreply.github.com> * Update Troubleshooting.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: saikoumudi <22682497+saikoumudi@users.noreply.github.com> Co-authored-by: Sai Koumudi Kaluvakolanu <saikoumudi@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…mption migration (microsoft#1211) * Add azure-upgrade skill for Azure Functions Consumption to Flex Consumption migration * feat: add azure-upgrade tests and improve SKILL.md frontmatter - Add unit, trigger, and integration tests for azure-upgrade skill - Add create-function-app-consumption.sh setup script - Condense SKILL.md description to meet 60-word limit - Register azure-upgrade in tests/skills.json * fix integration tests * added upgrade skills to skills.json * run sensei on azure-upgrade, refactor azure-diagnostics skill description, update integration test * added invocation tests and removed end to end tests * bump version and trigger snapshot for azure-diagnostics, sensei: improve azure-diagnostics frontmatter --------- Co-authored-by: Koumudi Kaluvakolanu <saikoumudi@gmail.com>
…and more. (microsoft#1219) * Fix Issue Triage workflow: add `github.token` fallback for missing `COPILOT_GITHUB_TOKEN` (microsoft#6) * Initial plan * Fix issue triage workflow: add github.token fallback for COPILOT_GITHUB_TOKEN The Issue Triage workflow was failing at the secret validation step because COPILOT_GITHUB_TOKEN was not configured. This adds github.token as a fallback in all 4 places where COPILOT_GITHUB_TOKEN is used for authentication: - agent job: validate-secret step and Execute step - detection job: validate-secret step and Execute step This is consistent with the existing fallback patterns in the workflow (e.g., secrets.GH_AW_GITHUB_MCP_SERVER_TOKEN || secrets.GH_AW_GITHUB_TOKEN || secrets.GITHUB_TOKEN) Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> * Fix/observe deploy loop logic (microsoft#7) * Fix broken auto-create evaluators step in deploy/observe loop The 'Auto-create evaluators & evaluation dataset' step was being skipped when the monolithic agent-observability-loop skill was split into separate deploy and observe skills. Neither skill owned the auto-create step, causing post-deploy users to jump directly to evaluation. Changes: - deploy.md: Replace generic 'set up evaluation?' prompt with automatic 6-step evaluator & dataset creation matching the reference behavior - observe.md: Add Loop Overview, fix entry points to route post-deploy users through auto-setup, add evaluator existence check - deploy-and-setup.md: Make auto-create primary content, demote deploy section to prerequisites Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add content tests for observe/deploy loop logic Tests verify: - observe.md has Loop Overview, post-deploy entry points, evaluator existence checks, behavioral rules, and all reference files - deploy.md has auto-create evaluators section that is automatic (not optional), includes evaluator categories, LLM-judge, artifact persistence, and routes to observe skill Step 2 - deploy-and-setup.md has auto-create as primary content with proper evaluator selection, dataset generation, and user prompt 49 tests total (29 observe + 20 deploy), all passing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: trigger CI checks * Fix * add local dataset gen enforcement * Merge * feat: prefer monitor_resource_log_query and local datasets - Replace azure-kusto delegation with monitor_resource_log_query for App Insights KQL queries in trace.md and troubleshoot.md - Mark evaluation_dataset_create as not available (MCP upload not ready) - Replace server-side dataset sections with local JSONL workflow - Update mcp-gap-analysis.md to reflect practical tool availability Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make dataset upload restriction more agent-proof - Add Do NOT section at top of trace-to-dataset.md (before Overview) - Add behavioral rule microsoft#7 to eval-datasets.md: never upload to cloud - Remove Option A/B structure; Step 4 is now local JSONL only - Eliminates subtle strikethrough formatting that agents miss Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix link * fix: make auto-create evaluators an explicit numbered step - Hosted workflow: add Step 10 after Step 9 with DO NOT stop gate - Prompt workflow: add Step 5 after Step 4 with DO NOT stop gate - Both link to existing After Deployment section as implementation - Prevents agents from treating evaluator setup as optional appendix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat: add dataset update loop with optimization guardrails - Add Dataset Update Loop (eval→compare→analyze→optimize→re-eval) to dataset-versioning.md after Creating a New Version - Add guardrails: never remove dataset rows or weaken evaluators to recover scores after dataset expansion - Add same guardrail to observe optimize-deploy.md Step 6 - Add behavioral rule microsoft#8 to eval-datasets.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add subscription parameter warning to trace-related skills Always pass subscription explicitly to Azure MCP tools like monitor_resource_log_query — they don't extract it from resource IDs. Added to trace.md, troubleshoot.md, and trace-to-dataset.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: make customEvents-to-traces eval correlation more obvious - Add Key Concept section to trace-to-dataset.md explaining that eval results live in customEvents (not dependencies) and the join key is gen_ai.response.id - Add table showing dependencies vs customEvents join pattern - Cross-reference trace skill's eval-correlation.md from both trace-to-dataset.md and eval-datasets.md Related Skills Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve cross-references and add KQL parse_json warning 1. Add parse_json(customDimensions) warning to Do NOT section 2. Add Related References section with skill-root paths 3. Add skill-root path hints to all cross-skill links 4. Add observe + trace to SKILL.md sub-skill routing table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve hosted agent KQL patterns and content extraction - Add Hosted Agent Harvest template (requests→dependencies join) - Fix Hosted Agent Attributes: appear on both requests and traces - Add gen_ai.agent.name duality callout (Foundry name vs class name) - Remove incorrect azure.ai.agentserver.agent_name fallback from dependencies queries - Document gen_ai.input.messages/gen_ai.output.messages as content source - Add operation_ParentId join example to Span Correlation section - Update search-traces.md hosted agent query to use requests entry point Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: improve trace sub-skills for hosted agent KQL patterns - search-traces: fix hosted agent query to group by operation_ParentId - conversation-detail: add content extraction from invoke_agent spans (gen_ai.input.messages / gen_ai.output.messages) - analyze-failures: add hosted agent gen_ai.agent.name duality warning and hosted agent variant query using requests→dependencies join - analyze-latency: same hosted agent warning and variant query - kql-templates: expand requests table description as preferred entry point; add gen_ai.input/output.messages to attributes table - trace.md: reword rule 6 to clarify hosted vs prompt agent filtering Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: restore routing keywords and update trigger snapshots - Add back critical routing keywords to SKILL.md description (578→779 chars): role assignment, permissions, capacity, region, deployment failure, AI Services, Cognitive Services, provision, knowledge index, monitoring, customize, onboard, availability - Update trigger test snapshots for new keyword set (24 snapshots) - Fix deploy trigger test: Docker IS our capability (remove false negative) - Fix customize-deployment tests: ensure prompts have ≥2 keyword matches - Fix deploy-model-optimal-region tests: use longer prompts for HA/PTU Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add 'create AI Services' to description for resource/create test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: bump microsoft-foundry version to 1.0.2 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(eval-datasets): enable Foundry dataset sync via MCP tools - Add Step 5 (Sync to Foundry) to trace-to-dataset pipeline using evaluation_dataset_create with connectionName and project_connection tools - Add server-side version discovery via evaluation_dataset_versions_get - Add dual experiment types to dataset-comparison (agent vs dataset comparison) - Update mcp-gap-analysis: mark resolved tools, update workarounds - Add AzureBlob to project connections reference - Bump microsoft-foundry version to 1.0.3 - Fix upstream section heading changes in unit tests - Update trigger snapshots for upstream keyword changes Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor(dataset-comparison): focus on dataset-version comparison only Remove agent comparison experiment type from dataset-comparison flow. Agent comparison belongs in the observe/eval loop, not the dataset skill. Update all examples to use dataset versions as baseline/treatment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove Playwright MCP server until skills require it (microsoft#1200) * Collapse token analysis comment (microsoft#1147) * update region-availability in prepare/validate/deploy skill (microsoft#1083) * update region-availability in prepare/deploy skill * update * update * fix * update date * Update plugin/skills/azure-deploy/references/region-availability.md * fix ci failure * bump version * build(deps): bump @github/copilot and @github/copilot-sdk in /tests (microsoft#1201) Bumps [@github/copilot](https://github.com/github/copilot-cli) to 1.0.2 and updates ancestor dependency [@github/copilot-sdk](https://github.com/github/copilot-sdk). These dependencies need to be updated together. Updates `@github/copilot` from 0.0.414 to 1.0.2 - [Release notes](https://github.com/github/copilot-cli/releases) - [Changelog](https://github.com/github/copilot-cli/blob/main/changelog.md) - [Commits](github/copilot-cli@v0.0.414...v1.0.2) Updates `@github/copilot-sdk` from 0.1.26 to 0.1.32 - [Release notes](https://github.com/github/copilot-sdk/releases) - [Changelog](https://github.com/github/copilot-sdk/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/copilot-sdk/commits/v0.1.32) --- updated-dependencies: - dependency-name: "@github/copilot" dependency-version: 1.0.2 dependency-type: indirect - dependency-name: "@github/copilot-sdk" dependency-version: 0.1.32 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * specify application path in prompt (microsoft#1204) * Add AVM (Azure Verified Modules) integration tests (microsoft#1171) * Add AVM (Azure Verified Modules) integration tests Add 3 integration tests validating the AVM module selection hierarchy for Bicep infrastructure generation: - avm-module-priority: Verifies AVM modules prioritized over non-AVM - avm-fallback-behavior: Verifies fallback stays within AVM ecosystem - avm-azd-pattern-preference: Verifies AZD pattern modules preferred Tests validate that the azure-deploy skill enforces the mandatory AVM selection order: Pattern modules > Resource modules > Utility modules, and never falls back to non-AVM alternatives. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add output assertions to AVM integration tests Address Copilot review feedback: add keyword-based output assertions using getAllAssistantMessages/getAllToolText to verify agent responses contain AVM hierarchy terms, not just skill invocation. Includes non-AVM fallback negative check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Strengthen AVM test output assertions per Copilot review - Split keyword checks into critical-term + context assertions - Add resource-before-utility ordering assertion for fallback test - Expand non-AVM negative check to use regex patterns - Require core keywords (avm+pattern, azd+pattern) explicitly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address Copilot round 3 — ordering assertions and context-aware non-AVM check - Add hierarchy ordering assertion to test 1 (pattern before resource/utility) - Make non-AVM detection context-aware: skip matches preceded by negation words (e.g., 'never fall back to non-AVM' is correct behavior, not a false positive) - Add pattern-before-resource ordering assertion to test 3 (AZD pattern preference) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: move AVM integration tests to avm/ subdirectory Move tests/azure-deploy/avm-integration.test.ts to tests/azure-deploy/avm/integration.test.ts so the file matches the **/integration.test.ts glob used by the custom ESLint rule (integration-test-name) and follows the subdirectory convention established by tests/microsoft-foundry/ (e.g. foundry-agent/). Import paths updated from ../utils/ to ../../utils/ to reflect the new depth. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address round 4 Copilot review feedback - Add 'fall back'/'fall-back' keyword variants for resilience - Extend non-AVM negation check to also scan following context - Use regex for AZD ordering assertion to match plural/prefixed variants Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update github workflows to use best practices (microsoft#1149) * Early terminate azure-deploy, azure-validate tests (microsoft#1205) Add comment for early termination to help AI grader * Replace script inline parameters with env var (microsoft#1209) * Early terminate azure-deploy tests on deploy link (microsoft#1208) * Early terminate azure-deploy tests on deploy link * Fix lint issue * Reduce char count of existing skills (microsoft#1210) * Reduce char count of existing skills * Update ci tests and snapshots * Enhance benchmark ci run script (microsoft#1176) * Add msbench_benchmarks repo clone to get model definition * Remove unused vars * Use mcp-pr repo before MI has access to msbench-benchmarks repo * Address copilot feedback * Change back to msbench-benchmarks repo * Get ADO token for repo clone * Fix line continuation character * Add run for all interested models * Extract run IDs * Fix yaml format issue * Schedule it to run nightly * Address copilot feedbacks * formalize .foundry and multi-environment support * fix * Feature/azure quotas (microsoft#1137) * update for using azure-quotas in skill * test update * unit test update * path update * add skill in skills.json * skills.json update * reduce the text * version update * skill version * skill description update * reduce text size * 1.0.4 for next prepare version * upload snap shot * update version * test update --------- Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com> * build(deps-dev): bump simple-git from 3.30.0 to 3.32.3 in /tests (microsoft#1213) Bumps [simple-git](https://github.com/steveukx/git-js/tree/HEAD/simple-git) from 3.30.0 to 3.32.3. - [Release notes](https://github.com/steveukx/git-js/releases) - [Changelog](https://github.com/steveukx/git-js/blob/main/simple-git/CHANGELOG.md) - [Commits](https://github.com/steveukx/git-js/commits/simple-git@3.32.3/simple-git) --- updated-dependencies: - dependency-name: simple-git dependency-version: 3.32.3 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Improve azure-compliance invocation rate (microsoft#1214) * Improve azure-compliance invocation rate * Race condition free report writing * Fix debug logging for report location * Bump skill version * Fix suffix base value * fix * llm judge model and eval group improvement --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chris Harris <charris@microsoft.com> Co-authored-by: JasonYeMSFT <chuye@microsoft.com> Co-authored-by: xfz11 <81600993+xfz11@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Ospina <70209456+jeo02@users.noreply.github.com> Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com> Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: rakal-dyh <33503911+rakal-dyh@users.noreply.github.com> Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com> * fix: expose observe prompt optimization routing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> # Conflicts: # tests/microsoft-foundry/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/create/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/deploy/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/invoke/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/observe/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/trace/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/foundry-agent/troubleshoot/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/capacity/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/customize-deployment/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/deploy-model-optimal-region/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/models/deploy/deploy-model/__snapshots__/triggers.test.ts.snap # tests/microsoft-foundry/resource/create/__snapshots__/triggers.test.ts.snap * chore: drop issue triage token fallback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: simplify deploy P0 test case guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: align microsoft-foundry version Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix comments * test: move eval-datasets invocation tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix comments * bump version --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: XOEEst <18523445+XOEEst@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Chris Harris <charris@microsoft.com> Co-authored-by: JasonYeMSFT <chuye@microsoft.com> Co-authored-by: xfz11 <81600993+xfz11@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Juan Ospina <70209456+jeo02@users.noreply.github.com> Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com> Co-authored-by: Wes Haggard <weshaggard@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: rakal-dyh <33503911+rakal-dyh@users.noreply.github.com> Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com>
* Add missing files * Bump up version * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add skill checks to make sure they meet the standard --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
1. When creating new issues for integration test failures, improve the logic for finding duplicates. Currently we look for existing issues with exactly the same title as the issue we would open. This isn't good enough, as it is unlikely the LM would produce exactly the same title every time even with identical inputs. Instead, we now find all open integration test failures for the given skill and check the information in the body to look for similarities. 2. Recompile analyze-test-run.md to produce analyze-test-run.lock.yml, which is actually what the workflow looks at. This wasn't done the last time the .md was updated, with the result that issues can't be tagged with the "integration-test" label.
…microsoft#1277) * Check for azure-foundry tool call in foundry test Bump copilot sdk test time out to 20 minutes * Reduce copilot-sdk test attempts to 3 * Still check for mcp tool name in response * Remove unused import
…1292) * Replace any https://github.com/microsoft/github-copilot-for-azure under https://github.com/microsoft/azure-skills for sync'ed files * Update repo name
…workflow (microsoft#1274) * Set skill-invocation rate success rate threshold * Bump skill version
…lls (microsoft#1310) * Initial plan * Remove marketplace.json and update all references to use microsoft/azure-skills Co-authored-by: tmeschter <10506730+tmeschter@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: tmeschter <10506730+tmeschter@users.noreply.github.com>
* Extend redact pattern * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* init troubleshoot * wip * wip * wip * wip * remove limit overrides * wip * fix token issues * rename folder for subskill naming consistency * fix PR issues * pr fixes * Pr fixes * fix test fail * fix references * fix copilot comments * revert staging test integration --------- Co-authored-by: Alex Okonechnikov <alexok@microsoft.com>
…ues (microsoft#1300) * Make analyze-test-run auto assign label and owner for its created issues * Add test-failure label
* Update plugin version automatically * Potential fix for code scanning alert no. 173: Workflow does not contain permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Update nodejs version * Addressed copilot feedback * Convert script to typescript * Updated package.json * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Fix lint issues * Add comment about the token * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * the script should not be invoked manually * Add new dependency * Update doc * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Fix release command and yml format * Update doc and add missing keywords * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Error out when not both files can be updated successfully * Move script under src dir * Update node version * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Automatically update CHANGELOG * Add test * Move exit code to main * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Fix type * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Fix regex format --------- Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Consider all shell commands Fixes microsoft#1322. There are a few places in our integration tests where we explicitly check if "powershell" was used to execute some command. Since these tests need to run on non-Windows system we should be checking for both "powershell" and "bash". * Update a comment Update a comment on a function to properly reflect what it does. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Fixes for azure-resource-visualizer Fixes microsoft#1266. 1. Update the skill `description` to use "WHEN" instead of "USE FOR", and remove the "DO NOT USE FOR" section. 2. Update the integration tests to more reliably check for a generated architecture.*.md file with a MermaidJS diagram. Previously we were dependent on Copilot using the `create` tool to create the file, but in some runs we've seen it use a bash script instead. In these cases the file would be created but we would assert that it hadn't. Looking for the actual file rather than depending on the manner of its creation should be more reliable. * Update snapshots * Further test test updates * Address PR review: use doesWorkspaceFileIncludePattern and add workspacePath assertions - Replace custom hasArchitectureDiagramFile() with existing doesWorkspaceFileIncludePattern() helper which properly skips node_modules and uses withFileTypes for compatibility - Remove unused fs and path imports - Add expect(workspacePath).toBeDefined() guards before workspacePath! usage in both tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…oundry-agent observe skill (microsoft#1247) * feat: show data viewer deeplink for dataset and eval result file * tweak wording * add vsc only * bump version to 1.0.6 * fix md format issue * fix folder issue * Revert "fix md format issue" This reverts commit 1873f7c. * fix markdown reference issue
* infra: Update azure-skills copy Currently the publish-to-marketplace.yml pipeline copies all of the "plugin" directory in this repo to the ".github/plugins/azure-skills" directory in the microsoft/azure-skills repo. This is really where we want the files to live long-term, especially as it allows us to put a different plugin in the same repo. However, VS Code currently only looks for plugins at the repo root, not in a subdirectory. Until the plugin support improves, we need to copy the plugin.json, .mcp.json, and skills directory to the repo root as well as the current location. Here we update the pipeline to do that. Note that currently azure-skills uses symlinks to try and achieve the same thing without needing to duplicate files, but it turns out that whether or not this works in practice depends very much on an individual user's git settings. Having two copies is annoying but will be much more reliable. * fix: guard grep in URL replacement and handle both URL case variants Address PR review feedback: - Wrap grep in if-statement to prevent step failure when no files match - Use case-insensitive grep (-i) to find both URL variants - Add second sed command to replace lowercase URL form Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
microsoft#1329) - Pin 57 GitHub Action references to full commit SHA across 13 workflow files - Harden npm ci with --ignore-scripts in PR-triggered workflows - Add explicit CODEOWNERS protection for .github/workflows/ - Fix 3 shell expression injection sites (move inputs to env vars) - Add Dependabot config for npm and github-actions ecosystems Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bumps [azure/login](https://github.com/azure/login) from 2.3.0 to 3.0.0. - [Release notes](https://github.com/azure/login/releases) - [Commits](Azure/login@a457da9...532459e) --- updated-dependencies: - dependency-name: azure/login dependency-version: 3.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…oft#1372) Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4.6.2 to 7.0.0. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](actions/upload-artifact@v4.6.2...bbbca2d) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#1371) Bumps [actions/setup-dotnet](https://github.com/actions/setup-dotnet) from 4.3.1 to 5.2.0. - [Release notes](https://github.com/actions/setup-dotnet/releases) - [Commits](actions/setup-dotnet@67a3573...c2fa09f) --- updated-dependencies: - dependency-name: actions/setup-dotnet dependency-version: 5.2.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…oy (microsoft#1845) * Fail deployment-validation tests immediately on premature azd up/deploy The deployment-validation tests waited the full 20-minute Jest timeout when the agent ran `azd up` before invoking azure-validate, because shouldEarlyTerminate only watched for validation commands and the azure-deploy skill — not direct deployment CLI commands. Add hasDeploymentCommand() helper that detects `azd up` and `azd deploy` in shell tool calls. Use it in shouldEarlyTerminate for all three deployment-validation tests so they abort immediately when the agent skips validation and jumps to deployment. Also add an explicit assertion that no deployment command ran, giving a clear failure message. Fixes microsoft#1694 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Delegate to matchesCommand instead of duplicating getShellCommands Address PR feedback from jongio: evaluate.ts already exports matchesCommand() with identical logic, so hasValidationCommand and hasDeploymentCommand now delegate to it. This removes the duplicated getShellCommands helper and SHELL_TOOL_NAMES constant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add comments explaining why deployment commands indicate a problem Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…icrosoft#1585) * fix: prevent azd+Terraform template variable interpolation failures (microsoft#1558) Address azd template variable interpolation gap that caused deployment timeouts in terraform-azure-container-apps-deploy integration tests. azure-prepare (v1.0.13): - Add warning against using Go-style template variables in .tfvars.json - Document correct variable passing: azd auto-mapping, TF_VAR_* env vars - Remove incorrect env() function usage in variable example - Add troubleshooting entries for template interpolation errors azure-validate (v1.0.3): - Add Step 10: Template Variable Resolution Check for azd+Terraform - Detect unresolved {{ .Env.* }} patterns and .tfvars.json files - Provide remediation steps to fix before deployment azure-deploy (v1.0.9): - Add Unresolved Terraform Template Variables error section with solution - Add pre-deploy Step 9: Verify Terraform Variable Resolution - Add Terraform state management error entries - Document azd state clearing behavior and remote backend recommendation Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore: bump azure-prepare skill version to 1.0.14 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugin/skills/azure-deploy/references/pre-deploy-checklist.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Restore current plugin version before checking for skill content changes (microsoft#1595) * fix: rename .azure/plan.md to .azure/deployment-plan.md to prevent agent confusion (microsoft#1584) * fix: rename .azure/plan.md to .azure/deployment-plan.md to prevent confusion with session-state plan.md The agent was confusing the workspace deployment plan (.azure/plan.md) with the session-state plan.md file, causing the 'creates correct files for AZD with Bicep recipe' integration test to fail (issue microsoft#1562). Renaming to deployment-plan.md eliminates the name collision and makes the file's purpose self-documenting. Updated all references across azure-prepare, azure-validate, and azure-deploy skills, their reference docs, and all tests. Closes microsoft#1562 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: update azure-deploy trigger keyword snapshots for deployment-plan rename Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * build(deps): bump github/codeql-action from 4.34.1 to 4.35.1 (microsoft#1570) Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.34.1 to 4.35.1. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@3869755...c10b806) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.35.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Remove strong verbiage from Enterprise Infra Planner skill (microsoft#1533) * feat: remove strong verbiage * feat: update phrasing * fix: undo tool name change * chore: bump version to 1.0.1 --------- Co-authored-by: Michael Ren <mren@microsoft.com> * fix: Add Docker build context validation to azure-validate skill (microsoft#1586) * Add Docker build context validation to azure-validate skill Pre-validate Docker build context during azure-validate by checking for package-lock.json when npm ci is specified in a Dockerfile. This prevents Docker build failures during azd package/up that waste time and can push deployments past test timeouts. Changes: - AZD recipe: Add step 9 (Docker Build Context Validation) between Build Verification and Package Validation - AZCLI recipe: Enhance Docker Build step with build context pre-validation before attempting docker build - AZD/AZCLI errors: Add npm ci / package-lock.json missing error entry - Bump azure-validate version to 1.0.3 Fixes microsoft#1557 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugin/skills/azure-validate/references/recipes/azd/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugin/skills/azure-validate/references/recipes/azd/README.md --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Copy .claude-plugin/plugin.json to repo root for Claude marketplace support (microsoft#1605) Add .claude-plugin/plugin.json to the sync-to-microsoft-azure-skills job so the Azure plugin is discoverable in the Claude marketplace. Updates the copy, URL replacement, version restore, and version bump steps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Copy hooks to top-level folder in azure-skills in addition to skills (microsoft#1606) * Copy .claude-plugin/plugin.json to repo root for Claude marketplace support Add .claude-plugin/plugin.json to the sync-to-microsoft-azure-skills job so the Azure plugin is discoverable in the Claude marketplace. Updates the copy, URL replacement, version restore, and version bump steps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Copy hooks to top-level folder in azure-skills in addition to skills Update the sync-to-microsoft-azure-skills job in the publish pipeline to also copy hooks/ and copilot-hooks.json to the repo root, matching how skills/ is already copied. Also add the new paths to the URL replacement step. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Azure skills update for verification of functionality and role assignments (microsoft#1220) * update azure-prepare skill to check subscription policies * update azure-prepare skill to check functionality before deploying * Add role assignment verification step to azure-prepare skill Add new Phase 2 step 5 (Verify Role Assignments) between security hardening and functional verification. Includes reference doc with service-to-role mapping table, MCP tool usage, and common RBAC mistakes (e.g., generic Contributor lacking data-plane access). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugin/skills/azure-prepare/references/role-verification.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugin/skills/azure-prepare/references/functional-verification.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add live role verification step to azure-validate skill Add step 4 (Live Role Verification) to query Azure for provisioned RBAC assignments and cross-check against expected roles. Complements the static role check in azure-prepare: prepare checks generated Bicep/Terraform, validate checks live Azure state. Includes reference doc with MCP tool usage, CLI commands, common issues table, and decision tree for pass/fail criteria. Bumps azure-validate version 1.0.0 -> 1.0.1. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify azure-prepare role check as static only Replace MCP live-query section with static code review guidance. Live role verification is the responsibility of azure-validate step 4 (live-role-verification.md). This removes the overlap between the two skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: move role verification across prepare/validate/deploy skills - Remove static role check (step 5) from azure-prepare — prepare just generates - Add static role check as step 4 in azure-validate (pre-deployment) - Move live role check from azure-validate step 4 to azure-deploy step 8 (post-deployment) - Move role-verification.md from azure-prepare to azure-validate references - Move live-role-verification.md from azure-validate to azure-deploy references - Update all step number cross-references in functional-verification.md - Bump versions: prepare 1.0.6->1.0.7, validate 1.0.1->1.0.2, deploy 1.0.5->1.0.6 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add azure__role to azure-deploy MCP Tools table Step 8 (Live Role Verification) references azure__role for RBAC assignment listing, but the tool was missing from the MCP Tools table. Agents could incorrectly assume only the three listed tools are available. Bump version 1.0.6 -> 1.0.7. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * update test snapshots * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * update the versions and added integration/ unit tests * Update plugin/skills/azure-validate/references/role-verification.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * update skills and the test runs * update snapshots * update to correct versions * update reference name * update versioning * update steps * fix: update azure-deploy trigger test snapshots Keywords were removed from the SKILL.md description in a previous PR but the trigger test snapshots were not regenerated, causing 2 snapshot failures in the pipeline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: update azure-deploy trigger test snapshots Keywords were removed from the SKILL.md description in a previous PR but the trigger test snapshots were not regenerated, causing 2 snapshot failures in the pipeline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * update snapshot --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix azure-prepare: add DTS bicep.md to workflow routing (microsoft#1540) (microsoft#1604) The workflow routing entries loaded durable.md and the DTS README but not bicep.md — so the agent had overview docs but not the Bicep patterns needed to generate .bicep files. Also adds 'order processing' keyword. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * add skill invocation rate dashboard (microsoft#1630) * Add unified azure-cost skills that combines azure-cost-query, azure-cost-forecast and azure cost-optimization skills (microsoft#1221) * initial implementation * update the guardrails for query and forecast * reduce token limit of reference files * update unit tests * fix breaking PR checks * fix pr check errors and code review comments * refactor to azure-cost (#1) * update tests and references to combined azure cost skill * Remove unused test fixture files Delete cost-query-sample.json and cost-forecast-sample.json from tests/azure-cost/fixtures/ as they are not referenced by any test files. No other skills in the repo use fixture files either, so these add maintenance overhead without value. Addresses PR review comment microsoft#14 and microsoft#15 (fixtures removed entirely rather than fixing hard-coded dates, since they were unused). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * updates tests * Consolidate azure-cost tests to standard 3-file layout, fix CI gates - Consolidate 12 test files into standard 3-file structure (unit/triggers/integration) - Rewrite integration tests using canonical withTestResult pattern - Move all positive trigger prompts into triggers.test.ts - Move all sub-area unit assertions into unit.test.ts - Delete 9 redundant sub-area test files - Regenerate snapshot with Jest 30 header format - Bump sensei version 1.0.1 -> 1.0.2 - Bump azure-prepare version 1.0.10 -> 1.0.11 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix PR review comments: canonical azqr tool name, table formatting - Change azure__extension_azqr to mcp_azure_mcp_extension_azqr in SKILL.md - Fix missing space in 429 table row in cost-forecast/error-handling.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments: MCP tools table, code block languages, remove phantom skill - Add azure__extension_azqr and azure__aks to MCP Tools table for consistency - Add yaml language to azqr code block in SKILL.md - Add text language to portal link code block in report-template.md - Remove non-existent azure-create-app row from tests/README.md coverage grid Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: extract azure-cost workflows into separate reference files Move each workflow (query, optimization, forecast) into dedicated reference files under references/ for progressive disclosure. This reduces SKILL.md from 575 lines (23KB) to 139 lines (7KB), so the agent only loads the workflow it needs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * refactor: move workflow files into their respective folders Move cost-query-workflow.md, cost-optimization-workflow.md, and cost-forecast-workflow.md from references/ into cost-query/, cost-optimization/, and cost-forecast/ as workflow.md. Update all links in SKILL.md and cross-references. Bump version to 1.0.2. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fixes skill quality issues * fix: update azure-cost tests for refactored skill structure - Update snapshot to match new description with DO NOT USE FOR clause - Update unit tests to load workflow files directly (content moved from SKILL.md to cost-query/, cost-forecast/, cost-optimization/ folders) - Fix heading level assertions (## not ### in standalone workflow files) - Remove 3 shouldNotTrigger prompts that contain cost keywords and correctly trigger the keyword-based matcher Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: reset azure-cost version to 1.0.0 and fix YAML comment syntax - Reset version to 1.0.0 for new skill directory (was incorrectly 1.0.3) - Change // optional to # optional in YAML code block Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Sai Koumudi Kaluvakolanu <saikoumudi@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Document principal type mismatch error in AZD errors reference (microsoft#1649) * Document principal type mismatch error in AZD errors reference AZD base templates (e.g. functions-quickstart-python-http-azd) create RBAC role assignments with hardcoded principalType 'User' for the deploying identity. In CI/CD where a service principal is used, ARM rejects this with a PrincipalType mismatch error. The agent had no guidance for this failure and spent multiple retries before finding the fix. Adding this to the AZD errors reference gives the agent a direct path to the solution: set allowUserIdentityPrincipal to false in main.bicep. It also warns against the ineffective workaround of clearing AZURE_PRINCIPAL_ID. Fixes microsoft#1624 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugin/skills/azure-deploy/references/recipes/azd/errors.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Clarify main.parameters.json format to prevent .bicepparam confusion (microsoft#1648) * Clarify main.parameters.json format to prevent .bicepparam confusion Add explicit warnings and complete examples to the three skill reference files used by azure-prepare, azure-validate, and azure-deploy: - patterns.md: Replace hard-coded values with azd \ substitution syntax and add a warning against .bicepparam syntax - iac-rules.md: Add a new Parameter File Format section with a full ARM JSON example and format warning - troubleshooting.md: Add \/contentVersion to the incomplete JSON example and add a format warning callout Addresses the root cause of issue microsoft#1623 where the agent created main.parameters.json with .bicepparam syntax (readEnvironmentVariable), causing 6 failed azd provision --preview attempts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump azure-prepare to 1.1.2 and azure-deploy to 1.0.11 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugin/skills/azure-prepare/references/recipes/bicep/patterns.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * refactor quota tests (microsoft#1489) * refactor quota tests * add more specification to test --------- Co-authored-by: Christopher Earley <cearley@microsoft.com> * update AKS cost spike prompts with specific time window (microsoft#1650) Co-authored-by: Harsha Nair <hnair@microsoft.com> * feat: GEPA integration for sensei skill + quality score CI workflow (microsoft#1498) * feat: add GEPA integration to sensei skill + quality score workflow Add GEPA (Genetic-Pareto) evolutionary optimization as an optional enhancement to sensei's Ralph loop for automated SKILL.md improvement. Changes: - .github/skills/sensei/SKILL.md: Added --gepa flag, GEPA mode docs, Step 5-GEPA in the Ralph loop - .github/skills/sensei/scripts/gepa/auto_evaluator.py: Auto-discovers test harness at runtime, builds GEPA evaluators, scores/optimizes skills - pipelines/gepa-quality-score.yml: PR quality gate that scores SKILL.md quality and posts results as PR comment The auto-evaluator requires zero manual configuration. It reads triggers.test.ts to extract shouldTrigger/shouldNotTrigger arrays and builds a composite evaluator (content quality + trigger accuracy). Existing tests are NOT replaced or modified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address PR review feedback for GEPA integration - Bump sensei SKILL.md version 1.0.0 → 1.0.2 (fixes Skill Structure CI) - Remove unused imports: sys, dataclass, field (fixes CodeQL warnings) - Extract strip_frontmatter() helper to replace fragile content.index() parsing that could raise ValueError on malformed frontmatter - Deduplicate frontmatter stripping logic between score_skill/optimize_skill - Add explicit permissions block (contents: read, pull-requests: write) - Use sticky comment pattern (<- Consolidate FileSystemWatcher usage: gepa-quality-score --> marker) to avoid PR comment spam on re-runs - Fix display results to match workflow_dispatch single-skill input - Rename quality gate step to '(advisory)' to clarify non-blocking behavior Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: skip PR comment step for forked PRs Forked PRs have reduced GITHUB_TOKEN permissions, which would cause the comment step to fail. Only post comments when the PR originates from the same repository. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: strip comments in trigger parsing + clarify GEPA step scope - Strip single-line (//) and multi-line (/* */) comments from trigger test arrays before extracting strings, preventing commented-out example prompts from polluting trigger accuracy scoring - Fix SKILL.md step 5b to clarify GEPA only replaces step 5 (IMPROVE FRONTMATTER), not step 6 (IMPROVE TESTS) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: correct docstring and SKILL.md to reflect actual evaluator behavior The evaluator parses trigger prompt arrays and uses content heuristics for scoring — it does not execute Jest tests or incorporate test pass/fail results. Updated docs to accurately describe this. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address round 3 review feedback - Remove unused params: as_json from score_skill, fast from build_evaluator - Pin all actions to commit SHAs matching repo convention (checkout v6, setup-python v6.2.0, upload-artifact v7.0.0, github-script v8.0.0) - Pin gepa dependency to v0.7.0 for reproducible CI - Remove DO NOT USE FOR from scoring criteria (conflicts with repo guidance that discourages it due to keyword contamination risk) - Add quality_score_raw field for full-precision threshold comparisons - Enhance parse_trigger_arrays to resolve ...varName spread patterns by extracting strings from referenced arrays in the same file - Clarify SKILL.md step 5b: GEPA uses trigger definitions as config, does not execute Jest tests - Add NOTE about future workflow_run commenting pattern migration Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: address PR review feedback — split workflow, fix regex, update docs - Split gepa-quality-score.yml into read-only scoring workflow + workflow_run-triggered commenter (gepa-quality-score-comment.yml), matching the repo's existing pr.yml / pr-comment.yml pattern - Fix API key regex to also match 'api key:' with whitespace separator - Update PR description to clarify ASI uses heuristic scoring (Jest integration is planned for future iteration) - Remove pull-requests:write from scoring workflow permissions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * build(deps-dev): bump the minor-and-patch group (microsoft#1572) Bumps the minor-and-patch group in /scripts with 4 updates: [@vitest/coverage-v8](https://github.com/vitest-dev/vitest/tree/HEAD/packages/coverage-v8), [fast-xml-parser](https://github.com/NaturalIntelligence/fast-xml-parser), [typescript-eslint](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/typescript-eslint) and [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest). Updates `@vitest/coverage-v8` from 4.1.0 to 4.1.2 - [Release notes](https://github.com/vitest-dev/vitest/releases) - [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.2/packages/coverage-v8) Updates `fast-xml-parser` from 5.5.8 to 5.5.9 - [Release notes](https://github.com/NaturalIntelligence/fast-xml-parser/releases) - [Changelog](https://github.com/NaturalIntelligence/fast-xml-parser/blob/master/CHANGELOG.md) - [Commits](NaturalIntelligence/fast-xml-parser@v5.5.8...v5.5.9) Updates `typescript-eslint` from 8.57.1 to 8.57.2 - [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases) - [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/typescript-eslint/CHANGELOG.md) - [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.57.2/packages/typescript-eslint) Updates `vitest` from 4.1.0 to 4.1.2 - [Release notes](https://github.com/vitest-dev/vitest/releases) - [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.2/packages/vitest) --- updated-dependencies: - dependency-name: "@vitest/coverage-v8" dependency-version: 4.1.2 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: minor-and-patch - dependency-name: fast-xml-parser dependency-version: 5.5.9 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: minor-and-patch - dependency-name: typescript-eslint dependency-version: 8.57.2 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: minor-and-patch - dependency-name: vitest dependency-version: 4.1.2 dependency-type: direct:development update-type: version-update:semver-patch dependency-group: minor-and-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add kvenkatrajan as codeowner for entra-app-registration (microsoft#1667) * Initial plan * Add kvenkatrajan as codeowner for entra-app-registration Agent-Logs-Url: https://github.com/microsoft/GitHub-Copilot-for-Azure/sessions/0062a31a-0103-4dbf-a191-8264b9deea81 Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com> * adding the azure skills gif (microsoft#1651) * Address wbreza PR review: fix main.tfvars.json guidance and auto-mapping claims Key changes based on wbreza's review of PR microsoft#1585: - Replace 'Do NOT generate main.tfvars.json' with correct guidance: use \\\ syntax (azd envsubst), not Go-style {{ .Env.* }} - Remove incorrect 'azd auto-mapping' claims — variables flow via main.tfvars.json substitution or explicit TF_VAR_* env vars - Fix pre-deploy check: validate syntax in main.tfvars.json instead of rejecting the file's existence - Scope grep patterns with --include='*.tf' --include='*.tfvars.json' to avoid false positives from .terraform/ and READMEs - Align grep patterns consistently across all files - Update remediation steps to fix syntax rather than delete files - Add main.tfvars.json example with correct \ syntax Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump skill versions: azure-prepare 1.0.15, azure-deploy 1.0.12, azure-validate 1.0.4 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump version * Remove extraneous file * Fix comments in code blocks Move comments above code blocks. This is especially important for JSON, as it does not actually support code comments and we wouldn't want the LM to copy the code block as-is. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Fan Yang <52458914+fanyang-mono@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Michael <37400755+micha31r@users.noreply.github.com> Co-authored-by: Michael Ren <mren@microsoft.com> Co-authored-by: KarishmaGhiya <kghiya8@gmail.com> Co-authored-by: greenie-msft <56556602+greenie-msft@users.noreply.github.com> Co-authored-by: msalaman <Marcossalamanca97@hotmail.com> Co-authored-by: taylorak <taykenned@gmail.com> Co-authored-by: Sai Koumudi Kaluvakolanu <saikoumudi@gmail.com> Co-authored-by: Christopher T Earley <jrekct@gmail.com> Co-authored-by: Christopher Earley <cearley@microsoft.com> Co-authored-by: Harsha Nair <hjjn26@gmail.com> Co-authored-by: Harsha Nair <hnair@microsoft.com> Co-authored-by: Shayne Boyer <spboyer@live.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: kvenkatrajan <102772054+kvenkatrajan@users.noreply.github.com> Co-authored-by: Yun Jung Choi <49920477+yunjchoi@users.noreply.github.com>
…_web_app (microsoft#1587) * Add Terraform reference for Static Web Apps using azurerm_static_web_app Fixes microsoft#1555: azure-prepare was missing a dedicated Terraform reference for Static Web Apps, causing the agent to fall back to Storage Account-based static_website hosting. That approach requires anonymous blob access, which violates enterprise Azure Policies (RequestDisallowedByPolicy). Changes: - Create terraform.md with azurerm_static_web_app patterns and an explicit warning against using Storage Account static_website hosting - Update README.md to link to the new Terraform reference - Update deployment.md with Terraform-specific deployment guidance and security warnings Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix broken relative link to recipes/azd/terraform.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugin/skills/azure-prepare/references/services/static-web-apps/terraform.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: Address wbreza PR feedback on terraform.md - Remove redundant sku_size (always mirrors sku_tier) - Add comment clarifying sku_tier defaults to Free - Fix 'location' wording nit in region availability callout Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update version number --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…osoft#1858) * Fix Aspire AddDockerfile test and skill docs (microsoft#1695) Aspire handles AddDockerfile container builds at runtime through the AppHost. The azure.yaml generated by azd init --from-code should only contain a single "app" service pointing to the AppHost — individual AddDockerfile services like ginapp should NOT appear as separate entries. Test changes: - Remove incorrect getDockerContext("ginapp") assertion - Assert the correct behavior: services section exists with "app" service Skill doc changes (recipes/azd/aspire.md): - Remove incorrect "Post-Init: Verify and Fix Docker Context" section - Remove incorrect "Docker Context (AddDockerfile Services)" section - Add note clarifying AddDockerfile services are handled at runtime - Update Validation Steps to reflect correct behavior Skill doc changes (recipes/azd/azure-yaml.md): - Clarify "Custom Docker Context" example is for non-Aspire projects - Add warning against manually adding per-service docker.context for Aspire Fixes microsoft#1695 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix unit test: remove docker.context assertions from aspire tests The unit test checked for "docker.context" in aspire.md, but that content was correctly removed in the prior commit. Update assertions to match the current docs which explain that AddDockerfile resources are handled at runtime by Aspire. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump azure-prepare skill version to 1.1.15 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove conflict markers --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Create workflow to collect dashboard data * Fix env var name
…es (microsoft#1829) Bumps the minor group with 2 updates in the /scripts directory: [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) and [eslint](https://github.com/eslint/eslint). Updates `@types/node` from 25.5.0 to 25.6.0 - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) Updates `eslint` from 10.1.0 to 10.2.0 - [Release notes](https://github.com/eslint/eslint/releases) - [Commits](eslint/eslint@v10.1.0...v10.2.0) --- updated-dependencies: - dependency-name: "@types/node" dependency-version: 25.6.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: minor - dependency-name: eslint dependency-version: 10.2.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…es (microsoft#1831) Bumps the minor group with 3 updates in the /tests directory: [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node), [eslint](https://github.com/eslint/eslint) and [simple-git](https://github.com/steveukx/git-js/tree/HEAD/simple-git). Updates `@types/node` from 25.5.0 to 25.6.0 - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) Updates `eslint` from 10.1.0 to 10.2.0 - [Release notes](https://github.com/eslint/eslint/releases) - [Commits](eslint/eslint@v10.1.0...v10.2.0) Updates `simple-git` from 3.33.0 to 3.36.0 - [Release notes](https://github.com/steveukx/git-js/releases) - [Changelog](https://github.com/steveukx/git-js/blob/main/simple-git/CHANGELOG.md) - [Commits](https://github.com/steveukx/git-js/commits/simple-git@3.36.0/simple-git) --- updated-dependencies: - dependency-name: "@types/node" dependency-version: 25.6.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: minor - dependency-name: eslint dependency-version: 10.2.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: minor - dependency-name: simple-git dependency-version: 3.36.0 dependency-type: direct:development update-type: version-update:semver-minor dependency-group: minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 8.0.3 to 8.0.7. - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v8.0.7/packages/vite) --- updated-dependencies: - dependency-name: vite dependency-version: 8.0.7 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ft#1739) Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) from 6.4.1 to 6.4.2. - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/v6.4.2/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v6.4.2/packages/vite) --- updated-dependencies: - dependency-name: vite dependency-version: 6.4.2 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* update scripts * update scripts to have strict regex check and make it a common script across 3 clients * update script to strip azure prefix for claude * normalize backslashes * using copilot-cli env variable * continue using toolArgs for backward compatibility for copilot cli versions that dont support copilot_cli env var
…#1767) Replace all manual file reading with doesWorkspaceFileIncludePattern, matching the pattern from azure-deploy/integration.test.ts. This handles node_modules skipping, isFile filtering, and try/catch for unreadable files. Also drops the fs and listFilesRecursive imports. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…1863) * Update storage account name in analyze-skill-issues skill * Update skill version
…osoft#1869) Change the integration test prompt from 'that I can delete' to 'that are wasting money' so the LLM routes to azure-cost instead of azure-resource-lookup. Fixes microsoft#1812 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add org info for msbench runs * Only run skill related benchmarks * Update org name * Update comment
…rosoft#1457) * fix: avoid common error on deploy * chore: tweak wording * chore: new version * Update plugin/skills/microsoft-foundry/foundry-agent/deploy/deploy.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: call out repository-scoped role * fix: call out repository-scoped role --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
microsoft#1887) Add guardrails and pre-flight safety checks to prevent running extension-backed commands (az vm user update, az vm run-command invoke, etc.) on VMs with unhealthy agents or stuck extensions. Changes: - Add Guardrails and Evidence Order sections to vm-troubleshooter.md - Add Phase 2.5: Pre-Flight Safety Checks (mandatory gate before extension-backed operations) - Add pre-flight CLI commands and safe/unsafe decision table to cannot-connect-to-vm.md - Add warning callouts and extension markers to all 6 sub-reference Quick Commands blocks - Add error handling rows for agent deadlock scenarios - Require user approval for restart/redeploy in escalation paths - Bump version from 2.1.0 to 2.2.0 Co-authored-by: Rebecca Mueller <muellerr@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Collect non-integration test results in dashboard data * Install tests dependencies
* quotas update and unit test update * update * bump up version * update, including quota name in vm-families.md * update snap shot * update * disable using quota mcp server in most place * update * update * update * update * update * bump version --------- Co-authored-by: Yinghui Dong <yinghuidong@microsoft.com>
…et storage (microsoft#1877) * azure-prepare: add proactive Step 4b for Aspire + Functions secret storage When an Aspire AppHost contains AddAzureFunctionsProject with WithHostStorage, the agent must proactively add .WithEnvironment("AzureWebJobsSecretStorageType", "Files") to AppHost.cs BEFORE deployment. Without this, Functions fail at startup with "Secret initialization from Blob storage failed". Previously this guidance was only in the Troubleshooting section of aspire.md (reactive) and in aspire-containerapps.md (orphaned from the workflow). Now it is a mandatory proactive step (4b) in the Aspire preparation workflow, with cross-references from scan.md and generate.md. Fixes microsoft#1860 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert generate.md change Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix "skip to Step 5" references in Step 4b Replace "skip to Step 5" (which does not exist) with "skip this step" so the agent knows to continue with the rest of the workflow. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove health-checks-ui integration test (microsoft#1849) The health-checks-ui sample is not meaningfully deployable to Azure. A user asking to deploy it would indicate a misunderstanding of the application. Remove the test rather than maintaining an assertion about agent behavior for an unrealistic scenario. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix ESLint: remove extra blank line Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Upload benchmark reports * Update pipelines/scripts/Invoke-UploadBenchmarkReports.ps1 Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update pipelines/scripts/Invoke-UploadBenchmarkReports.ps1 Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review feedbacks --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Before double quotes were included as the value for org * Update script name
* Add skills Add skills for investigating integration test failures and submitting PRs with fixes. * Update .github/skills/submit-skill-fix-pr/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update .github/skills/submit-skill-fix-pr/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update .github/skills/investigate-integration-test/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
azd env new triggers a "Set new environment as default?" interactive prompt when an environment already exists. This blocked the flask calculator integration test for 30+ minutes until timeout (issue microsoft#1884). Add --no-prompt flag to all azd env new and azd init commands across the azure-deploy, azure-validate, and azure-prepare skill reference docs. Fixes microsoft#1884 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| try { | ||
| const relativePath = normalizePath(relative(rootDir, filePath)); | ||
| validatePath(relativePath); | ||
| const content = execSync(`git show "${ref}:${relativePath}"`, { |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes microsoft#1884
Problem
azd env newtriggers an interactive prompt ("Set new environment as default? (y/N)") when an existing azd environment is present. During the flask calculator integration test, retry cycles created new environments which hit this prompt, blocking the session for 30+ minutes until timeout.Fix
Added
--no-promptflag to allazd env newandazd initcommands across 3 skills:pre-deploy-checklist.md(7 instances),recipes/azd/README.md,recipes/azd/errors.mdrecipes/azd/environment.md,recipes/azd/errors.mdazure-context.md,recipes/azd/terraform.mdValidation
npm run frontmatter✅npm run references✅