Skip to content

Add azure-project-plan skill and tests#1992

Open
nturinski wants to merge 17 commits intomicrosoft:mainfrom
nturinski:nat/azure-project-plan
Open

Add azure-project-plan skill and tests#1992
nturinski wants to merge 17 commits intomicrosoft:mainfrom
nturinski:nat/azure-project-plan

Conversation

@nturinski
Copy link
Copy Markdown
Member

Description

This PR adds the azure-project-plan skill, which implements a plan-first workflow for creating Azure-centric applications. The skill captures project requirements through interactive UI (vscode_askQuestions), generates a structured .azure/project-plan.md, and auto-chains to azure-project-scaffold after user approval. The SKILL.md inlines all planning context — inference rules, environment variable mappings, a 11-section plan template, canonical project structure, and error contracts — so the agent needs zero external file reads during planning.

As we add more runtimes beyond TypeScript, we should extract runtime-specific content into references/ files — project structure templates per runtime, Blazor-specific frontend guidance, package manager nuances for pip/poetry/dotnet, and test runner patterns for pytest and xUnit. The current all-inlined approach works at ~15K chars but will exceed the 20K skill character budget once we add content for two more runtimes. Moving to the thin-orchestrator pattern used by azure-deploy (small SKILL.md + progressive disclosure via references) will keep the skill maintainable and within token limits.

That being said, everything is currently inline for better performance. Here are the benefits:

Zero external reads — the skill explicitly says "Planning requires ZERO external file reads." The agent gets everything it needs in one load, reducing latency and tool calls.
Self-contained — plan template, inference rules, env-var mappings, project structure, question definitions are all right there. Nothing can be missed due to a failed file read.
Better for creative/generative work — planning requires lots of interrelated context (inference rules feed into questions, which feed into the plan template). Having it all visible helps the agent make coherent decisions.

Checklist

  • Tests pass locally (cd tests && npm test)
  • If modifying skill descriptions: verified routing correctness with integration tests (npm run test:skills:integration -- <skill>)
  • If modifying skill USE FOR / DO NOT USE FOR / PREFER OVER clauses: confirmed no routing regressions for competing skills
  • Version bumped in skill frontmatter (if skill files changed)

Related Issues

Copilot AI review requested due to automatic review settings April 21, 2026 21:42
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a new azure-project-plan skill intended to drive a “plan-first” workflow for Azure-centric apps, and adds unit/trigger/integration tests to validate skill metadata, routing, and basic behavioral expectations.

Changes:

  • Added plugin/skills/azure-project-plan/SKILL.md plus reference docs under plugin/skills/azure-project-plan/references/.
  • Added unit + trigger tests (with snapshots) for the new skill.
  • Added integration tests + helper utilities to exercise plan creation and (intended) auto-chaining behavior.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 13 comments.

Show a summary per file
File Description
plugin/skills/azure-project-plan/SKILL.md New skill definition and inlined plan-first workflow + template content.
plugin/skills/azure-project-plan/references/plan-template.md Human-readable plan template reference (not used at runtime).
plugin/skills/azure-project-plan/references/requirements.md Requirements-gathering reference guidance.
tests/azure-project-plan/unit.test.ts Validates frontmatter + expected content patterns for the skill.
tests/azure-project-plan/triggers.test.ts Validates trigger matching and snapshots extracted keywords/description.
tests/azure-project-plan/__snapshots__/triggers.test.ts.snap Snapshot baselines for trigger keyword extraction.
tests/azure-project-plan/integration.test.ts Agent-session tests for invocation, plan generation, and intended chaining.
tests/azure-project-plan/utils.ts Helper utilities for early termination and tool-call logging in integration tests.

Comment on lines +35 to +43
## ❌ DO NOT Activate When

| User Intent | Correct Skill |
|-------------|---------------|
| Execute approved plan / scaffold backend | **azure-project-scaffold** |
| Docker Compose, emulators, VS Code F5 | **azure-localdev** |
| Add test coverage to scaffolded project | **azure-project-verify** |
| Deploy to Azure | **azure-prepare** |
| Generate Bicep/Terraform | **azure-prepare** |
Copy link

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The skill instructions reference azure-project-scaffold, azure-localdev, and azure-project-verify, but those skills don't exist under plugin/skills/ in this repo. That means the DO NOT Activate redirect guidance and the auto-chain step cannot be executed by the agent as configured in tests (skillDirectories only include plugin/skills). Either add those skills in this PR, or change these references to existing skill names (and update tests accordingly).

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These skills will all be added later in a later PR.

Comment thread plugin/skills/azure-project-plan/SKILL.md
Comment on lines +67 to +103
## 📦 Context Management

> **Planning requires ZERO external file reads.** All context inlined below.

| Phase | External File Reads |
|-------|-------------------|
| Planning | **None** — all inlined below |

---

## ═══════════════════════════════════════════════════
## PHASE 1: PLANNING
## ═══════════════════════════════════════════════════

### Step 1: Detect Workspace

**BEFORE gathering requirements**, scan workspace:

#### 1a. Scan for Existing Project Files

| Signal | Detection Method | Action |
|--------|-----------------|--------|
| `package.json` with deps | Scan `dependencies` / `devDependencies` | Detect runtime (Node.js), frameworks, test runners |
| `pyproject.toml` or `requirements.txt` | Scan for Python | Detect runtime (Python), frameworks |
| `*.csproj` or `*.sln` | Scan for .NET | Detect runtime (.NET), frameworks |
| `host.json` or `local.settings.json` | Scan root/src dirs | Azure Functions exists — augment, don't recreate |
| Test files or config | Scan for `*.test.*`, `*.spec.*`, `vitest.config.*`, `jest.config.*` | Detect test infra — respect it |
| `docker-compose.yml` | Scan root | Emulators may be configured |

> ⚠️ Check actual **workspace files** — not user prompt.

#### 1b. Check for `.azure/plan.md` (Deployment Plan)

| Check | Action |
|-------|--------|
| `.azure/plan.md` exists | **Read it.** Extract Architecture → service mapping. Use these — do NOT re-ask user. |
| `.azure/plan.md` does not exist | Proceed normally — detect from code, ask user as needed. |
Copy link

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The skill says “Planning requires ZERO external file reads” but Step 1/2 explicitly instruct reading workspace files and reading .azure/plan.md when present. If the intent is “no external reference file reads (everything needed is inlined)”, consider rephrasing this section to avoid contradicting the required workspace scan / .azure/plan.md read.

Copilot uses AI. Check for mistakes.
Comment thread plugin/skills/azure-project-plan/SKILL.md Outdated
Comment on lines +289 to +296
// The skill should either invoke scaffold or mention it as next step
const chained = didAutoChainToScaffold(agentMetadata);
if (!chained) {
agentMetadata.testComments.push(
"⚠️ Expected auto-chain to azure-project-scaffold after plan approval."
);
}
expect(typeof chained).toBe("boolean");
Copy link

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This integration test doesn’t currently assert the auto-chain behavior: it computes chained but then only checks expect(typeof chained).toBe("boolean"), which will always pass. If auto-chaining is required behavior, this should be a hard assertion (e.g., expect(chained).toBe(true) and ideally assert an actual skill tool invocation), otherwise the test suite won’t catch regressions.

Suggested change
// The skill should either invoke scaffold or mention it as next step
const chained = didAutoChainToScaffold(agentMetadata);
if (!chained) {
agentMetadata.testComments.push(
"⚠️ Expected auto-chain to azure-project-scaffold after plan approval."
);
}
expect(typeof chained).toBe("boolean");
// Auto-chaining after approval is required behavior for this test.
const chained = didAutoChainToScaffold(agentMetadata);
if (!chained) {
agentMetadata.testComments.push(
"⚠️ Expected auto-chain to azure-project-scaffold after plan approval."
);
}
expect(chained).toBe(true);
expect(isSkillInvoked(agentMetadata, "azure-project-scaffold")).toBe(true);

Copilot uses AI. Check for mistakes.
Comment thread tests/azure-project-plan/unit.test.ts
Comment on lines +11 to +32
export const MAX_PLAN_API_CALLS = 8;

/**
* Early termination that combines skill invocation check with an API call ceiling.
* Use for skill-invocation rate tests where we just need to know if skill was called.
*/
export function earlyTerminateForPlan(metadata: AgentMetadata, skillName: string): boolean {
if (shouldEarlyTerminateForSkillInvocation(metadata, skillName)) return true;
const messageCount = metadata.events.filter(e => e.type === "assistant.message").length;
return messageCount >= MAX_PLAN_API_CALLS;
}

/**
* Early termination that only caps API calls — does NOT terminate on skill invocation.
* Use for plan-generation tests where the agent needs to finish writing files.
*/
export function earlyTerminateOnApiCeiling(metadata: AgentMetadata): boolean {
const messageCount = metadata.events.filter(e => e.type === "assistant.message").length;
return messageCount >= MAX_PLAN_API_CALLS;
}

/**
Copy link

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAX_PLAN_API_CALLS is described and implemented as a ceiling on assistant message count (assistant.message events), not API call count. Consider renaming it (and related comments/function names like earlyTerminateOnApiCeiling) to reflect what’s actually being capped, or update the logic to use real API call metrics if that’s the intent.

Suggested change
export const MAX_PLAN_API_CALLS = 8;
/**
* Early termination that combines skill invocation check with an API call ceiling.
* Use for skill-invocation rate tests where we just need to know if skill was called.
*/
export function earlyTerminateForPlan(metadata: AgentMetadata, skillName: string): boolean {
if (shouldEarlyTerminateForSkillInvocation(metadata, skillName)) return true;
const messageCount = metadata.events.filter(e => e.type === "assistant.message").length;
return messageCount >= MAX_PLAN_API_CALLS;
}
/**
* Early termination that only caps API calls does NOT terminate on skill invocation.
* Use for plan-generation tests where the agent needs to finish writing files.
*/
export function earlyTerminateOnApiCeiling(metadata: AgentMetadata): boolean {
const messageCount = metadata.events.filter(e => e.type === "assistant.message").length;
return messageCount >= MAX_PLAN_API_CALLS;
}
/**
export const MAX_PLAN_ASSISTANT_MESSAGES = 8;
/** @deprecated Use MAX_PLAN_ASSISTANT_MESSAGES. Kept as a compatibility alias. */
export const MAX_PLAN_API_CALLS = MAX_PLAN_ASSISTANT_MESSAGES;
/**
* Early termination that combines skill invocation check with an assistant message ceiling.
* Use for skill-invocation rate tests where we just need to know if skill was called.
*/
export function earlyTerminateForPlan(metadata: AgentMetadata, skillName: string): boolean {
if (shouldEarlyTerminateForSkillInvocation(metadata, skillName)) return true;
const messageCount = metadata.events.filter(e => e.type === "assistant.message").length;
return messageCount >= MAX_PLAN_ASSISTANT_MESSAGES;
}
/**
* Early termination that only caps assistant messages does NOT terminate on skill invocation.
* Use for plan-generation tests where the agent needs to finish writing files.
*/
export function earlyTerminateOnAssistantMessageCeiling(metadata: AgentMetadata): boolean {
const messageCount = metadata.events.filter(e => e.type === "assistant.message").length;
return messageCount >= MAX_PLAN_ASSISTANT_MESSAGES;
}
/**
* @deprecated Use earlyTerminateOnAssistantMessageCeiling.
* Early termination that only caps assistant messages does NOT terminate on skill invocation.
*/
export function earlyTerminateOnApiCeiling(metadata: AgentMetadata): boolean {
return earlyTerminateOnAssistantMessageCeiling(metadata);
}
/**

Copilot uses AI. Check for mistakes.

## 8. Service Dependency Classification

> Classify each external service to determine failure handling. See [resilience.md](references/resilience.md).
Copy link

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reference file includes several links to sibling reference docs (e.g., testing.md, resilience.md, database-integrity.md) that are not present in plugin/skills/azure-project-plan/references/ in this PR. Either add the referenced files, or adjust/remove the links so the reference doc doesn’t contain dead links.

Suggested change
> Classify each external service to determine failure handling. See [resilience.md](references/resilience.md).
> Classify each external service to determine failure handling. Use the project plan's resilience guidance to determine whether a dependency is essential or an enhancement.

Copilot uses AI. Check for mistakes.
Comment on lines +9 to +16
**ALWAYS attempt to infer requirements from workspace before asking user.**

Run workspace detection from SKILL.md Step 0 first. After detection, you should know:
- Runtime and language (from existing project files)
- Existing test infrastructure (from test configs and deps)
- Azure SDKs already in use (from dep lists)
- Frontend framework (from existing frontend code)

Copy link

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reference doc says “Run workspace detection from SKILL.md Step 0 first”, but the SKILL.md in this PR starts at “Step 1: Detect Workspace”. Update the step numbering/reference so the docs stay consistent.

Copilot uses AI. Check for mistakes.

> What test runner do you want to use?

Present options based on selected runtime. Include brief pros/cons. See [testing.md](testing.md) → Test Runner Quick Reference.
Copy link

Copilot AI Apr 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reference doc links to testing.md (“See testing.md → Test Runner Quick Reference”), but testing.md is not present in this skill’s references/ directory in this PR. Add the file or adjust the link/text to avoid dead references.

Suggested change
Present options based on selected runtime. Include brief pros/cons. See [testing.md](testing.md) → Test Runner Quick Reference.
Present options based on selected runtime. Include brief pros/cons and a short test runner quick reference in your response.

Copilot uses AI. Check for mistakes.
license: MIT
metadata:
author: Microsoft
version: "1.0.0"
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We now handle version updates automatically. Set version to "0.0.0-placeholder" here and create a version.json file next to SKILL.md with the version set to "1.0"--see the other skills for examples.

Copy link
Copy Markdown
Collaborator

@jongio jongio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All three CI checks fail because the version field is not the expected placeholder. A few test/SKILL.md alignment issues will surface once CI gets past the stamping step:

Issues to address:

  • SKILL.md:7 - version must be "0.0.0-placeholder", not "1.0.0" (blocks all CI)
  • unit.test.ts:96 - asserts azure-localdev but SKILL.md uses azure-local-development
  • unit.test.ts:286 - expects references/ directory but no reference files in this PR

The copilot bot's findings about the section count mismatch (12 vs 11) and runtime option inconsistency are also valid and worth addressing.

Comment thread plugin/skills/azure-project-plan/SKILL.md Outdated
Comment thread tests/azure-project-plan/unit.test.ts Outdated

test("DO NOT Activate table references correct redirect skills", () => {
expect(skill.content).toContain("azure-project-scaffold");
expect(skill.content).toContain("azure-localdev");
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[HIGH] SKILL.md uses azure-local-development but this asserts azure-localdev. One or the other needs updating.

Suggested change
expect(skill.content).toContain("azure-localdev");
expect(skill.content).toContain("azure-local-development");

Comment thread tests/azure-project-plan/unit.test.ts Outdated
});

describe("References Directory", () => {
test("has a references/ subdirectory", () => {
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] The PR diff doesn't include any files under references/. If the directory was removed in a later commit, this assertion will fail.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 23, 2026 16:53
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 9 comments.

Comment thread tests/azure-project-plan/unit.test.ts Outdated

test("DO NOT Activate table references correct redirect skills", () => {
expect(skill.content).toContain("azure-project-scaffold");
expect(skill.content).toContain("azure-localdev");
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test asserts the content contains azure-localdev, but the skill text currently uses azure-local-development in the DO NOT Activate table. Align the expected redirect skill name with the SKILL.md (or update SKILL.md if azure-localdev is the intended canonical name) so the unit test can pass.

Suggested change
expect(skill.content).toContain("azure-localdev");
expect(skill.content).toContain("azure-local-development");

Copilot uses AI. Check for mistakes.
Comment thread tests/azure-project-plan/unit.test.ts Outdated
Comment on lines +125 to +129
test("defines all 12 plan template sections", () => {
expect(skill.content).toContain("## 1. Project Overview");
expect(skill.content).toContain("## 2. Runtime & Framework");
expect(skill.content).toContain("## 3. Test Runner & Configuration");
expect(skill.content).toContain("## 4. Services Required");
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These assertions expect the plan template to include ## 10. Test Suite Plan, but the current SKILL.md plan template does not include that section. Update the SKILL.md template or adjust the expected section list so they agree.

Copilot uses AI. Check for mistakes.
Comment thread tests/azure-project-plan/unit.test.ts Outdated
Comment on lines +286 to +288
test("has a references/ subdirectory", () => {
const refsDir = join(skill.path, "references");
expect(existsSync(refsDir)).toBe(true);
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test requires plugin/skills/azure-project-plan/references/ to exist, but the skill directory currently only contains SKILL.md (no references/ folder). Either add an empty references/ directory (preferred if you want a consistent skill layout) or remove/relax this assertion.

Suggested change
test("has a references/ subdirectory", () => {
const refsDir = join(skill.path, "references");
expect(existsSync(refsDir)).toBe(true);
test("has a references/ subdirectory when the skill uses reference files", () => {
const refsDir = join(skill.path, "references");
const referencesAreUsed = /references\//i.test(skill.content);
if (referencesAreUsed) {
expect(existsSync(refsDir)).toBe(true);
} else {
expect(existsSync(refsDir)).toBe(existsSync(refsDir));
}

Copilot uses AI. Check for mistakes.
Comment on lines +69 to +73
> **Planning requires ZERO external file reads.** All context inlined below.

| Phase | External File Reads |
|-------|-------------------|
| Planning | **None** — all inlined below |
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This claims “Planning requires ZERO external file reads” and the table says Planning = None, but later steps require scanning workspace files and even reading .azure/plan.md when present. Please clarify what “external” means here (e.g., “no reads from skill references/ docs”), or update the table/wording so it’s not self-contradictory.

Suggested change
> **Planning requires ZERO external file reads.** All context inlined below.
| Phase | External File Reads |
|-------|-------------------|
| Planning | **None**all inlined below |
> **Planning requires ZERO external reference/doc reads.** All required planning guidance is inlined below. Workspace inspection is still allowed for project detection and reuse of existing planning artifacts.
| Phase | External File Reads |
|-------|-------------------|
| Planning | **No skill-reference/doc reads required**guidance is inlined below; workspace file inspection is allowed (for example existing project files and `.azure/project-plan.md` when present) |

Copilot uses AI. Check for mistakes.
Comment thread tests/azure-project-plan/integration.test.ts Outdated
Comment thread tests/azure-project-plan/unit.test.ts
"allowFreeformInput": false,
"options": [
{ "label": "TypeScript", "description": "Node.js — Azure Functions v4 programming model", "recommended": true },
{ "label": "C# (.NET 8)", "description": "Isolated worker model" }
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example vscode_askQuestions JSON uses C# (.NET 8) for the runtime option, but the earlier Q2 definition lists C# (.NET 10). Align these labels/versions so the guidance is consistent.

Suggested change
{ "label": "C# (.NET 8)", "description": "Isolated worker model" }
{ "label": "C# (.NET 10)", "description": "Isolated worker model" }

Copilot uses AI. Check for mistakes.
Comment on lines +1 to +5
---
name: azure-project-plan
description: "Plan and design an Azure-centric project with user requirements gathering and interactive plan approval. Generates .azure/project-plan.md, then auto-chains to azure-project-scaffold. WHEN: \"plan project\", \"design app\", \"new project\", \"project requirements\", \"create project plan\", \"plan my app\", \"what should I build\", \"scaffold project\", \"new Azure app\", \"create testable app\", \"new API project\", \"full-stack Azure app\", \"new project with tests\", \"create app\", \"bootstrap project\", \"new fullstack project\", \"testable API\", \"create functions project\"."
license: MIT
metadata:
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI will fail due to the Skill Structure check that requires tests/skills.json to exactly enumerate all directories under plugin/skills/. This PR adds plugin/skills/azure-project-plan/ but does not update tests/skills.json to include it (and to schedule it in integrationTestSchedule).

Copilot uses AI. Check for mistakes.
Comment on lines +144 to +146
**Q2: Runtime** (ask if not detectable)
- Options: `TypeScript` (Node.js — Functions v4), `C# (.NET 10)` (Isolated worker)
- `allowFreeformInput`: false
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The runtime version is inconsistent: Q2 lists C# (.NET 10) here, but the later example vscode_askQuestions JSON uses C# (.NET 8). Pick one and make it consistent across the doc so the agent doesn’t present conflicting options.

Copilot uses AI. Check for mistakes.
Copilot AI review requested due to automatic review settings April 23, 2026 17:03
nturinski and others added 2 commits April 23, 2026 10:04
Co-authored-by: Jon Gallant <2163001+jongio@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Comment thread tests/azure-project-plan/unit.test.ts Fixed
Comment thread tests/azure-project-plan/unit.test.ts Fixed
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Comment on lines +289 to +297
// The skill should either invoke scaffold or mention it as next step
const chained = didAutoChainToScaffold(agentMetadata);
if (!chained) {
agentMetadata.testComments.push(
"⚠️ Expected auto-chain to azure-project-scaffold after plan approval."
);
}
expect(chained).toBe(true);
});
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test is titled as verifying auto-chaining after approval, but it never asserts that chaining actually happened. It only checks typeof chained === "boolean", so the test will pass even when the chain fails (it just adds a warning comment). Consider asserting chained is true (or explicitly mark/skip the test until azure-project-scaffold is available) so regressions are caught.

Copilot uses AI. Check for mistakes.
Comment on lines +11 to +29
export const MAX_PLAN_API_CALLS = 8;

/**
* Early termination that combines skill invocation check with an API call ceiling.
* Use for skill-invocation rate tests where we just need to know if skill was called.
*/
export function earlyTerminateForPlan(metadata: AgentMetadata, skillName: string): boolean {
if (shouldEarlyTerminateForSkillInvocation(metadata, skillName)) return true;
const messageCount = metadata.events.filter(e => e.type === "assistant.message").length;
return messageCount >= MAX_PLAN_API_CALLS;
}

/**
* Early termination that only caps API calls — does NOT terminate on skill invocation.
* Use for plan-generation tests where the agent needs to finish writing files.
*/
export function earlyTerminateOnApiCeiling(metadata: AgentMetadata): boolean {
const messageCount = metadata.events.filter(e => e.type === "assistant.message").length;
return messageCount >= MAX_PLAN_API_CALLS;
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MAX_PLAN_API_CALLS is derived from counting assistant.message events, not actual API call count. The current name is misleading and makes future tuning harder. Consider renaming it to reflect what’s being capped (e.g., max assistant messages), or switch to using the actual tokenUsage.apiCallCount if that’s what you intend to limit.

Suggested change
export const MAX_PLAN_API_CALLS = 8;
/**
* Early termination that combines skill invocation check with an API call ceiling.
* Use for skill-invocation rate tests where we just need to know if skill was called.
*/
export function earlyTerminateForPlan(metadata: AgentMetadata, skillName: string): boolean {
if (shouldEarlyTerminateForSkillInvocation(metadata, skillName)) return true;
const messageCount = metadata.events.filter(e => e.type === "assistant.message").length;
return messageCount >= MAX_PLAN_API_CALLS;
}
/**
* Early termination that only caps API calls does NOT terminate on skill invocation.
* Use for plan-generation tests where the agent needs to finish writing files.
*/
export function earlyTerminateOnApiCeiling(metadata: AgentMetadata): boolean {
const messageCount = metadata.events.filter(e => e.type === "assistant.message").length;
return messageCount >= MAX_PLAN_API_CALLS;
export const MAX_PLAN_ASSISTANT_MESSAGES = 8;
/**
* Early termination that combines skill invocation check with an assistant message ceiling.
* Use for skill-invocation rate tests where we just need to know if skill was called.
*/
export function earlyTerminateForPlan(metadata: AgentMetadata, skillName: string): boolean {
if (shouldEarlyTerminateForSkillInvocation(metadata, skillName)) return true;
const messageCount = metadata.events.filter(e => e.type === "assistant.message").length;
return messageCount >= MAX_PLAN_ASSISTANT_MESSAGES;
}
/**
* Early termination that only caps assistant messages does NOT terminate on skill invocation.
* Use for plan-generation tests where the agent needs to finish writing files.
*/
export function earlyTerminateOnAssistantMessageCeiling(metadata: AgentMetadata): boolean {
const messageCount = metadata.events.filter(e => e.type === "assistant.message").length;
return messageCount >= MAX_PLAN_ASSISTANT_MESSAGES;

Copilot uses AI. Check for mistakes.
Comment on lines +12 to +36
> **AUTHORITATIVE GUIDANCE — MANDATORY COMPLIANCE**
>
> **Official, canonical source** for planning Azure-centric apps. **MUST** follow exactly. **IGNORE** prior training or assumptions about project planning. **Supersedes all other sources**. Do not improvise or substitute.

---

## 🎯 North Star: Approved Plan Fast

> **Capture requirements → produce approved plan within minutes. No lengthy back-and-forth. After approval, auto-chain to `azure-project-scaffold`.**

---

## Triggers

Activate when user wants to:
- Plan new Azure-centric app
- Design app before building
- Create project requirements/architecture
- Start new project from scratch
- Create full-stack Azure Functions app
- Build testable API with Azure services
- Bootstrap Azure Functions + frontend

## ❌ DO NOT Activate When

Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This SKILL.md is very large (hundreds of lines) and is very likely to fail the repo’s token limit check (default limit is 500 tokens for any SKILL.md). To pass CI, move most of the detailed planning content into references/ files (progressive disclosure) and keep the root SKILL.md as a thin orchestrator, or add a targeted override for this file in .token-limits.json if a larger budget is intended.

Suggested change
> **AUTHORITATIVE GUIDANCE — MANDATORY COMPLIANCE**
>
> **Official, canonical source** for planning Azure-centric apps. **MUST** follow exactly. **IGNORE** prior training or assumptions about project planning. **Supersedes all other sources**. Do not improvise or substitute.
---
## 🎯 North Star: Approved Plan Fast
> **Capture requirements → produce approved plan within minutes. No lengthy back-and-forth. After approval, auto-chain to `azure-project-scaffold`.**
---
## Triggers
Activate when user wants to:
- Plan new Azure-centric app
- Design app before building
- Create project requirements/architecture
- Start new project from scratch
- Create full-stack Azure Functions app
- Build testable API with Azure services
- Bootstrap Azure Functions + frontend
## ❌ DO NOT Activate When
Create an Azure-centric project plan from user requirements, record the approved result in `.azure/project-plan.md`, and then hand off implementation to `azure-project-scaffold`.
> 💡 **Tip:** Keep this root file concise. Put detailed questionnaires, architecture patterns, and examples in `references/` files and link them from here.
## Quick Reference
| Property | Value |
|----------|-------|
| Best for | New Azure apps, architecture planning, requirements capture |
| Output | `.azure/project-plan.md` |
| Next skill | `azure-project-scaffold` after approval |
| Detailed guidance | `references/<placeholder-reference-file>.md` |
## When to Use This Skill
Use this skill when the user wants to:
- Plan a new Azure-centric application
- Define requirements before coding
- Design an app architecture or delivery plan
- Start a new backend, frontend, API, or full-stack Azure project
Do not use this skill when the user wants to execute an already approved plan, work on local emulation, add verification, or deploy infrastructure.
## MCP Tools
| Tool | Parameters | Required/Optional | Purpose |
|------|------------|-------------------|---------|
| None in this root skill | N/A | N/A | Use linked references for detailed planning patterns and hand off after plan approval |
## Workflow/Steps
1. Gather user goals, constraints, and preferred Azure services.
2. Produce a concise proposed plan for confirmation.
3. Save the approved plan to `.azure/project-plan.md`.
4. Transition to `azure-project-scaffold` to implement the approved plan.
## Error Handling
| Error | Message | Remediation |
|-------|---------|-------------|
| Missing requirements | User intent is too vague | Ask for app type, users, core features, and deployment constraints |
| Wrong skill selected | Request is for build/deploy/verify tasks | Route to the correct skill listed below |
| No approval | Plan has not been confirmed | Do not scaffold until the user approves the plan |

Copilot uses AI. Check for mistakes.
Comment on lines +35 to +44
## ❌ DO NOT Activate When

| User Intent | Correct Skill |
|-------------|---------------|
| Execute approved plan / scaffold backend | **azure-project-scaffold** |
| Docker Compose, emulators, VS Code F5 | **azure-local-development** |
| Add test coverage to scaffolded project | **azure-project-verify** |
| Deploy to Azure | **azure-prepare** |
| Generate Bicep/Terraform | **azure-prepare** |

Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Many of the markdown tables use || at the start of rows (e.g., the “DO NOT Activate When” table). In Markdown this creates an extra empty column and renders incorrectly. Use a single leading/trailing | per row (| col1 | col2 |) consistently.

Copilot uses AI. Check for mistakes.
Comment on lines +144 to +146
**Q2: Runtime** (ask if not detectable)
- Options: `TypeScript` (Node.js — Functions v4), `C# (.NET 10)` (Isolated worker)
- `allowFreeformInput`: false
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Runtime guidance is internally inconsistent: workspace detection/inference includes Python (pyproject.toml), the plan template allows {TypeScript / Python / C#}, and Q4 includes a Python row, but Q2 “Runtime” options omit Python. Also Q2 mentions “C# (.NET 10)” while the JSON example uses “C# (.NET 8)”. Align the supported runtime list and .NET version across questions and the example so agents/tests don’t drift.

Copilot uses AI. Check for mistakes.
Copilot AI review requested due to automatic review settings April 23, 2026 17:53
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

**Q3: Data Stores** (ask if not detectable from SDK imports or `.azure/plan.md`)
- Options: `Blob Storage` (Files/images), `Queue Storage` (Async queue), `PostgreSQL` (Relational), `CosmosDB` (NoSQL), `Redis` (Cache), `Azure SQL` (Managed SQL Server)
- `multiSelect`: true
- `allowFreeformInput`: true
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Q3 says allowFreeformInput: true, but the example vscode_askQuestions JSON sets allowFreeformInput: false for “Data Stores”. Make the definition and example consistent (and likewise ensure multiSelect/freeform behavior matches what you want).

Suggested change
- `allowFreeformInput`: true
- `allowFreeformInput`: false

Copilot uses AI. Check for mistakes.
Comment thread tests/azure-project-plan/integration.test.ts
Comment thread plugin/skills/azure-project-plan/SKILL.md
Comment thread plugin/skills/azure-project-plan/SKILL.md Outdated
Copilot AI review requested due to automatic review settings April 23, 2026 19:33
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Comment thread plugin/skills/azure-project-plan/SKILL.md Outdated
Comment thread tests/azure-project-plan/integration.test.ts Outdated
Comment thread tests/skills.json
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 24, 2026 17:59
nturinski and others added 2 commits April 24, 2026 10:59
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Comment on lines +98 to +103
#### 1b. Check for `.azure/plan.md` (Deployment Plan)

| Check | Action |
|-------|--------|
| `.azure/plan.md` exists | **Read it.** Extract Architecture → service mapping. Use these — do NOT re-ask user. |
| `.azure/plan.md` does not exist | Proceed normally — detect from code, ask user as needed. |
Copy link

Copilot AI Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This skill instructs checking/reading .azure/plan.md as a “Deployment Plan”, but azure-prepare explicitly warns not to use .azure/plan.md and standardizes on .azure/deployment-plan.md. Using a different filename here will make cross-skill workflows inconsistent. Consider switching this to .azure/deployment-plan.md (or clearly distinguishing a different artifact name) and update the later inference rules/template references accordingly.

Suggested change
#### 1b. Check for `.azure/plan.md` (Deployment Plan)
| Check | Action |
|-------|--------|
| `.azure/plan.md` exists | **Read it.** Extract Architecture → service mapping. Use these — do NOT re-ask user. |
| `.azure/plan.md` does not exist | Proceed normally — detect from code, ask user as needed. |
#### 1b. Check for `.azure/deployment-plan.md` (Deployment Plan)
| Check | Action |
|-------|--------|
| `.azure/deployment-plan.md` exists | **Read it.** Extract Architecture → service mapping. Use these — do NOT re-ask user. |
| `.azure/deployment-plan.md` does not exist | Proceed normally — detect from code, ask user as needed. |

Copilot uses AI. Check for mistakes.
Comment on lines +134 to +136
**Use `vscode_askQuestions`** for interactive quick-pick UI. Never plain-text chat. Batch ALL unanswered into **single** call.

After applying Inference Rules, remove answered questions. If ALL answered by inference, skip call, proceed to Step 3.
Copy link

Copilot AI Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rule 4 says to always use vscode_askQuestions, but later you say “If ALL answered by inference, skip call”. These directions conflict. Make the rule conditional (e.g., “use vscode_askQuestions when there are any unanswered questions”) so agents don’t get contradictory instructions.

Suggested change
**Use `vscode_askQuestions`** for interactive quick-pick UI. Never plain-text chat. Batch ALL unanswered into **single** call.
After applying Inference Rules, remove answered questions. If ALL answered by inference, skip call, proceed to Step 3.
After applying Inference Rules, remove answered questions. **If any questions remain unanswered, use `vscode_askQuestions`** for interactive quick-pick UI. Never plain-text chat. Batch ALL unanswered into **single** call.
If ALL questions are answered by inference, skip the `vscode_askQuestions` call and proceed to Step 3.

Copilot uses AI. Check for mistakes.
Comment on lines +1 to +5
---
name: azure-project-plan
description: "Plan and design an Azure-centric project with user requirements gathering and interactive plan approval. Generates .azure/project-plan.md, then auto-chains to azure-project-scaffold. PREFER OVER azure-prepare when the user wants a NEW project from scratch (azure-prepare ships existing code). WHEN: \"plan project\", \"design app\", \"new project\", \"project requirements\", \"create project plan\", \"plan my app\", \"what should I build\", \"scaffold project\", \"new Azure app\", \"create testable app\", \"new API project\", \"full-stack Azure app\", \"new project with tests\", \"create app\", \"bootstrap project\", \"new fullstack project\", \"testable API\", \"create functions project\". DO NOT USE FOR: shipping existing codebases to Azure (use azure-prepare), authoring Bicep/Terraform for already-built apps (use azure-prepare)."
license: MIT
metadata:
Copy link

Copilot AI Apr 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new skill folder is missing the per-skill version.json that other plugin/skills/* skills include for NBGV versioning. Without it, gulp version stamping will fall back to the nearest parent version.json (e.g., plugin/version.json), which breaks independent per-skill versioning and makes future releases harder to manage. Add plugin/skills/azure-project-plan/version.json with a skill-specific version and pathFilters: ["."] (matching other skills).

Copilot uses AI. Check for mistakes.

> **AUTHORITATIVE GUIDANCE — MANDATORY COMPLIANCE**
>
> **Official, canonical source** for planning Azure-centric apps. **MUST** follow exactly. **IGNORE** prior training or assumptions about project planning. **Supersedes all other sources**. Do not improvise or substitute.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will trigger 3P security alerts. We should avoid asking the model to ignore any of its training or policies.


## North Star: Approved Plan Fast

> **Capture requirements → produce approved plan within minutes. No lengthy back-and-forth. After approval, auto-chain to `azure-project-scaffold`.**
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should add this skill in the same PR otherwise it won't have anything to chain to.


---

## Triggers
Copy link
Copy Markdown
Member

@JasonYeMSFT JasonYeMSFT Apr 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The trigger phrases aren't very useful in the skill content. They won't help the model decide whether to load the skill or not.


---

## ❌ PLAN-FIRST WORKFLOW — MANDATORY
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ❌ feels odd for the first step for the agent to take.

@@ -0,0 +1,519 @@
---
name: azure-project-plan
description: "Plan and design an Azure-centric project with user requirements gathering and interactive plan approval. Generates .azure/project-plan.md, then auto-chains to azure-project-scaffold. PREFER OVER azure-prepare when the user wants a NEW project from scratch (azure-prepare ships existing code). WHEN: \"plan project\", \"design app\", \"new project\", \"project requirements\", \"create project plan\", \"plan my app\", \"what should I build\", \"scaffold project\", \"new Azure app\", \"create testable app\", \"new API project\", \"full-stack Azure app\", \"new project with tests\", \"create app\", \"bootstrap project\", \"new fullstack project\", \"testable API\", \"create functions project\". DO NOT USE FOR: shipping existing codebases to Azure (use azure-prepare), authoring Bicep/Terraform for already-built apps (use azure-prepare)."
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the description can be a little more explicit to encourage the agent to only use the skill when the workspace is empty or almost empty.

console.log(`\n🎯 ${SKILL_NAME} invoked: ${invoked}`);

softCheckPlanSkills(agentMetadata);
expect(typeof invoked).toBe("boolean");
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this was used for local development, please remove it before merging.


const SKILL_NAME = "azure-project-plan";
const RUNS_PER_PROMPT = 3;
const invocationRateThreshold = 0.3;
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should set this to a higher value, we generally use 80%.

Copy link
Copy Markdown
Collaborator

@kvenkatrajan kvenkatrajan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My first take on this, this shouldnt be a new skill, this has heavy overlap with azure-prepare and consider folding into it and add scaffolding considerations into it.

const mentionsReact = /react/i.test(combined);
const mentionsStorage = /blob|storage/i.test(combined);
if (!mentionsReact || !mentionsStorage) {
agentMetadata.testComments.push(
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inference often happens in the reasoning tokens, not in the assistant messages sent back to the user. You may want to add new util function to extract the reasoning tokens and check inference-like conditions from there.

@JasonYeMSFT
Copy link
Copy Markdown
Member

~15K chars but will exceed the 20K skill character budget once we add content for two more runtimes

The 15K char budget is for the aggregated descriptions of all skills. The content of each skill doesn't count. Adding more runtimes into the body of the skill or as reference files won't increase the usage of the char budget. We just need to make sure the skill description is clear and precise.

Copy link
Copy Markdown
Collaborator

@jongio jongio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Previous issues addressed. One new item:

  • tests/azure-project-plan/utils.ts:72 - didAutoChainToScaffold fallback regex will match any assistant message that mentions "scaffolding" in natural language, not just actual skill invocation

The .NET 8 vs .NET 10 inconsistency (SKILL.md line 152 vs 196) and missing version.json were already flagged - confirming they're still present in the latest push. Merge conflicts also need resolving.

export function didAutoChainToScaffold(agentMetadata: AgentMetadata): boolean {
if (isSkillInvoked(agentMetadata, "azure-project-scaffold")) return true;
// Fallback: check if assistant mentions scaffolding as next step
const content = getAllAssistantMessages(agentMetadata);
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] didAutoChainToScaffold fallback regex /scaffold(ing)?\b/i matches any assistant message that casually mentions "scaffolding". Since azure-project-scaffold doesn't exist in the repo yet, isSkillInvoked on line 69 always returns false and the auto-chain test at integration.test.ts:949 depends entirely on this loose pattern match. Any response like "you can run scaffolding next" would pass without actual chaining.

Consider tightening the fallback or making the auto-chain test a soft-check until the target skill lands.

@nturinski
Copy link
Copy Markdown
Member Author

My first take on this, this shouldnt be a new skill, this has heavy overlap with azure-prepare and consider folding into it and add scaffolding considerations into it.

Fair concern, but I think the intentions between the two skills are quite different. Specifically, when I tried to start with an empty workspace with azure-prepare, it was pretty unreliable at creating a functioning application. Folding it into azure-prepare would probably add way too much context and I fear that it would overwhelm the skill chain since it has to do azure-validate and azure-deploy immediately after.

These are the different scenarios that I imagine:

azure-project-plan → azure-project-scaffold: greenfield. User has nothing, we design and write the app code (handlers, services, migrations, frontend). Plan artifact is .azure/project-plan.md with routes and entities.
azure-prepare: existing code. User has a working app, we write the infra (Bicep/Terraform, azure.yaml). Plan artifact is deployment-plan.md with recipe and services.

Though since this PR only has azure-project-prepare in it right now, it's hard to show how it'll work with scaffold so I will work on getting the remaining draft PRs for the skills so that the user story is a little more cohesive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants