feat(scripts): add instruction applyTo scope Pester suite - #1640
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1640 +/- ##
==========================================
+ Coverage 80.55% 85.62% +5.06%
==========================================
Files 112 86 -26
Lines 18679 12549 -6130
==========================================
- Hits 15047 10745 -4302
+ Misses 3632 1804 -1828
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Document that planner-related Pester tests are intentionally split between `scripts/tests/linting/` (rule validators) and `scripts/tests/security/` (artifact signing). Consolidation under a single `planner/` subdirectory is deferred to backlog item IV-011. Reference: .copilot-tracking/reviews/2026-05-22/stacked-prs-from-pr-1497-review.md (IV-011)
bindsi
left a comment
There was a problem hiding this comment.
Approving — triage review.
Verdict: LGTM. The applyTo-scope Pester suite is a solid regression guard (phase-narrowed applyTo for security/** and rai-planning/**, empty-allowlist enforcement).
NIT (optional): the README note references a .copilot-tracking/... backlog path (IV-011) which is gitignored and won't exist in the repo — minor doc smell. Non-blocking.
) # Pull Request ## Description This PR migrates the planner disclaimer single-source-of-truth (SSOT) from the standalone YAML config at `.github/config/disclaimers.yml` to a markdown authoring contract embedded in `.github/instructions/shared/disclaimer-language.instructions.md`. It stacks on **#1497** (`feat/sssc-planner-rai-parity`) because removing `disclaimers.yml` would conflict with any branch in the stack that does not include this migration. The instruction file now defines the disclaimer corpus as H2 sections whose slugified headings (for example, *RAI Planning* → `rai-planner`) are the parser-derived keys. Each section carries a CAUTION blockquote that captures the session-start disclaimer, plus prose paragraphs joined into a single string by the parser. A short *Authoring Contract* preamble documents the rules so future edits stay machine-readable without referring to the validator source. The validator, **`scripts/linting/Validate-PlannerArtifacts.ps1`**, swaps its YAML loader for `Import-DisclaimerSource`, which parses the new markdown SSOT, derives planner keys from heading slugs, extracts CAUTION blockquotes (stripping the `> [!CAUTION]` prefix), and joins multi-line prose. The function's `-DisclaimerConfigPath` parameter is renamed to `-DisclaimerSourcePath` to reflect the markdown origin. A new contract suite, **`scripts/tests/linting/Test-DisclaimerArtifacts.Tests.ps1`** (8 cases), asserts that: * the SSOT instruction file parses into the expected planner keys with non-empty CAUTION blockquotes and prose; and * every planner identity file (RAI and Security) carries the required *Session Start Display* literal and an *Exit Point Reminder* section that references each named exit point. To satisfy the second assertion, this PR also adds the *Disclaimer and Attribution Protocol* section to **`.github/instructions/security/identity.instructions.md`**, sourced verbatim from the snapshot. The hunks in **`scripts/tests/linting/Validate-PlannerArtifacts.Tests.ps1`** are trimmed to disclaimer-related coverage only; the SSSC-parity hunks are deferred to a later PR in the stack. The three regenerated **`plugins/*/README.md`** files reflect the disclaimer SSOT description text and are produced by `npm run plugin:generate`. ## Related Issue(s) Stacks on #1497. No other issue references in commits or branch name. ## Type of Change Select all that apply: **Code & Documentation:** * [ ] Bug fix (non-breaking change fixing an issue) * [ ] New feature (non-breaking change adding functionality) * [x] Breaking change (fix or feature causing existing functionality to change) * [x] Documentation update **Infrastructure & Configuration:** * [ ] GitHub Actions workflow * [x] Linting configuration (markdown, PowerShell, etc.) * [ ] Security configuration * [ ] DevContainer configuration * [ ] Dependency update **AI Artifacts:** * [ ] Reviewed contribution with `prompt-builder` agent and addressed all feedback * [x] Copilot instructions (`.github/instructions/*.instructions.md`) * [ ] Copilot prompt (`.github/prompts/*.prompt.md`) * [ ] Copilot agent (`.github/agents/*.agent.md`) * [ ] Copilot skill (`.github/skills/*/SKILL.md`) > Note for AI Artifact Contributors: > > * Agents: Research, indexing/referencing other project (using standard VS Code GitHub Copilot/MCP tools), planning, and general implementation agents likely already exist. Review `.github/agents/` before creating new ones. > * Skills: Must include both bash and PowerShell scripts. See [Skills](../docs/contributing/skills.md). > * Model Versions: Only contributions targeting the **latest Anthropic and OpenAI models** will be accepted. Older model versions (e.g., GPT-3.5, Claude 3) will be rejected. > * See [Agents Not Accepted](../docs/contributing/custom-agents.md#agents-not-accepted) and [Model Version Requirements](../docs/contributing/ai-artifacts-common.md#model-version-requirements). **Other:** * [x] Script/automation (`.ps1`, `.sh`, `.py`) * [ ] Other (please describe): ## Sample Prompts (for AI Artifact Contributions) <!-- Not applicable as a workflow trigger: the instruction file is a passive SSOT consumed by the validator and is referenced by planner agents at session-start. --> For detailed contribution requirements, see: * Common Standards: [docs/contributing/ai-artifacts-common.md](../docs/contributing/ai-artifacts-common.md) - Shared standards for XML blocks, markdown quality, RFC 2119, validation, and testing * Agents: [docs/contributing/custom-agents.md](../docs/contributing/custom-agents.md) - Agent configurations with tools and behavior patterns * Prompts: [docs/contributing/prompts.md](../docs/contributing/prompts.md) - Workflow-specific guidance with template variables * Instructions: [docs/contributing/instructions.md](../docs/contributing/instructions.md) - Technology-specific standards with glob patterns * Skills: [docs/contributing/skills.md](../docs/contributing/skills.md) - Task execution utilities with cross-platform scripts ## Testing Automated validation performed by the agent: * `npm run lint:md` — markdown linting: **Passed** (211 files, 0 errors). * `npm run lint:frontmatter` — frontmatter validation: **Passed** (541 files, 0 errors). * `npm run lint:ps` — PowerShell analysis: **Passed**. * `npm run lint:yaml` — YAML validation: **Passed** (50 files). * `npm run plugin:validate` — plugin metadata: **Passed**. * `npm run plugin:generate` — plugin outputs regenerated and committed. * `npm run test:ps -- -TestPath scripts/tests/linting/Test-DisclaimerArtifacts.Tests.ps1` — **8/8 passed**. * `npm run test:ps -- -TestPath scripts/tests/linting/Validate-PlannerArtifacts.Tests.ps1` — **34/34 passed**. * `grep -r "config/disclaimers" .github docs scripts collections plugins` — **empty**, confirming no orphan references. Security analysis findings: * No secrets, credentials, or customer data in the diff. * No new runtime dependencies introduced. The validator uses only built-in PowerShell cmdlets to parse markdown. * No changes that broaden privilege boundaries. Diff-based assessments: * All changed files match expected paths derived from the commit subject and the migration scope documented in the stack plan. * The markdown SSOT, validator, and tests follow the existing `.github/instructions/`, `scripts/linting/`, and `scripts/tests/linting/` conventions. > [!NOTE] > Manual testing was not performed. ## Checklist ### Required Checks * [x] Documentation is updated (if applicable) — instruction file gains an authoring contract and the SSOT corpus. * [x] Files follow existing naming conventions. * [x] Changes are backwards compatible (if applicable) — *(N/A — this PR is a BREAKING change. `.github/config/disclaimers.yml` is removed; consumers must reference `disclaimer-language.instructions.md`. Tree-wide grep confirms zero in-codebase consumers of the deleted path; all internal callers are updated in this PR.)* * [x] Tests added for new functionality (if applicable). ### AI Artifact Contributions * [ ] Used `/prompt-analyze` to review contribution * [ ] Addressed all feedback from `prompt-builder` review * [ ] Verified contribution follows common standards and type-specific requirements ### Required Automated Checks The following validation commands must pass before merging: * [x] Markdown linting: `npm run lint:md` * [ ] Spell checking: `npm run spell-check` * [x] Frontmatter validation: `npm run lint:frontmatter` * [ ] Skill structure validation: `npm run validate:skills` * [ ] Link validation: `npm run lint:md-links` * [x] PowerShell analysis: `npm run lint:ps` * [x] Plugin freshness: `npm run plugin:generate` * [ ] Docusaurus tests: `npm run docs:test` ## Security Considerations <!--⚠️ WARNING: Do not commit sensitive information such as API keys, passwords, or personal data --> * [x] This PR does not contain any sensitive or NDA information * [x] Any new dependencies have been reviewed for security issues — *(N/A — no new dependencies.)* * [x] Security-related scripts follow the principle of least privilege — *(N/A — no security-related scripts modified.)* ## Additional Notes This PR is the second stacked increment in the post-#1497 stack (PR C in plan ordering). The base will switch from `feat/sssc-planner-rai-parity` to `main` after #1497 merges. The companion PRs in the stack are: * **PR A (#1638)** — Security Planner state schema and fixtures. * **PR B** — Security Planner agent and phase-gate parity (depends on PR A). * **PR D** — Planner linter hardening (sibling off `main`). * **PR E** — Dev environment alignment (sibling off `main`, optional). A small *Disclaimer and Attribution Protocol* section is added to `.github/instructions/security/identity.instructions.md` (sourced verbatim from the snapshot) so the new contract test passes for both planner identities. The SSSC-parity hunks in `Validate-PlannerArtifacts.Tests.ps1` are intentionally deferred to **PR B**, which lands them alongside the Security Planner content they exercise. ## Merge Order This PR is part of a stack derived from PR #1497. Required merge sequence: 1. #1638 (PR A — `stack/security-state-schema`) — adds `disclaimerShownAt` to canonical schemas 2. #1639 (PR C — this PR) — relocates disclaimer protocol to shared instructions 3. #1642 (PR B — `stack/security-planner-parity`) — consumes both #1640 and #1641 are siblings and may merge in any order independent of this chain.
bindsi
left a comment
There was a problem hiding this comment.
Automated batch review: no actionable findings.
# Pull Request ## Description This PR makes two small, surgical edits to **`.github/workflows/copilot-setup-steps.yml`** — the workflow that prebuilds the Copilot Coding Agent environment: 1. **PowerShell module install scope** — change `-Scope CurrentUser` to `-Scope AllUsers` for the three required modules (`PowerShell-Yaml 0.4.7`, `PSScriptAnalyzer 1.25.0`, `Pester 5.7.1`). Versions and the optional `HVE_PSGALLERY_SOURCE_URL` repository registration are unchanged. `AllUsers` writes modules to `/usr/local/share/powershell/Modules`, which is on the default `PSModulePath` for any user and any shell on the runner, so subsequent workflow steps and ad-hoc shells do not need per-user `PSModulePath` fixups to resolve the modules. 2. **Remove an erroneous verification line** — drop `npx --no vally -- --version` from the tool verification block. The intended command does not exist; it was a stray typo (likely from `--no-install` followed by a partial package name) and currently fails silently or noisily depending on the environment. This PR is a **sibling** of #1497, branched off `main`. It is independent of PRs A/B/C/D in the post-#1497 stack and can land in any order relative to them. ### Scope This PR ships **only** the workflow edit. The matching local-environment changes from the original snapshot (`.devcontainer/scripts/on-create.sh` and `.vscode/settings.json`) are deliberately deferred to a separate PR (PR F) pending user confirmation of intent. See follow-on **WI-02** in the planning log. ## Related Issue(s) Sibling of #1497. No other issue references. ## Type of Change Select all that apply: **Code & Documentation:** * [x] Bug fix (non-breaking change fixing an issue) * [ ] New feature (non-breaking change adding functionality) * [ ] Breaking change (fix or feature causing existing functionality to change) * [ ] Documentation update **Infrastructure & Configuration:** * [x] GitHub Actions workflow * [ ] Linting configuration (markdown, PowerShell, etc.) * [ ] Security configuration * [ ] DevContainer configuration * [ ] Dependency update **AI Artifacts:** * [ ] Reviewed contribution with `prompt-builder` agent and addressed all feedback * [ ] Copilot instructions (`.github/instructions/*.instructions.md`) * [ ] Copilot prompt (`.github/prompts/*.prompt.md`) * [ ] Copilot agent (`.github/agents/*.agent.md`) * [ ] Copilot skill (`.github/skills/*/SKILL.md`) **Other:** * [ ] Script/automation (`.ps1`, `.sh`, `.py`) * [ ] Other (please describe): ## Sample Prompts (for AI Artifact Contributions) <!-- Not applicable: this PR is a CI workflow adjustment. --> For detailed contribution requirements, see: * Common Standards: [docs/contributing/ai-artifacts-common.md](../docs/contributing/ai-artifacts-common.md) * Agents: [docs/contributing/custom-agents.md](../docs/contributing/custom-agents.md) * Prompts: [docs/contributing/prompts.md](../docs/contributing/prompts.md) * Instructions: [docs/contributing/instructions.md](../docs/contributing/instructions.md) * Skills: [docs/contributing/skills.md](../docs/contributing/skills.md) ## Testing Automated validation performed by the agent: * `npm run lint:yaml` — **PASS** (50 workflow files, including `copilot-setup-steps.yml`). * `actionlint .github/workflows/copilot-setup-steps.yml` — **PASS** (`EXIT=0`). * `npm run lint:ps` — *skipped* in the sandbox (PSGallery network restricted); the diff contains no `.ps1` changes, so PSScriptAnalyzer would have nothing new to flag. Security analysis findings: * No secrets, credentials, or customer data in the diff. * No new external dependencies. Module names and pinned versions are unchanged; only the install scope changes. * `-Scope AllUsers` requires root on the runner. The `copilot-setup` job already runs as root on `ubuntu-latest`, so no privilege escalation is introduced. * No surface area is added: the typo line `npx --no vally -- --version` is removed, not replaced. Diff-based assessments: * Three line-level scope rewrites plus one line removal. Net change: `+3 / -4`, one file, zero new files, zero structural changes to the workflow graph. > [!NOTE] > Manual testing was not performed. The next live run of `copilot-setup-steps.yml` on a Coding Agent task will exercise the new install scope end-to-end. ## Checklist ### Required Checks * [x] Documentation is updated (if applicable) — *(N/A — workflow-only change.)* * [x] Files follow existing naming conventions. * [x] Changes are backwards compatible. * [x] Tests added for new functionality (if applicable) — *(N/A — no new behavior surface.)* ### AI Artifact Contributions * [ ] Used `/prompt-analyze` to review contribution * [ ] Addressed all feedback from `prompt-builder` review * [ ] Verified contribution follows common standards and type-specific requirements ### Required Automated Checks The following validation commands must pass before merging: * [ ] Markdown linting: `npm run lint:md` * [ ] Spell checking: `npm run spell-check` * [ ] Frontmatter validation: `npm run lint:frontmatter` * [ ] Skill structure validation: `npm run validate:skills` * [ ] Link validation: `npm run lint:md-links` * [ ] PowerShell analysis: `npm run lint:ps` * [ ] Plugin freshness: `npm run plugin:generate` * [ ] Docusaurus tests: `npm run docs:test` ## Security Considerations <!--⚠️ WARNING: Do not commit sensitive information such as API keys, passwords, or personal data --> * [x] This PR does not contain any sensitive or NDA information * [x] Any new dependencies have been reviewed for security issues — *(N/A — no new dependencies.)* * [x] Security-related scripts follow the principle of least privilege — *(N/A — workflow already runs as root on the runner; scope change does not broaden it.)* ## Additional Notes Companion PRs in the post-#1497 work stream: * **PR A (#1638)** — Security Planner state schema and fixtures (stacked on `feat/sssc-planner-rai-parity`). * **PR C (#1639)** — Planner disclaimer SSOT migration (stacked on `feat/sssc-planner-rai-parity`). * **PR D (#1640)** — Planner linter hardening: `applyTo` regression guard (sibling off `main`). * **PR B** — Security Planner agent and phase-gate parity (depends on PR A); will also carry deferred Pester suites and content edits. * **PR F** *(future, optional)* — Devcontainer and `.vscode/settings.json` companion changes, pending user confirmation of intent (WI-02).
## Summary Brings the `security-planner` agent, its supporting instructions, and the shared RAI capture-coaching guidance up to the same phase-gate, state, and question-cadence parity established by the `sssc-planner` work in PR #1497. Bundles the planner-startup, cadence Rule 5 ordering, risk-grid grammar, and state-schema test suites that guard those conventions going forward. ## Stacking - **Base:** `stack/security-state-schema` (PR #1638, "PR A") - **Upstream:** PR #1497 (sssc-planner) - **Siblings:** PR #1639 (PR C — Disclaimer/Attribution Protocol), PR #1640 (PR D) - **Do not merge** until the PR #1497 → PR #1638 cascade completes. ## Changes **Modified (10):** - `.github/agents/security/security-planner.agent.md` - `.github/instructions/security/identity.instructions.md` - `.github/instructions/security/security-model.instructions.md` - `.github/instructions/security/backlog-handoff.instructions.md` - `.github/instructions/rai-planning/rai-capture-coaching.instructions.md` - `.github/prompts/security/security-capture.prompt.md` - `.github/prompts/security/security-plan-from-prd.prompt.md` - `plugins/hve-core-all/README.md` (auto-regen) - `plugins/project-planning/README.md` (auto-regen) - `plugins/security/README.md` (auto-regen) **Added (5 Pester suites):** - `scripts/tests/linting/Test-PlannerStateSchema.Tests.ps1` — 8 tests - `scripts/tests/linting/Test-PlannerStateSchemas.Tests.ps1` — 4 tests - `scripts/tests/linting/Test-CadenceRule5Ordering.Tests.ps1` — 2 tests - `scripts/tests/linting/Test-PlannerStartupBlocks.Tests.ps1` — 6 tests - `scripts/tests/linting/Test-RiskGridGrammar.Tests.ps1` — 7 tests ## Validation | Check | Result | | --- | --- | | `npm run plugin:validate` | ✅ 13 collections | | `npm run lint:md` | ✅ 211 files, 0 errors | | `npm run lint:frontmatter` | ✅ 541 files, 0 errors | | `npm run lint:md-links` | ✅ | | `npm run lint:ps` | ✅ | | `Test-PlannerStateSchema` | ✅ 8/8 | | `Test-PlannerStateSchemas` | ✅ 4/4 | | `Test-CadenceRule5Ordering` | ✅ 2/2 | | `Test-PlannerStartupBlocks` | ✅ 6/6 | | `Test-RiskGridGrammar` | ✅ 7/7 | ## Notes for Reviewers 1. **`disclaimerShownAt: null` in identity state schema** — Adds one line to the inline state schema in `identity.instructions.md` so the canonical state shape declares the field with a default. Full Disclaimer/Attribution Protocol prose that consumes the field remains in PR C (#1639). Three-way overlap with PR #1639 on this file is expected. 2. **`Test-PlannerStateSchemas.Tests.ps1` assertion direction** — Authored draft used `Should -Not -Contain 'disclaimerShownAt'` against the `required` arrays. Per plan decision DD-06/ID-02 (`.copilot-tracking/plans/2026-05-22/stacked-prs-from-pr-1497-plan.instructions.md` line 41), the snapshot demotion of `disclaimerShownAt`/`signingManifestPath` from `required` was **rejected for uniformity** across `security-state`, `rai-state`, and `sssc-state` schemas. Flipped both assertions to `Should -Contain` so the test now affirms the documented uniformity decision. Schemas themselves are unchanged. 3. **Auto-regenerated plugin READMEs** — `plugins/hve-core-all/README.md`, `plugins/project-planning/README.md`, and `plugins/security/README.md` reflect descriptive SSSC text drift produced by `npm run plugin:generate`. Benign; produced by tooling. 4. **Sandbox/environment note** — All Pester suites and pwsh-yaml-dependent lints (`plugin:generate`, `plugin:validate`, `lint:frontmatter`, `lint:md-links`, `lint:ps`) require unsandboxed execution because the `PowerShell-Yaml` module is installed under `~/.local/share/powershell/Modules`. ## Merge Order This PR is part of a 5-PR stack split from PR #1497. Merge in this order: 1. **#1638** — `feat(scripts): security-planner state schema` (base) 2. **#1639** — `feat(prompts): RAI disclaimer/attribution protocol` (independent of B, but lands before B for review continuity) 3. **#1642** (this PR) — `feat(agents): security-planner SSSC parity` (depends on #1638) Independent of the above sequence: - **#1640** — `feat(scripts): risk-grid grammar enforcement` (no shared files) - **#1641** — `chore(plugins): regenerated plugin READMEs` (tooling output only) After #1638 and #1639 merge, rebase this branch onto `main` before merging. --------- Signed-off-by: williamberryiii <wberry@microsoft.com> Co-authored-by: Bill Berry <wbery@microsoft.com>
Pull Request
Description
This PR adds a single Pester regression guard,
scripts/tests/linting/Test-InstructionApplyToScope.Tests.ps1, that enforces phase-narrowedapplyToglobs on every instruction file under.github/instructions/security/**and.github/instructions/rai-planning/**. Any file whose frontmatter declaresapplyTo: '**'is rejected unless its repo-relative path is on the in-script allowlist (currently empty, matching the Phase 6 audit-matrix migration map).The suite is data-driven (
It -ForEach) so each scanned file produces a discrete test case (currently 18 cases, all passing). It usespowershell-yaml(already required byscripts/linting/Validate-MarkdownFrontmatter.ps1and other existing suites — no new repo-level dependency) and explicitly throws on missing frontmatter so silent skips cannot mask regressions.This PR is a sibling of #1497, branched off
main. It is independent of PRs A/C/B in the post-#1497 stack and can land in any order relative to them.Scope reduction
The originating plan listed four Pester suites under "planner linter hardening." Three are deferred to PR B because they assert content not yet on
main:security/identity.instructions.mdandsssc-identity.instructions.md.## StartupH2 in six prompts under.github/prompts/security/and.github/prompts/security/sssc/.security-model.instructions.md.Shipping those tests now would produce a red base. They will land in PR B alongside the content edits they exercise.
Related Issue(s)
Sibling of #1497. No other issue references.
Type of Change
Code & Documentation:
Infrastructure & Configuration:
Other:
.ps1,.sh,.py)Testing
Automated validation performed by the agent:
npm run test:ps -- -TestPath scripts/tests/linting/Test-InstructionApplyToScope.Tests.ps1— 18/18 passed.npm run lint:ps— clean for repository code.Security analysis findings:
powershell-yamlis already required by existing linting and test suites.Diff-based assessments:
scripts/tests/linting/Test-*.Tests.ps1naming and structure.Note
Manual testing was not performed.
Checklist
Required Checks
Required Automated Checks
npm run lint:psSecurity Considerations
Additional Notes
Companion PRs in the post-#1497 work stream:
main, optional).