Improve clarity in mobile scanner workflow issues and PRs

[kotlarmilos]

Description

The mobile platform failure scanner has been failing with permission-denied errors and produced a mix of PRs, issues, and repeated comments on the same issues across runs. This change makes the workflow simpler and more predictable.

Now every mobile failure either becomes a draft PR (when there's a per-test fix like [SkipOnPlatform]) or a tracking issue (for product bugs, native crashes, multi-assembly regressions, or infra problems that need an owner). The workflow no longer files comments and no longer ends a run with noop.

Other changes:

• Cap raised to 5 PRs and 3 issues per run.
• PRs now only touch test files (src/libraries/**/tests/** and matching .csproj); anything else fails PR creation.
• PR and issue bodies use a fixed structure: Reasoning, Impact on platforms, Errors log, First build it occurred (issues add a Recommended action section).
• Removed broken references in the prompt (pwsh Get-CIStatus.ps1, gh search prs) and dropped pwsh from the allowlist.

Validated with one workflow_dispatch run on this branch where it produced three issues #127563, #127564, #127565.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

Updates the Mobile Platform Failure Scanner agentic workflow prompt and safe-output configuration to make outcomes more predictable: each actionable mobile failure should result in either a draft PR (per-test attribute fix) or a tracking issue (everything else), with clearer, standardized body templates and stricter PR guardrails.

Changes:

• Simplifies and restructures the workflow prompt to remove comment/noop-centric behavior and standardize PR/issue body sections.
• Increases per-run caps (up to 5 PRs / 3 issues) and changes protected-file handling to “blocked”.
• Adds an allowed-files policy for PR creation and removes pwsh from the shell allowlist / prompt references (with corresponding lock-file updates).

Show a summary per file

File	Description
.github/workflows/mobile-scan.md	Updates workflow description, safe-outputs constraints, and rewrites the agent prompt to enforce PR-vs-issue outcomes and standard body structure.
.github/workflows/mobile-scan.lock.yml	Regenerates the compiled workflow to reflect the updated prompt, safe-outputs toolset, permissions, and allowlists.

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 2

[github-actions]

🤖 Copilot Code Review — PR #127568

Note

This review was generated by GitHub Copilot.

Holistic Assessment

Motivation: The workflow's previous prompt was a verbose, procedural 8-step shell walkthrough that hand-held the agent through every curl/jq/bash call. This micromanagement was fragile — it led to missing_tool failures when shell guards blocked $(...) or > redirection, and limited the agent to only 2 PRs and 2 issues per run. Shifting to an outcome-driven prompt that states what the agent should produce (draft PRs or tracking issues) rather than how to run each shell command is a sound improvement.

Approach: The rewrite correctly distills the previous 150+ lines of step-by-step instructions into ~60 lines of outcome specification, data-source pointers, and hard constraints. The frontmatter tightening (protected-files: blocked, explicit allowed-files, removal of add-comment, removal of pwsh) reduces the agent's blast radius. The lock file changes are consistent auto-generated reflections of the .md changes.

Summary: ✅ LGTM. The changes are well-scoped, internally consistent, and the lock file correctly mirrors the .md frontmatter. No blocking issues found.

---

Detailed Findings

✅ Safe-outputs policy tightening — Correct and safer

The switch from protected-files: fallback-to-issue to protected-files: blocked with explicit allowed-files globs (src/libraries/**/tests/**, src/libraries/**/*.csproj, src/libraries/Common/tests/**) is a clear improvement. The old policy silently degraded to filing an issue when the agent touched protected files; the new policy hard-blocks it, forcing the agent to only produce PRs that touch test code. The lock file's GH_AW_SAFE_OUTPUTS_HANDLER_CONFIG JSON correctly includes the allowed_files array and "protected_files_policy":"blocked".

✅ Capacity increase — Reasonable given outcome-driven design

Increasing create-pull-request.max from 2→5 and create-issue.max from 2→3 makes sense given the prompt now encourages grouping failures with shared fixes into single PRs and same-root-cause failures into single issues. The old capacity was artificially low for a pipeline that can have many distinct mobile failures in a single build.

✅ Removal of add-comment and untriaged label — Clean simplification

Removing add-comment (with target: "*") eliminates the ability to post comments on arbitrary issues/PRs, which was an overly broad permission. Removing untriaged from issue labels is sensible — the agent shouldn't label its issues as untriaged since they already contain a diagnosis.

✅ Removal of pwsh from bash allowlist — Consistent with new prompt

The old bash allowlist included pwsh, and the old prompt referenced PowerShell scripts (Get-CIStatus.ps1). Since the new prompt removes all PowerShell references and the "Hard environment constraints" section explicitly states "No gh, no pwsh, no python", removing pwsh from the allowlist is correct. The lock file confirms pwsh was removed from the --allow-tool list.

✅ Removal of discussions: write permission — Correct least-privilege

The workflow doesn't create discussions, so this permission was unnecessary. Removed from both the safe-outputs and post-process jobs in the lock file.

✅ Prompt body rewrite — Outcome-driven with no loss of critical information

The new prompt preserves all essential information: AzDO definition ID (154), Helix REST patterns, mobile job name regex, dead-letter detection, PR body structure requirements, and the hard environment constraints that previously caused agent failures. The "Data sources" section consolidates what was scattered across Steps 2–5. The explicit "first seen in scanned window" lookback requirement (20 builds) and per-signature log drill-down are retained.

💡 Minor: allowed-files glob overlap

src/libraries/Common/tests/** is already covered by src/libraries/**/tests/**. The explicit entry is harmless (likely added for clarity/emphasis), but it is redundant.

💡 Minor: Data-safety reminder restructured

The old prompt had an explicit "Data safety: CI logs can contain user paths, environment variables with secrets..." callout. The new prompt instead says "Strip JWTs, bearer tokens, ApplicationGatewayAffinity*=, and per-user paths" in the "Errors log" section, which is more specific and actionable — arguably better. Just noting the change in approach.

Generated by Code Review for issue #127568 · ◷

[Copilot]

Pull request overview

Updates the Mobile Platform Failure Scanner agentic workflow prompt and its compiled lock file to reduce noisy/duplicated outputs and make outcomes more predictable (each actionable failure results in either a draft PR or a tracking issue), while tightening integrity and Safe Outputs constraints.

Changes:

• Tightens GitHub MCP integrity filtering (min-integrity: approved) and removes pwsh from the shell allowlist.
• Adjusts Safe Outputs policy: up to 5 draft PRs and 3 issues per run; PRs restricted to library test files only; removes comment filing.
• Rewrites the workflow prompt into a shorter, more structured set of outcome rules and fixed PR/issue body templates.

Show a summary per file

File	Description
.github/workflows/mobile-scan.md	Updates workflow description, tool configuration, Safe Outputs limits/policies, and rewrites the prompt guidance for PR vs issue outcomes.
.github/workflows/mobile-scan.lock.yml	Regenerated compiled workflow to reflect new integrity policy, Safe Outputs config, and removal of comment/discussion-related handling.

Copilot's findings

• Files reviewed: 2/2 changed files
• Comments generated: 4