[prompt-clustering] Copilot Agent Prompt Clustering Analysis — 2026-08-01

[github-actions[bot]]

Summary

Analysis period: 2026-07-11 to 2026-08-01 (trailing ~1000 copilot-authored PRs)
Total PRs analyzed: 1,000 (747 merged, 240 closed, 13 open)
Overall merge success rate: 75.7%
Clusters identified: 4 (k selected by silhouette score, range tested 4–10)

Success rate has been declining for a month — down from an 82.4% peak on 2026-07-01 to 75.7% today, a steady ~6.7-point erosion with no single-day reversal since mid-July.

Full Analysis Report

General Insights

• Most common task type: conventional "add / implement" work — cluster 3 (added / workflow / changes / new), 429 PRs (43% of total), 75.8% merged.
• Highest success-rate cluster: cluster 1 at 85.4% merged — but see the data-quality note below; its "top terms" are a boilerplate phrase, not a real theme.
• Lowest success-rate cluster: cluster 0 ("actions / fix / job") — repeated [WIP] Fix failing GitHub Actions job ... PRs, only 63.2% merged (12/19), and cluster 2 ("start / coding / copilot") at 67.4% (229/340), the largest low-success group.
• Trend: overall success rate peaked at 82.4% on 2026-07-01 and has fallen almost every week since, now at its lowest point in the 30-day window (75.7%, essentially tied with the 2026-07-30 low of 75.4%).

Cluster Analysis

Cluster 1: "added / workflow / changes / new" — general feature & fix work

• Size: 429 tasks (42.9% of total)
• Success rate: 75.8%
• Dominant type: feat (119), fix (96), other (70)
• Characteristics: broad catch-all for conventional feature/fix PRs without a narrower shared theme — CLI version bumps, schema extensions, tool-cache fallbacks, stacked-PR gating.
• Example PRs: Add llms.txt to repo root for AI tool discoverability #49560 (merged), chore: CLI version updates — Copilot 1.0.77, Pi 0.83.0, Playwright Browser v1.62.1, Syft v1.50.0, Grype v0.116.1 #49521 (merged), Prefer compat toolcache fallback for explicit Copilot CLI versions #49478 (closed), Extend stacked PR max-stack gating to pull_request_review #49453 (merged), Make github/gh-aw installable as the agentic-workflows skill #49354 (closed)

---

Cluster 2: "start / coding / copilot" — engine/schema/refactor fixes, lowest-volume success

• Size: 340 tasks (34.0% of total)
• Success rate: 67.4% — well below the 75.7% overall average
• Dominant type: fix (121), other (60), feat (41)
• Characteristics: the recurring "START COPILOT CODING AGENT" prompt boilerplate dominates this cluster's vocabulary, so it reads more as "everything else that's a fix/refactor" than a single theme — schema fixes, log-path corrections, CODEOWNERS routing, call-site migrations. Its below-average success rate is the most actionable finding here: nearly a third of these PRs are closed without merging.
• Example PRs: fix(schema): add antigravity to engine.id descriptions and examples #49558 (closed), fix: correct Squid access.log path and add runtime presence detection #49557 (closed), chore: add .github/CODEOWNERS with path-based review routing for security-sensitive paths #49556 (closed), refactor: migrate remaining RunGHWithHost call sites to RunGHContextWithHost #49554 (closed), Normalize eval outputs to binary YES/NO #49553 (closed)

---

Cluster 3: "pr / chef / sous" — see data-quality note

• Size: 212 tasks (21.2% of total)
• Success rate: 85.4% (highest of the four)
• Dominant type: fix (87), feat (55), chore (24)
• Characteristics: top TF-IDF terms are "pr sous chef" / "sous chef" — this is a recurring bot/tooling signature phrase in PR bodies (also seen in every prior clustering run back to June), not a task theme. The underlying PRs are actually diverse: threat-detection error handling, engine timeout forwarding, permission scoping. Its high success rate is likely a side effect of PR maturity/review routing tied to whatever surfaces that signature, not the subject matter — worth investigating what "sous chef" refers to in the body template if a real signal is wanted here.
• Example PRs: fix: use distinct XML marker for threat detection engine errors vs real threats #49527 (closed), improve: threat detection engine error template with progressive disclosure #49497 (closed), eslint-factory: flag core.error + core.setFailed(prefix + msg) as redundant #49496 (merged), engine: forward timeout-minutes to execution step for codex, gemini, pi, antigravity #49495 (closed), Allow add-labels to opt out of pull-requests: write #49477 (closed)

---

Cluster 4: "actions / fix / job" — CI failure firefighting

• Size: 19 tasks (1.9% of total)
• Success rate: 63.2% (lowest)
• Dominant type: fix (19 of 19)
• Characteristics: small, sharply-defined cluster of [WIP] Fix failing GitHub Actions job ... PRs — one per failing CI job/test shard. Nearly 40% never merge, consistent with reactive CI-firefighting tasks that often get superseded by a follow-up fix before landing.
• Example PRs: [WIP] Fix failing GitHub Actions job 'Unit tests (S-Z and other)' #49134 (closed), [WIP] Fix failing GitHub Actions job for unit tests (D-L) #49133 (closed), [WIP] Fix failing GitHub Actions job Unit tests (A-C) #49132 (merged), [WIP] Fix failing GitHub Actions job Integration: CMD Tests #49110 (merged), [WIP] Fix failing GitHub Actions job lint-go #48975 (merged)

Breakdown by Conventional-Commit Type

Type	PRs	Merged	Closed	Success Rate
fix	323	252	63	78.0%
feat	215	156	57	72.6%
other	146	114	32	78.1%
chore	111	81	29	73.0%
refactor	100	62	36	62.0%
docs	67	52	15	77.6%
test	29	23	6	79.3%
perf	4	3	1	75.0%
ci	2	2	0	100%

refactor is the weakest category by type (62.0%, 36/100 closed) — notably lower than every other category with meaningful volume.

Breakdown by Risk / Priority Label

Risk	PRs	Success Rate
low	183	76.5%
medium	168	80.4%
high	45	73.3%

Priority	PRs	Success Rate
low	123	70.7%
medium	175	78.3%
high	96	85.4%
critical	2	100%

Only 439/1000 PRs carry pr-risk/pr-priority/pr-type labels — the rest have no labels, so these breakdowns cover ~44% of the dataset. High-priority work merges noticeably more often (85.4%) than low-priority work (70.7%), suggesting prioritization labels correlate with better follow-through, or that low-priority items are more readily abandoned/closed rather than merged.

Key Findings

1. Success rate is on a sustained downward trend. From an 82.4% peak on 2026-07-01, the rate has dropped almost monotonically to 75.7% today — a 6.7-point decline over one month, now at the low end of the observed range.
2. Refactor PRs underperform. At 62.0% success, refactors close without merging far more often than fixes (78.0%) or features (72.6%) — worth checking whether refactor PRs are getting stuck in review longer or superseded by newer refactors.
3. CI-firefighting PRs ([WIP] Fix failing GitHub Actions job ...) rarely all land. Only 63.2% merge; these look like transient, often-superseded single-purpose patches.
4. A recurring "PR Sous Chef" signature phrase distorts TF-IDF clustering. It has appeared as a top-term artifact in every clustering run since early June (see cache history) — a real thematic clustering would benefit from stripping this boilerplate before vectorizing.
5. Labeling coverage is low (44%). Risk/priority/type labels, when present, correlate with outcome (e.g., high-priority PRs merge at 85.4% vs 70.7% for low-priority), but most PRs aren't labeled, limiting how much can be concluded from this slice.

Recommendations

1. Investigate the July success-rate decline. Pull a sample of recently-closed (non-merged) PRs from the last two weeks and check for a common root cause — reviewer bandwidth, a stricter gate, flaky CI, or a change in task complexity.
2. Audit refactor-type prompts. Given the lowest success rate (62.0%) of any major category, review a handful of closed refactor PRs to see whether the task framing (scope too broad, conflicting with parallel refactors) is the issue.
3. Strip recurring boilerplate before TF-IDF vectorization. The "PR Sous Chef" signature and the "START COPILOT CODING AGENT" prompt header both dominate cluster vocabulary without carrying task signal — removing them should produce more interpretable clusters in future runs.
4. Increase label coverage. Only 44% of PRs carry risk/priority/type labels; wider labeling would make the risk/priority breakdown representative of the full dataset rather than a self-selected subset.

---

Methodology note: No workflow-run logs (aw_info.json) were available this run, so turn counts / duration / cost could not be joined to individual PRs; this report is based on PR metadata and body-text clustering only. TF-IDF (300 features, 1-3 grams) + K-means (k selected via silhouette score, tested k=4–10) on cleaned PR body text; conventional-commit type inferred from pr-type:* labels where present, else title prefix/verb heuristics.

Generated by Prompt Clustering Analysis (Run: 30695398045)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · agent · 142.4 AIC · ⌖ 25.8 AIC · ⊞ 13.4K · ◷

• expires on Aug 2, 2026, 2:42 AM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-02T10:42:05.107Z.

Closed by Workflow