📊 Workflow Optimization Report — 60% Flake Rate, 76% Wasted Triggers, Security Concerns

[github-actions]

📊 Workflow Optimization Report

Auto-generated analysis of the last 100 workflow runs in austenstone/copilot-cli
Period covered: Oct 2025 – Apr 2026

---

Summary Table

Workflow	Total Runs	✅ Success	❌ Failure	⏭️ Skipped/Other	Success Rate	Avg Duration	Flake Rate
Fake CI (ci.yml)	9	0	9	0	0% (intentional)	~9s	N/A — designed to fail
test-copilot (.github/workflows/test-copilot.yml)	15	6	9	0	40%	~20s (success) / ~4s (fail)	60%
copilot-ci-fix (.github/workflows/copilot-ci-fix.yml)	37	8	0	28 skipped + 1 cancelled	22% (of non-skipped)	~16s	Low
copilot-dependabot-update	8	1	0	7 skipped	12.5%	~4s	Low
copilot-labeler	1	1	0	0	100%	~21s	Low
Copilot coding agent (dynamic)	1	1	0	0	100%	~179s	Low
Dependabot Updates (dynamic)	10	10	0	0	100%	~36s	Low

Overall fleet health: 62/100 concluded runs succeeded (62%). Queue times: ~0s avg (healthy runner capacity).

---

🔴 Finding 1: test-copilot — 60% Flake Rate (135 Wasted Job-Runs)

Severity: HIGH

The test-copilot.yml workflow has a 60% failure rate over the last 100 runs. All 15 parallel jobs fail or succeed together, indicating a systemic environment issue rather than individual test logic bugs.

Root Cause — Branch Dependency:
The workflow uses austenstone/copilot-cli@notoken as the action ref. All successes occurred on the notoken branch, while all recent failures occurred on main, v3, v3.0, and dependabot branches. When these branches are pushed, the action resolves correctly from @notoken, but the token/permission context differs:

Date	Branch	Conclusion
2026-03-23	main	❌ failure
2026-03-23	v3	❌ failure
2026-03-11	main	❌ failure (×3)
2026-03-11	v3.0	❌ failure
2026-03-11	notoken	✅ success (×6)
2026-01-29	main	✅ success

The single notoken-branch failure on 2026-03-11T16:52 and then 3 consecutive successes suggest a transient Copilot API availability issue also exists.

📋 Raw: Failed run job breakdown (Run #23451003397, branch: v3, 2026-03-23)

All 15 jobs failed within 2–5 seconds of starting — consistent with an authentication/bootstrap failure before any Copilot API call is made:

github-mcp:      failure  (started 17:29:20 → completed 17:29:24, 4s)
basic:           failure  (started 17:29:21 → completed 17:29:24, 3s)
silent:          failure  (started 17:29:20 → completed 17:29:22, 2s)
autopilot:       failure  (started 17:29:20 → completed 17:29:22, 2s)
denied-tools:    failure  (started 17:29:20 → completed 17:29:22, 2s)
additional-dirs: failure  (started 17:29:20 → completed 17:29:22, 2s)
json-output:     failure  (started 17:29:20 → completed 17:29:22, 2s)
model:           failure  (started 17:29:19 → completed 17:29:21, 2s)
agent:           failure  (started 17:29:20 → completed 17:29:22, 2s)
url-controls:    failure  (started 17:29:20 → completed 17:29:23, 3s)
config:          failure  (started 17:29:20 → completed 17:29:23, 3s)
share:           failure  (started 17:29:20 → completed 17:29:22, 2s)
experimental:    failure  (started 17:29:20 → completed 17:29:22, 2s)
no-builtin-mcps: failure  (started 17:29:20 → completed 17:29:22, 2s)
fail-on-error:   failure  (started 17:29:20 → completed 17:29:22, 2s)

vs. successful run #22964512980 (branch: notoken): all 15 jobs complete in 14–33 seconds.

📋 Raw: Successful vs failed run comparison

Success (Run #22964512980, branch: notoken, 2026-03-11)

share:           success  17:00:17 → 17:00:39  (22s)
basic:           success  17:00:18 → 17:00:44  (26s)
github-mcp:      success  17:00:17 → 17:00:40  (23s)
silent:          success  17:00:17 → 17:00:36  (19s)
fail-on-error:   success  17:00:18 → 17:00:50  (32s)
... (all 15 jobs pass)

Failure (Run #23451003397, branch: v3, 2026-03-23)

model:           failure  17:29:19 → 17:29:21  (2s — never reaches API)
basic:           failure  17:29:21 → 17:29:24  (3s — never reaches API)
... (all 15 jobs fail instantly)

---

🟡 Finding 2: copilot-ci-fix — 75.7% Wasted Trigger Rate

Severity: MEDIUM

copilot-ci-fix.yml listens to all workflow completions (workflows: ['*']), then uses an if: condition to skip non-failures. This causes 28 out of 37 triggered runs (75.7%) to be skipped — consuming runner startup overhead for no purpose.

# Current — inefficient:
on:
  workflow_run:
    workflows: ['*']   # Fires on EVERY workflow completion
    types: [completed]

jobs:
  ci-failure:
    if: ${{ github.event.workflow_run.conclusion == 'failure' }}  # Then skips 76% of the time

Additionally:

• The concurrency group uses github.event.pull_request.number which is not populated in workflow_run events, likely always resolving to an empty string and causing unintended cancellations (evidenced by 1 cancelled run).
• permissions: write-all is overly broad; minimum required is contents: write + pull-requests: write + issues: write.

📋 Raw: copilot-ci-fix run history (last 20 runs)

skipped         2026-03-28  workflow_run  main
success         2026-03-24  workflow_run  main   (16:13 → 16:19, 6min)
skipped         2026-03-23  workflow_run  main
skipped         2026-03-23  workflow_run  main
skipped         2026-03-21  workflow_run  main
skipped         2026-03-14  workflow_run  main
action_required 2026-03-13  workflow_run  main
skipped         2026-03-13  workflow_run  main
skipped         2026-03-12  workflow_run  main
skipped         2026-03-11  workflow_run  main   (x7 — burst from Fake CI/test runs)
success         2026-03-11  workflow_run  main   (x4)
cancelled       2026-03-11  workflow_run  main   (concurrency conflict)
skipped         2026-03-10 through 2026-02-17  (daily/near-daily triggers, all skipped)

---

🟡 Finding 3: Fake CI — Permanent 100% Failure (By Design, But Triggers Expensive Downstream Runs)

Severity: MEDIUM

ci.yml always fails (exit 1) — this is intentional, serving as a test harness for copilot-ci-fix. However, each Fake CI run triggers copilot-ci-fix to execute the Copilot coding agent (~6 minutes of AI-assisted PR creation). With 9 runs in the window, this generated 9 Copilot coding agent invocations.

# ci.yml — Fake CI (intentional fail)
steps:
  - run: echo "Hello, world!"
  - run: exit 1   # always fails

Consider rate-limiting or scoping this to specific branches (e.g., main only) to reduce noise and unwanted Copilot usage spend.

---

🟢 Finding 4: Duration Trends (Stable, No Regression)

Severity: LOW / Informational

Build durations are stable with no upward trend:

Workflow	Avg Duration	Min	Max
test-copilot (success)	~20–33s	~14s	~33s
copilot-ci-fix (success)	~16s	~13s	~21s
Copilot coding agent	~179s	~64s	~367s
Dependabot Updates	~36s	~30s	~41s

Queue times are effectively 0 seconds across all 100 runs — no runner capacity issues.

One outlier: github_actions in /. - Update #1270024114 took 367 seconds (6× the average). This may have involved a complex dependency resolution.

---

🟢 Finding 5: Missing Caching — Checkout Repeated Across 15 Parallel Jobs

Severity: LOW

test-copilot.yml runs actions/checkout@v5 in each of the 15 parallel jobs independently. Since all jobs run on separate runners, this is unavoidable per-job, but the checkout step itself (shallow clone) is already fast (~1–2s). No caching opportunity exists here without restructuring to a matrix strategy with artifact passing.

However, actions/checkout@v5 uses a floating semver tag. Pinning to a full SHA (e.g., actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683) improves supply-chain security and prevents silent breakage on minor/patch bumps.

---

High Impact Fixes

• [P0] Fix test-copilot.yml branch-based failures — The workflow's push trigger fires on all branches, but the action ref @notoken only works correctly in certain contexts. Add a branch filter or fix the action to work on all branches:

  on:
    push:
      branches: [notoken]  # Limit to the branch that works, OR
    workflow_dispatch:

  Or investigate why @notoken fails on main/v3 pushes — likely a missing copilot-requests: write permission on non-default branches.

• [P1] Scope copilot-ci-fix.yml trigger to specific failing workflows — Replace wildcard with named workflows to eliminate 76% wasted skipped runs:

  on:
    workflow_run:
      workflows: ["Fake CI"]  # Only watch the workflows you want to fix
      types: [completed]

• [P1] Fix copilot-ci-fix.yml concurrency group — github.event.pull_request.number is always empty in workflow_run events. Use github.event.workflow_run.id instead:

  concurrency:
    group: ${{ github.workflow }}-${{ github.event.workflow_run.id }}
    cancel-in-progress: false  # Don't cancel in-flight repair jobs

• [P2] Reduce copilot-ci-fix.yml permissions — Replace permissions: write-all with minimum required:

  permissions:
    contents: write
    pull-requests: write
    issues: write
    actions: read

• [P2] Rate-limit Fake CI trigger — Add a branch filter to ci.yml so it only runs on main/PRs, preventing rapid-fire Copilot coding agent invocations from development branch pushes:

  on:
    pull_request:
      branches: [main]
    workflow_dispatch:

• [P3] Pin action versions to full SHAs — Replace floating tags (@v5, @v6) with pinned SHAs in all workflow files for supply-chain security compliance.

• [P3] Investigate the one transient notoken-branch failure (Run 22964185450, 2026-03-11T16:52) — Review logs to determine if it was a Copilot API rate limit or transient service error. If so, add retry logic.

---

📊 Full Raw Run Data (100 runs)

Run ID	Workflow	Branch	Event	Conclusion	Duration
23695445812	copilot-ci-fix	main	workflow_run	skipped	1s
23499839366	copilot-ci-fix	main	workflow_run	success	378s
23499819595	Fake CI	main	workflow_dispatch	failure	24s
23451040373	copilot-ci-fix	main	workflow_run	skipped	2s
23451003397	test-copilot	v3	push	failure	7s
23450998930	test-copilot	main	push	failure	7s
23450925715	test-copilot	main	push	failure	8s
23389911221	copilot-ci-fix	main	workflow_run	skipped	1s
23097427242	copilot-ci-fix	main	workflow_run	skipped	1s
23066323598	copilot-ci-fix	main	workflow_run	action_required	0s
23062236793	copilot-ci-fix	main	workflow_run	skipped	2s
23014572612	copilot-ci-fix	main	workflow_run	skipped	3s
22965309225	copilot-ci-fix	main	workflow_run	skipped	1s
22965269159	copilot-ci-fix	main	workflow_run	skipped	2s
22965263008	test-copilot	main	push	failure	7s
22965225905	test-copilot	main	push	failure	9s
22965179472	copilot-ci-fix	main	workflow_run	skipped	1s
22964659688	copilot-ci-fix	main	workflow_run	skipped	1s
22964648739	test-copilot	v3.0	push	failure	15s
22964643465	copilot-ci-fix	main	workflow_run	success	20s
22964636958	copilot-ci-fix	main	workflow_run	skipped	1s
22964635922	Fake CI	dependabot/upload-artifact-7	pull_request	failure	7s
22964635120	copilot-ci-fix	main	workflow_run	skipped	1s
22964634774	test-copilot	dependabot/upload-artifact-7	push	failure	7s
22964592681	test-copilot	main	push	failure	8s
22964576340	copilot-ci-fix	main	workflow_run	skipped	2s
22964546200	copilot-ci-fix	main	workflow_run	skipped	1s
22964521763	copilot-ci-fix	main	workflow_run	success	25s
22964515014	Fake CI	notoken	pull_request	failure	7s
22964512980	test-copilot	notoken	push	success	37s
22964503029	copilot-ci-fix	main	workflow_run	success	20s
22964493561	Fake CI	notoken	pull_request	failure	8s
22964493401	test-copilot	notoken	push	success	31s
22964367873	copilot-ci-fix	main	workflow_run	skipped	1s
22964328648	copilot-ci-fix	main	workflow_run	success	17s
22964319741	Fake CI	notoken	pull_request	failure	9s
22964318246	test-copilot	notoken	push	success	25s
22964232869	copilot-ci-fix	main	workflow_run	skipped	1s
22964194684	copilot-ci-fix	main	workflow_run	success	20s
22964186287	Fake CI	notoken	pull_request	failure	9s
22964185450	test-copilot	notoken	push	failure	31s
22964122223	copilot-ci-fix	main	workflow_run	skipped	2s
22964086333	copilot-ci-fix	main	workflow_run	success	21s
22964078621	Fake CI	notoken	pull_request	failure	6s
22964076866	test-copilot	notoken	push	success	40s
22963387451	copilot-ci-fix	main	workflow_run	skipped	1s
22963374495	copilot-ci-fix	main	workflow_run	skipped	1s
22963348818	copilot-ci-fix	main	workflow_run	success	21s
22963341625	Fake CI	notoken	pull_request	failure	8s
22963340649	test-copilot	notoken	push	success	33s
22963336966	copilot-ci-fix	main	workflow_run	cancelled	21s
22963331035	Fake CI	notoken	pull_request	failure	7s
22963301138	test-copilot	notoken	push	success	26s
22915037806	copilot-ci-fix	main	workflow_run	skipped	2s
22869743171	copilot-ci-fix	main	workflow_run	skipped	1s
(45 more Dependabot/copilot-dependabot/copilot-labeler/copilot-usage-report runs omitted)

Full data available via: gh api "repos/austenstone/copilot-cli/actions/runs?per_page=100"

---

Report generated: 2026-04-01 by GitHub Copilot CLI Workflow Analyzer
Methodology: Last 100 runs fetched via GitHub Actions REST API; durations calculated from created_at to updated_at; flake rate = (failure runs / total concluded runs) per workflow.