Optimize Documentation Maintainer token usage and prompt/cache behavior by Copilot · Pull Request #3202 · github/gh-aw-firewall

Copilot · 2026-05-15T13:26:02Z

✨ Enhancement

The Documentation Maintainer workflow was consuming ~1.17M tokens/run with 28 LLM requests, driven largely by repeated bash round-trips (git show per commit) and unstable prompt caching. This change restructures the workflow to reduce request fan-out, shift work into precomputed step outputs, and run on a smaller model.

Model sizing
- Switched workflow engine config to Copilot + claude-haiku-4-5 for lower-cost doc maintenance tasks.
Precomputed change context (fewer tool round-trips)
- Replaced commit-list-only step with a single-pass diff aggregation step:
  - git-changes now emits RECENT_DIFFS with bounded unified diffs for src/, containers/, scripts/ (+ relevant doc paths).
- Added affected-docs step to emit prioritized candidate files (AFFECTED_DOCS) from recent source-touching paths.
- Added has-changes step to emit has_changes/changed_count for explicit no-op handling.
Prompt structure + execution guidance (cache/write and scope control)
- Reordered prompt so stable instructions remain first and dynamic run context is appended at the end.
- Updated agent instructions to:
  - exit immediately with no-op when has_changes=false
  - use precomputed RECENT_DIFFS as primary input
  - prioritize AFFECTED_DOCS before broad doc review
  - avoid per-commit git show unless strictly necessary
Related failure context
- Recent run failure root cause was in safe-outputs PR creation (CLAUDE.md protected-file modification), independent of this token-optimization flow.

steps:
  - name: Check for relevant changes
    id: has-changes
    run: |
      COUNT=$(git log --since="7 days ago" --oneline -- src/ containers/ scripts/ | wc -l)
      echo "changed_count=$COUNT" >> $GITHUB_OUTPUT
      echo "has_changes=$([ "$COUNT" -gt 0 ] && echo true || echo false)" >> $GITHUB_OUTPUT

  - name: Gather recent git diffs
    id: git-changes
    run: |
      echo "RECENT_DIFFS<<EOF" >> $GITHUB_OUTPUT
      git log --since="7 days ago" --format="%H %s" -- src/ containers/ scripts/ | \
        while read -r sha title; do
          echo "=== Commit $sha: $title ==="
          git show --stat --unified=3 "$sha" -- src/ containers/ scripts/ docs/ '*.md' 2>/dev/null | head -150
        done | head -500
      echo "EOF" >> $GITHUB_OUTPUT

Copilot

Pull request overview

This PR updates the Documentation Maintainer agent workflow to reduce token usage by switching to a smaller Copilot model and attempting to precompute change context for the agent prompt.

Changes:

Sets the Documentation Maintainer engine to Copilot with claude-haiku-4-5.
Adds pre-agent steps for source-change detection, recent diffs, and affected-doc candidates.
Reorders prompt context so run-specific data is appended near the end.

Show a summary per file

File	Description
`.github/workflows/doc-maintainer.md`	Updates workflow source configuration, precomputed git context steps, and prompt instructions/context.
`.github/workflows/doc-maintainer.lock.yml`	Regenerated compiled workflow reflecting model, prompt substitutions, and new pre-agent steps.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comments suppressed due to low confidence (1)

.github/workflows/doc-maintainer.md:65

The AFFECTED_DOCS body is also printed to stdout instead of $GITHUB_OUTPUT, so the output will be empty even when relevant files exist. The multiline output needs to append the command output between the heredoc markers.

      echo "AFFECTED_DOCS<<EOF" >> $GITHUB_OUTPUT
      git log --since="7 days ago" --name-only --format="" -- src/ containers/ scripts/ | \
        grep -v '^$' | sort -u | head -30

Files reviewed: 2/2 changed files
Comments generated: 7

+      git log --since="7 days ago" --format="%H %s" -- src/ containers/ scripts/ | \
+        while read -r sha title; do
+          echo "=== Commit $sha: $title ==="
+          git show --stat --unified=3 "$sha" -- src/ containers/ scripts/ docs/ '*.md' 2>/dev/null | head -150
+        done | head -500


+          GH_AW_EXPR_4072EBFD: ${{ steps.has-changes.outputs.has_changes }}
+          GH_AW_EXPR_8CF6695E: ${{ steps.has-changes.outputs.changed_count }}
+          GH_AW_EXPR_8F5D95B0: ${{ steps.git-changes.outputs.RECENT_DIFFS }}
+          GH_AW_EXPR_FAA4E739: ${{ steps.affected-docs.outputs.AFFECTED_DOCS }}


+  - name: Check for relevant changes
+    id: has-changes
+    run: |
+      COUNT=$(git log --since="7 days ago" --oneline -- src/ containers/ scripts/ | wc -l)


+      git log --since="7 days ago" --name-only --format="" -- src/ containers/ scripts/ | \
+        grep -v '^$' | sort -u | head -30


+      echo "RECENT_DIFFS<<EOF" >> $GITHUB_OUTPUT
+      git log --since="7 days ago" --format="%H %s" -- src/ containers/ scripts/ | \
+        while read -r sha title; do
+          echo "=== Commit $sha: $title ==="
+          git show --stat --unified=3 "$sha" -- src/ containers/ scripts/ docs/ '*.md' 2>/dev/null | head -150
+        done | head -500
      echo "EOF" >> $GITHUB_OUTPUT


 ### 3. Review Current Documentation

-Read the documentation files listed above that are likely affected by the recent changes.
+Start with the prioritized list from `${{ steps.affected-docs.outputs.AFFECTED_DOCS }}`. Review the broader documentation list only when there is a clear link to the recent source changes.


 ### 1. Analyze Pre-computed Changes

-Review the git commit list above. For commits that touch source code (`src/`, `containers/`, `scripts/`), use `git show <sha>` to examine the actual diffs and understand what changed.
+If `${{ steps.has-changes.outputs.has_changes }}` is `false`, exit immediately using a no-op result without editing files or creating a PR.


lpcox · 2026-05-15T14:31:01Z

@copilot address review feedback

Copilot · 2026-05-15T14:43:46Z

@copilot address review feedback

Addressed in 80a8b78. I fixed the workflow review items: robust $GITHUB_OUTPUT writes with safe delimiters, shallow-history handling, affected docs now emit doc paths, moved dynamic context out of activation-time prompt interpolation, and regenerated the lock file accordingly.

github-actions · 2026-05-15T14:51:27Z

Smoke Test Results

Test Summary: 2/3 PASS (1 FAIL)

Test	Result	Details
GitHub API	❌ FAIL	gh CLI authentication failed (401: Bad credentials)
Playwright	✅ PASS	Successfully navigated to github.com, title contains 'GitHub'
File Verify	✅ PASS	smoke-test-claude-25923980148.txt exists with expected content

Note: GitHub API test failed due to gh CLI authentication in this environment. Playwright and file verification tests passed successfully.

💥 [THE END] — Illustrated by Smoke Claude

github-actions · 2026-05-15T14:51:41Z

Smoke Test: Copilot BYOK (Offline) Mode

Test	Result
GitHub MCP connectivity	❌ 401 Bad credentials
GitHub.com HTTP	⚠️ Pre-step data unavailable (template not expanded)
File write/read	⚠️ Pre-step data unavailable (template not expanded)
BYOK inference (api-proxy → api.githubcopilot.com)	✅

Running in BYOK offline mode (COPILOT_OFFLINE=true) via api-proxy → api.githubcopilot.com.

Overall: FAIL — GitHub MCP returned 401 and pre-step outputs were not injected.

🔑 BYOK report filed by Smoke Copilot BYOK

github-actions · 2026-05-15T14:52:00Z

🔬 Smoke Test Results

Test	Status
GitHub MCP connectivity	❌ 401 Bad credentials
GitHub.com HTTP connectivity	⚠️ Template vars unresolved
File write/read	⚠️ Template vars unresolved

Overall: FAIL — GitHub MCP returned 401; workflow template variables (${{ steps.smoke-data.outputs.* }}) were not substituted before agent execution.

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-05-15T14:52:38Z

Smoke Test

PRs: Cap /reflect effective-token totals at configured maxEffectiveTokens; refactor: [Export Audit] Remove test-only re-exports from barrel modules
GitHub PR review: ✅
SafeInputs GH / Tavily / Discussion: ❌
Playwright / File / Bash / Build: ✅
Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-05-15T14:53:01Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	passed	✅ PASS
Node.js	execa	✅	passed	✅ PASS
Node.js	p-limit	✅	passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #3202 · ● 5.1M · ◷

github-actions · 2026-05-15T14:53:14Z

Smoke test in progress...

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

github-actions · 2026-05-15T14:54:41Z

Smoke Test Results — FAIL ❌

Check	Result
Redis PING	❌ Timeout (no response)
PostgreSQL pg_isready	❌ No response
PostgreSQL SELECT 1	❌ No response

host.docker.internal is unreachable from this environment. Service containers do not appear to be running or accessible. Overall: FAIL

🔌 Service connectivity validated by Smoke Services

Initial plan

c4ccda6

Copilot AI assigned Copilot and lpcox May 15, 2026

Copilot started work on behalf of lpcox May 15, 2026 13:26 View session

Copilot AI linked an issue May 15, 2026 that may be closed by this pull request

⚡ Copilot Token Optimization2026-05-15 — Documentation Maintainer #3200

Closed

9 tasks

feat: optimize documentation maintainer workflow token usage

fd601d5

Copilot AI changed the title ~~[WIP] Optimize Copilot token usage based on analysis~~ Optimize Documentation Maintainer token usage and prompt/cache behavior May 15, 2026

Copilot AI requested a review from lpcox May 15, 2026 13:33

Copilot finished work on behalf of lpcox May 15, 2026 13:33

lpcox marked this pull request as ready for review May 15, 2026 13:52

lpcox requested a review from Mossaka as a code owner May 15, 2026 13:52

Copilot AI review requested due to automatic review settings May 15, 2026 13:52

Copilot started reviewing on behalf of lpcox May 15, 2026 13:53 View session

This comment has been minimized.

Sign in to view

github-actions Bot added the build-test label May 15, 2026

This comment has been minimized.

Sign in to view

github-actions Bot added the smoke-gemini label May 15, 2026

Copilot AI reviewed May 15, 2026

View reviewed changes

Copilot started work on behalf of lpcox May 15, 2026 14:37 View session

fix: address doc maintainer workflow review feedback

80a8b78

Copilot finished work on behalf of lpcox May 15, 2026 14:44

lpcox merged commit 39720f5 into main May 15, 2026
61 of 65 checks passed

lpcox deleted the copilot/optimize-copilot-token-usage branch May 15, 2026 15:10

		git log --since="7 days ago" --name-only --format="" -- src/ containers/ scripts/ \| \
		grep -v '^$' \| sort -u \| head -30

Conversation

Copilot AI commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✨ Enhancement

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

lpcox commented May 15, 2026

Uh oh!

Copilot AI commented May 15, 2026

Uh oh!

github-actions Bot commented May 15, 2026

Smoke Test Results

Uh oh!

github-actions Bot commented May 15, 2026

Smoke Test: Copilot BYOK (Offline) Mode

Uh oh!

github-actions Bot commented May 15, 2026

🔬 Smoke Test Results

Uh oh!

github-actions Bot commented May 15, 2026

Smoke Test

Uh oh!

github-actions Bot commented May 15, 2026

🏗️ Build Test Suite Results

Uh oh!

github-actions Bot commented May 15, 2026

Uh oh!

github-actions Bot commented May 15, 2026

Smoke Test Results — FAIL ❌

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented May 15, 2026 •

edited

Loading