Skip to content

Optimize test-coverage-improver prompt caching and run cadence#3298

Merged
lpcox merged 2 commits into
mainfrom
copilot/test-coverage-improver
May 17, 2026
Merged

Optimize test-coverage-improver prompt caching and run cadence#3298
lpcox merged 2 commits into
mainfrom
copilot/test-coverage-improver

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 17, 2026

test-coverage-improver was spending most of its token budget on uncached ambient prompt context: dynamic coverage data was injected into the middle of the prompt, forcing every request to miss Claude prefix caching. The workflow also exposed redundant expensive bash commands and ran twice daily despite low incremental value.

  • Prompt cacheability

    • Moved the run-specific ## Current Coverage Status block to the end of the workflow body so the static guidance becomes a reusable prompt prefix across runs.
    • Removed the full COVERAGE_JSON interpolation from the prompt and from the pre-step outputs. The workflow now keeps only the human-readable summary and the prioritized low-coverage list in prompt context.
  • Tool surface reduction

    • Dropped redundant bash permissions for:
      • npm run build
      • npm run test:coverage
    • Kept npm run test, npm run lint, and file inspection commands so the agent can still iterate on tests without re-running the full precomputed pipeline.
  • Run frequency

    • Changed the schedule from twice daily to once daily at 08:00 UTC.
  • Prompt guidance

    • Updated the validation section to direct the agent toward precomputed coverage artifacts first, with coverage/coverage-summary.json available only as an on-demand file read.

Example of the prompt reshaping:

## Static guidance
...all instructions, priorities, and examples...

## Current Coverage Status

### COVERAGE_SUMMARY.md
${{ steps.coverage-md.outputs.COVERAGE_MD }}

### Files Below 80% Coverage
${{ steps.low-coverage.outputs.LOW_COVERAGE }}

Copilot AI changed the title [WIP] Optimize Copilot token usage for test coverage improvement Optimize test-coverage-improver prompt caching and run cadence May 17, 2026
Copilot AI requested a review from lpcox May 17, 2026 16:08
Copilot finished work on behalf of lpcox May 17, 2026 16:08
@lpcox lpcox marked this pull request as ready for review May 17, 2026 17:14
@lpcox lpcox requested a review from Mossaka as a code owner May 17, 2026 17:14
Copilot AI review requested due to automatic review settings May 17, 2026 17:14
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Optimizes the test-coverage-improver workflow to improve Claude prompt prefix caching, reduce tool surface, and lower run cadence from twice daily to once daily.

Changes:

  • Moved run-specific coverage context to the end of the prompt and removed the full COVERAGE_JSON interpolation so static guidance can be cached as a prefix.
  • Dropped redundant npm run build and npm run test:coverage bash permissions; kept npm run test/lint.
  • Changed schedule from 0 8,20 * * * to 0 8 * * * and regenerated the lock file accordingly.
Show a summary per file
File Description
.github/workflows/test-coverage-improver.md Reordered prompt, removed COVERAGE_JSON step/interpolation, trimmed bash tools, updated schedule and validation guidance.
.github/workflows/test-coverage-improver.lock.yml Regenerated lock to reflect frontmatter changes (cron, allow-tool list, removed coverage-summary step/env).

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 2/2 changed files
  • Comments generated: 0

@github-actions
Copy link
Copy Markdown
Contributor

Smoke Test Results

  • ❌ GitHub API: Failed (gh CLI authentication issue - 401: Bad credentials)
  • ✅ Playwright: Passed (navigated to github.com, title confirmed)
  • ✅ File verify: Passed (smoke-test-claude-25995885841.txt verified)

Status: 2/3 tests passed

💥 [THE END] — Illustrated by Smoke Claude

@github-actions
Copy link
Copy Markdown
Contributor

🔍 Smoke Test Results

Test Result
GitHub MCP connectivity ❌ (401 — MCP not authenticated)
GitHub.com HTTP connectivity ⚠️ (pre-step template vars not expanded)
File write/read ⚠️ (pre-step template vars not expanded)

Overall: FAIL — Workflow template variables (${{ steps.smoke-data.outputs.* }}) were not substituted before agent execution, and GitHub MCP returned 401 Bad Credentials.

📰 BREAKING: Report filed by Smoke Copilot

@github-actions
Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Offline) Mode

Test Result
GitHub MCP connectivity ❌ (401 - MCP tools not authenticated in this environment)
GitHub.com HTTP ⚠️ (pre-step data not injected — template vars unexpanded)
File write/read ⚠️ (pre-step data not injected — template vars unexpanded)
BYOK inference (this response)

Running in BYOK offline mode (COPILOT_OFFLINE=true) via api-proxy → api.githubcopilot.com

Overall: FAIL — pre-computed smoke data was not injected (template variables unexpanded); GitHub MCP returned 401.

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions
Copy link
Copy Markdown
Contributor

Smoke Test

Merged PRs: chore: recompile test-coverage-improver lock file; chore: run test coverage improver twice daily
SafeInputs PRs: fix: postprocess claude-token-optimizer lock file to use local awf build; refactor: split api-proxy-service.test.ts into focused test modules
✅ Playwright, file/bash, build
❌ GitHub MCP/safeinputs, Tavily, discussion query
Overall status: FAIL

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

@github-actions
Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia 1/1 passed ✅ PASS
Bun hono 1/1 passed ✅ PASS
C++ fmt N/A ✅ PASS
C++ json N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world N/A ✅ PASS
.NET json-parse N/A ✅ PASS
Go color 1/1 passed ✅ PASS
Go env 1/1 passed ✅ PASS
Go uuid 1/1 passed ✅ PASS
Java gson 1/1 passed ✅ PASS
Java caffeine 1/1 passed ✅ PASS
Node.js clsx All passed ✅ PASS
Node.js execa All passed ✅ PASS
Node.js p-limit All passed ✅ PASS
Rust fd 1/1 passed ✅ PASS
Rust zoxide 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #3298 · ● 4.8M ·

@github-actions
Copy link
Copy Markdown
Contributor

Gemini Smoke Test: FAIL (SSL 35, MCP 401)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

@github-actions
Copy link
Copy Markdown
Contributor

Smoke Test Results

  • ❌ Redis (PING): timeout — no response from host.docker.internal:6379
  • ❌ PostgreSQL pg_isready: no response from host.docker.internal:5432
  • ❌ PostgreSQL SELECT 1: timeout

Overall: FAILhost.docker.internal is not reachable from this runner environment. Service containers may not be running or the hostname is not resolvable.

🔌 Service connectivity validated by Smoke Services

@lpcox lpcox merged commit 43bd596 into main May 17, 2026
65 of 70 checks passed
@lpcox lpcox deleted the copilot/test-coverage-improver branch May 17, 2026 18:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

⚡ Copilot Token Optimization2026-05-17 — test-coverage-improver

3 participants