fix(ci): prevent empty safeoutputs schema probes in smoke-claude workflow by Copilot · Pull Request #5307 · github/gh-aw-firewall

Copilot · 2026-06-19T22:29:11Z

Reproduce the failing Smoke Claude run and inspect logs
Add a prompt guardrail preventing empty add_comment / add_labels calls
Regenerate smoke-claude.lock.yml for the source change
Add a focused smoke workflow test assertion for the guardrail
Run targeted tests and workflow compile/postprocess steps
Run CodeQL check

github-actions · 2026-06-19T23:21:51Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	97.62%	97.66%	📈 +0.04%
Statements	97.56%	97.60%	📈 +0.04%
Functions	98.85%	98.85%	➡️ +0.00%
Branches	93.24%	93.27%	📈 +0.03%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/workdir-setup.ts`	92.7% → 94.5% (+1.82%)	92.7% → 94.5% (+1.82%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

This PR updates the Smoke Claude workflow prompt guidance (and its lock metadata) to prevent “schema probe” / empty-argument invocations of safeoutputs tools, and adds a focused Jest assertion to ensure the guardrail stays present in the source workflow.

Changes:

Add an explicit prompt instruction to never call add_comment / add_labels with empty arguments (avoid schema-probing).
Update the smoke workflow Jest test to assert the new instruction exists in smoke-claude.md.
Regenerate smoke-claude.lock.yml metadata to reflect the updated workflow source.

Show a summary per file

File	Description
scripts/ci/smoke-claude-workflow.test.ts	Adds a test assertion to enforce the new “no empty safeoutputs calls” guardrail in the workflow source.
.github/workflows/smoke-claude.md	Adds the explicit instruction to avoid empty-argument `add_comment` / `add_labels` calls.
.github/workflows/smoke-claude.lock.yml	Updates compiled workflow metadata (body hash) corresponding to the source prompt change.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 3/3 changed files
Comments generated: 0

lpcox · 2026-06-19T23:49:26Z

@copilot address the review feedback and fix the pr title and summary

lpcox · 2026-06-20T02:02:26Z

@copilot address the review feedback and fix the pr title and summary

github-actions · 2026-06-20T02:24:15Z

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

github-actions · 2026-06-20T02:24:17Z

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

github-actions · 2026-06-20T02:24:18Z

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

github-actions · 2026-06-20T02:24:19Z

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

github-actions · 2026-06-20T02:24:19Z

✅ Build Test Suite completed successfully!

github-actions · 2026-06-20T02:24:20Z

❌ Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed...

github-actions · 2026-06-20T02:24:21Z

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

github-actions · 2026-06-20T02:24:21Z

✅ Smoke Gemini completed. All facets verified. 💎

github-actions · 2026-06-20T02:24:22Z

❌ Smoke Claude failed

github-actions · 2026-06-20T02:24:23Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

github-actions · 2026-06-20T02:24:25Z

🚀 Security Guard has started processing this pull request

github-actions · 2026-06-20T02:24:29Z

❌ Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

github-actions · 2026-06-20T02:24:43Z

🔌 Smoke Services — All services reachable! ✅

github-actions · 2026-06-20T02:24:48Z

✅ Contribution Check completed successfully!

github-actions · 2026-06-20T02:27:20Z

🔬 Smoke Test Results

PR: [WIP] Debug smoke Claude workflow failure
Author: @Copilot | Assignees: @lpcox, @Copilot

Test	Result
✅ GitHub MCP connectivity	PASS — PR list fetched
✅ GitHub.com connectivity	PASS — HTTP 200
❌ File write/read test	FAIL — pre-step template vars not substituted

Overall: FAIL (pre-computed data missing due to unresolved ${{ }} expressions)

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-06-20T02:27:27Z

Smoke Test: Copilot PAT — PASS ✅

Test	Result
GitHub MCP connectivity	✅
GitHub.com HTTP (200)	✅
File write/read	✅

Auth mode: PAT (COPILOT_GITHUB_TOKEN)
PR: "[WIP] Debug smoke Claude workflow failure" — @Copilot (author), @lpcox (assignee)

🔑 PAT report filed by Smoke Copilot PAT

github-actions · 2026-06-20T02:27:58Z

Smoke Test: Copilot BYOK (Direct) Mode ✅ PASS

Results:

✅ MCP connectivity: Retrieved 2 merged PRs
✅ GitHub.com: HTTP 200
✅ File I/O: Created and verified
✅ BYOK inference: Direct mode working (agent → api-proxy → api.githubcopilot.com)

Running in direct BYOK mode via COPILOT_PROVIDER_API_KEY.

/cc @lpcox @Copilot

🔑 BYOK report filed by Smoke Copilot BYOK

github-actions · 2026-06-20T02:28:49Z

Smoke Test: API Proxy OpenTelemetry Tracing

Scenario	Result	Detail
1. Module Loading	✅	`otel.js` loads cleanly; exports: `startRequestSpan`, `setTokenAttributes`, `setBudgetAttributes`, `endSpan`, `endSpanError`, `shutdown`, `isEnabled`
2. Test Suite	✅	59 tests passed, 0 failed across 2 suites (`otel.test.js`, `otel-fanout.test.js`)
3. Env Var Forwarding	✅	`api-proxy-service-config.ts` forwards `GH_AW_OTLP_ENDPOINTS`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, `GITHUB_AW_OTEL_TRACE_ID`, `GITHUB_AW_OTEL_PARENT_SPAN_ID`, `OTEL_SERVICE_NAME`
4. Token Tracker Integration	✅	`onUsage` callback present in `token-tracker-http.js` (line 324) as OTEL hook point
5. OTEL Diagnostics	✅	`isEnabled()` returns `true`; graceful fallback to `otel.jsonl` file export when no OTLP endpoint configured

All 5 scenarios pass. OTEL tracing integration is functioning correctly.

📡 OTel tracing validated by Smoke OTel Tracing

github-actions · 2026-06-20T02:28:55Z

Chroot Runtime Version Comparison

Runtime	Host Version	Chroot Version	Match?
Python	`Python 3.12.13`	`Python 3.12.3`	❌
Node.js	`v24.16.0`	`v22.22.3`	❌
Go	`go1.22.12`	`go1.22.12`	✅

Result: 1/3 tests passed — Go matches, but Python and Node.js versions differ between host and chroot environments.

Tested by Smoke Chroot

github-actions · 2026-06-20T02:29:06Z

Gemini Smoke Test Results

chore: upgrade gh-aw to v0.80.6 and recompile workflows #5294 chore: upgrade gh-aw to v0.80.6 and recompile workflows
[WIP] Refactor entrypoint.sh by extracting initialization phases into functions #5285 [WIP] Refactor entrypoint.sh by extracting initialization phases into functions
GitHub MCP Testing: ✅
GitHub.com Connectivity: ✅
File Writing Testing: ✅
Bash Tool Testing: ✅

Overall status: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

github-actions · 2026-06-20T02:29:49Z

[WIP] Debug smoke Claude workflow failure by Copilot
Merged PRs reviewed:

chore: upgrade gh-aw to v0.80.6 and recompile workflows
[WIP] Refactor entrypoint.sh by extracting initialization phases into functions
Checks:
GitHub title: ✅
Merged PR review: ✅
Discussion query: ✅
Temp file: ✅
Build: ✅
Overall: PASS

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-06-20T02:30:00Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	passed	✅ PASS
Node.js	execa	✅	passed	✅ PASS
Node.js	p-limit	✅	passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #5307 · 42.2 AIC · ⊞ 7.7K · ◷

github-actions · 2026-06-20T02:30:28Z

Smoke Test Results

Redis PING: ❌ (timeout, no PONG)
PostgreSQL pg_isready: ❌ (no response on port 5432)
PostgreSQL SELECT 1: ❌ (timeout, no output)

host.docker.internal resolves to 172.17.0.1 but service containers are unreachable.

Overall: FAIL

🔌 Service connectivity validated by Smoke Services

Copilot · 2026-06-20T03:16:46Z

@copilot address the review feedback and fix the pr title and summary

Updated the PR metadata: title is now fix(ci): prevent empty safeoutputs schema probes in smoke-claude workflow and the summary checklist has been rewritten to reflect the actual changes. There were no actionable reviewer findings to address in code, so no new commit was needed (latest commit remains 5bf60bd).

Initial plan

f784448

Copilot AI assigned Copilot and lpcox Jun 19, 2026

Copilot started work on behalf of lpcox June 19, 2026 22:36 View session

fix: prevent empty safeoutputs probes in smoke-claude prompt

5bf60bd

Copilot AI requested a review from lpcox June 19, 2026 22:51

lpcox marked this pull request as ready for review June 19, 2026 23:20

Copilot AI review requested due to automatic review settings June 19, 2026 23:20

Copilot started reviewing on behalf of lpcox June 19, 2026 23:20 View session

Copilot AI reviewed Jun 19, 2026

View reviewed changes

Copilot started work on behalf of lpcox June 19, 2026 23:55 View session

Copilot started work on behalf of lpcox June 20, 2026 02:02 View session

Copilot finished work on behalf of lpcox June 20, 2026 02:17

lpcox added the ready-for-aw label Jun 20, 2026

lpcox had a problem deploying to aoai-model June 20, 2026 02:24 — with GitHub Actions Failure

lpcox temporarily deployed to aoai-model June 20, 2026 02:27 — with GitHub Actions Inactive

github-actions Bot added the smoke-copilot-pat label Jun 20, 2026

github-actions Bot added the smoke-copilot-byok label Jun 20, 2026

github-actions Bot mentioned this pull request Jun 20, 2026

[aw] Contribution Check is missing required data #5289

Closed

github-actions Bot added the smoke-gemini label Jun 20, 2026

github-actions Bot added smoke-codex build-test labels Jun 20, 2026

Copilot started work on behalf of lpcox June 20, 2026 03:16 View session

lpcox changed the title ~~[WIP] Debug smoke Claude workflow failure~~ fix(ci): prevent empty safeoutputs schema probes in smoke-claude workflow Jun 20, 2026

Copilot finished work on behalf of lpcox June 20, 2026 03:17

lpcox merged commit 39bcb8b into main Jun 20, 2026
90 of 95 checks passed

lpcox deleted the copilot/debug-smoke-claude-failure-again branch June 20, 2026 15:40

Conversation

Copilot AI commented Jun 19, 2026 • edited by lpcox Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 19, 2026

✅ Coverage Check Passed

Overall Coverage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

lpcox commented Jun 19, 2026

Uh oh!

lpcox commented Jun 20, 2026

Uh oh!

github-actions Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026

Uh oh!

github-actions Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 20, 2026

🔬 Smoke Test Results

Uh oh!

github-actions Bot commented Jun 20, 2026

Smoke Test: Copilot PAT — PASS ✅

Uh oh!

github-actions Bot commented Jun 20, 2026

Smoke Test: Copilot BYOK (Direct) Mode ✅ PASS

Uh oh!

github-actions Bot commented Jun 20, 2026

Smoke Test: API Proxy OpenTelemetry Tracing

Uh oh!

github-actions Bot commented Jun 20, 2026

Chroot Runtime Version Comparison

Uh oh!

github-actions Bot commented Jun 20, 2026

Gemini Smoke Test Results

Uh oh!

github-actions Bot commented Jun 20, 2026

Uh oh!

github-actions Bot commented Jun 20, 2026

🏗️ Build Test Suite Results

Copilot AI commented Jun 19, 2026 •

edited by lpcox

Loading

github-actions Bot commented Jun 20, 2026 •

edited

Loading

github-actions Bot commented Jun 20, 2026 •

edited

Loading

github-actions Bot commented Jun 20, 2026 •

edited

Loading

github-actions Bot commented Jun 20, 2026 •

edited

Loading

github-actions Bot commented Jun 20, 2026 •

edited

Loading

github-actions Bot commented Jun 20, 2026 •

edited

Loading

github-actions Bot commented Jun 20, 2026 •

edited

Loading

github-actions Bot commented Jun 20, 2026 •

edited

Loading

github-actions Bot commented Jun 20, 2026 •

edited

Loading

github-actions Bot commented Jun 20, 2026 •

edited

Loading

github-actions Bot commented Jun 20, 2026 •

edited

Loading

github-actions Bot commented Jun 20, 2026 •

edited

Loading

github-actions Bot commented Jun 20, 2026 •

edited

Loading