Skip to content

feat: add GitHub Copilot CLI and app support#2323

Open
andrey-esipov wants to merge 1 commit into
garrytan:mainfrom
andrey-esipov:andrey-esipov-copilot-host-support
Open

feat: add GitHub Copilot CLI and app support#2323
andrey-esipov wants to merge 1 commit into
garrytan:mainfrom
andrey-esipov:andrey-esipov-copilot-host-support

Conversation

@andrey-esipov

@andrey-esipov andrey-esipov commented Jul 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Problem

gstack has no official GitHub Copilot host, so generated skills, installation paths, setup detection, and runtime assets do not work as one supported flow in either the standalone GitHub Copilot CLI or GitHub Copilot desktop app project sessions.

GitHub documents one Agent Skills contract for both surfaces. This change adds one declarative copilot host instead of duplicating path roots or maintaining separate CLI and app variants.

Verified Copilot contract

Contract Verified behavior Source
Executable Detect the standalone copilot executable. Do not treat legacy gh copilot as a current detection alias. Copilot CLI reference
Personal discovery Copilot discovers personal skills from ~/.copilot/skills and ~/.agents/skills. gstack uses ~/.copilot/skills to avoid Codex's existing .agents namespace. Add skills to Copilot CLI
Repository discovery Copilot discovers repository skills from .github/skills, .agents/skills, and .claude/skills. gstack uses .github/skills so local Copilot installs cannot clobber Codex's .agents/skills/gstack. About Agent Skills
CLI and app contract GitHub documents Agent Skills for Copilot CLI, the Copilot coding agent, and Copilot app customization. Both surfaces consume the same skill format and discovery model. About Agent Skills, Add skills to the cloud agent, Customize the GitHub Copilot app
Frontmatter Skills expose name and description; names match their directories, use lowercase hyphenated identifiers, and stay within 64 characters; descriptions stay within 1024 characters. Agent Skills specification
Config root COPILOT_HOME relocates Copilot CLI configuration. gstack resolves the personal skill root through it and normalizes Git Bash paths. Copilot CLI config directory reference

The resulting host contract is:

hostSubdir:      .copilot
globalRoot:      ${COPILOT_HOME:-~/.copilot}/skills/gstack
localSkillRoot:  .github/skills/gstack
surfaces:        cli, app

Implementation

  • Adds hosts/copilot.ts and registers it as the eleventh host.
  • Generates 54 Copilot skills under ignored .copilot/skills, with gstack- frontmatter names that match their generated directories.
  • Keeps only name and description in frontmatter and enforces GitHub's name and description limits.
  • Rewrites Claude/Codex paths and tool names to Copilot-native bash, view, task, skill, and ask_user semantics.
  • Preserves same-host skill orchestration because Copilot exposes both skill and task; suppresses optional gbrain context/save blocks and excludes /codex.
  • Adds config-driven setup detection, global and repository-local installation, Windows copy refresh, COPILOT_HOME handling, uninstall cleanup, runtime sidecars, and host-aware upgrades.
  • Stores .source-path in the Copilot runtime root so upgrades refresh from the source checkout without copying a repository into the installed skill tree.
  • Documents CLI and app installation, shared discovery behavior, local-install portability, and host-author guidance.
  • Adds focused Copilot regression tests for detection, path isolation, frontmatter, content rewrites, Windows paths, runtime closure, global/local installs, command execution, upgrades, and uninstall.

Validation

Command or flow Result
bun run scripts/host-config-export.ts validate 11 host configs valid
bun run gen:skill-docs --host copilot 54 Copilot skills generated; all fresh
bun run gen:skill-docs --host all --dry-run 610 generated artifacts fresh
bun test test/copilot-host.test.ts test/host-config.test.ts test/gen-skill-docs.test.ts test/setup-windows-fallback.test.ts 505 passed, 0 failed
bun run build Passed; generated all hosts and compiled all binaries
bun test Complete free suite passed, exit 0
Hermetic global install in temporary HOME/COPILOT_HOME 54 enabled personal gstack skills discovered by copilot skill list --json; runtime assets resolved
Hermetic repository install in a temporary repo 54 enabled project gstack skills discovered from .github/skills; runtime assets resolved
Live Copilot CLI 1.0.72 Generated gstack-careful invoked successfully and returned GSTACK_COPILOT_SKILL_OK
Fresh Copilot desktop app project session Native app skill tool discovered and invoked gstack-careful, returning APP_GSTACK_COPILOT_SKILL_OK
Independent pre-landing review One latent name-limit diagnostic bug found and fixed; final review has 0 unresolved findings

bun run skill:check remains blocked by the existing missing generated claude/SKILL.md; the same failure reproduces on origin/main. The Windows-safe suite's observability marker assertion also reproduces on origin/main because it expects six markers while main currently has five. Neither baseline was changed or hidden here.

Paid evals were not run locally. GitHub started them for this fork PR, but pull_request workflows do not receive the base repository's ANTHROPIC_API_KEY. Seven Anthropic-backed jobs therefore stop before skill execution with error_api, zero tokens, and zero cost; the aggregate report job mirrors those failures. The same judge and /gstack-upgrade evals reproduce on origin/main when run without credentials. Deterministic freshness, Windows, setup, lint, and build-image checks pass.

This change is deterministic host plumbing covered by static validation, generator tests, setup tests, hermetic installs, and live discovery/invocation.

Live versus contract-level app coverage

The desktop app claim is not inferred from CLI behavior. A newly created Copilot project-session agent independently discovered and invoked the installed personal gstack-careful skill through the app's native skill tool.

Repository-local .github/skills behavior is covered by GitHub's shared Agent Skills documentation, focused host assertions, and a real copilot skill list --json project discovery flow. The app project-session smoke test exercised the personal root, not a second live repository-local app install.

Compatibility and risk

  • .agents/skills/gstack remains owned by Codex. Copilot stages under .copilot and installs local skills under .github/skills, so the hosts do not share or overwrite roots.
  • Repository-local installs use links on Unix and refreshed copies on Windows. They are machine-local development installs, not portable committed bundles.
  • The generated gstack-ship skill remains above the existing advisory 160 KB token ceiling. Generation warns but does not fail, matching repository policy.
  • No VERSION bump is included.

@andrey-esipov
feat: add GitHub Copilot host support
0b41da9 
Add one shared Agent Skills host contract for Copilot CLI and the Copilot desktop app, including generation, setup, upgrades, uninstall, documentation, and focused regression coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@trunk-io

trunk-io Bot commented Jul 22, 2026

Copy link
Copy Markdown

Merging to main in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@andrey-esipov

andrey-esipov commented Jul 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

E2E failure root-cause report

I downloaded the complete logs and result artifacts for all seven failing eval shards plus report, then reran every underlying failing test on both this PR (0b41da9d) and clean origin/main (a3259400).

This is a cross-repository fork PR (andrey-esipov/astack to garrytan/gstack). GitHub documents that pull_request workflows from forks do not receive repository secrets and get a read-only GITHUB_TOKEN: fork workflow security.

Check First causal failure CI evidence PR/main comparison Branch-caused?
e2e-browse claude -p cannot authenticate 4 tests x 3 attempts, each error_api, 1 turn, 0 tokens, $0.00 Same 4 named tests fail on clean main with credentials scrubbed No
e2e-workflow claude -p cannot authenticate 2 tests x 3 attempts, each error_api, 1 turn, 0 tokens, $0.00 Same 2 fail on clean main No
llm-judge Anthropic SDK throws before a request: Could not resolve authentication method Both selected tests fail all 3 attempts; collector records 0 tests because setup throws before result recording Same 2 fail on clean main No
e2e-deploy claude -p cannot authenticate 3 tests x 3 attempts, each error_api, 1 turn, 0 tokens, $0.00 Same 3 fail on clean main No
e2e-plan claude -p cannot authenticate 7 tests x 3 attempts, each error_api, 1 turn, 0 tokens, $0.00 Same 7 fail on clean main No
e2e-pty-plan-smoke Interactive Claude starts unauthenticated Both tests show Not logged in, then hit their 300-second timeout; 3 attempts each Faithful project-skill registration on clean main produces the same 2 timeouts No
e2e-design claude -p cannot authenticate 3 tests x 3 attempts, each error_api, 1 turn, 0 tokens, $0.00 Same 3 fail on clean main No
report Fork token cannot create the PR comment Report assembly completes, then gh pr comment fails with Resource not accessible by integration (addComment) Independent of branch code; fork token is read-only No

The workflow explicitly maps ANTHROPIC_API_KEY, OPENAI_API_KEY, and GEMINI_API_KEY from repository secrets. All three are empty in every failing shard. Branch code does run during the preceding successful build and fixture setup, but no model executes a skill instruction before authentication fails. The result artifacts contain 57 failed retry attempts, zero tokens, and $0.00.

Why these tests ran: scripts/gen-skill-docs.ts is a declared touchfile for the browse, plan, deploy, design, ship, and voice tests. The upgrade tests are also selected by gstack-upgrade/SKILL.md.tmpl. The two PTY gate tests do not use diff selection and therefore run on every PR.

Local controls:

  • Scrubbed credentials: PR and main reproduce the same 19/19 agent failures, 2/2 judge failures, and 2/2 PTY timeouts.
  • Authenticated control: no usable local Anthropic environment is available. claude auth status surfaced stale cached claude.ai metadata, but every non-interactive and PTY request was rejected with 400 Not a valid API key for this workspace on both revisions. This is not a working subscription or authenticated control. The direct SDK judge also requires an API key. There is still no PR/main divergence.

I am not changing the workflow in this host-support PR. Using pull_request_target to run untrusted fork code with secrets would be unsafe. A clean fork skip would require a repository-level policy decision about whether required paid evals should be neutral on forks and how maintainers trigger a trusted rerun; adding that here would be unrelated and could mask required coverage.

No code change or empty commit is warranted. The actionable remediation is a trusted base-repository rerun with provider secrets, or a separate workflow-policy change owned by maintainers.

@andrey-esipov
andrey-esipov marked this pull request as ready for review July 22, 2026 19:57
@dgrant dgrant mentioned this pull request Jul 22, 2026
Open
@andrey-esipov

andrey-esipov commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Eval failures are fork-secret injection, not a code defect

The red evals (*) / report checks are all the same root cause: this is a fork PR, and GitHub does not expose repository secrets (ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY) to a pull_request triggered from a fork. Every selected eval that calls the Anthropic API therefore dies at the SDK auth step:

error: Could not resolve authentication method. Expected either apiKey or authToken to be set...

That surfaces as Received: "error_api" in the browse/plan/design/workflow/deploy suites and the auth error in llm-judge. The passing eval jobs are not real passes — e2e-codex/e2e-gemini self-skip (those CLIs aren't in the CI container) and e2e-routing/e2e-review/e2e-qa-* selected 0 tests for this diff.

Why this PR runs the evals when most fork PRs don't

The diff-based selector expands to 42/181 tests here because this change touches core/shared files (setup, scripts/resolvers/*, scripts/gen-skill-docs.ts, scripts/host-config.ts, hosts/index.ts). Narrow PRs select 0 LLM tests and stay green; a host-support change necessarily touches shared plumbing, so it fans out to API-calling suites that a fork can never authenticate.

The non-secret gates all pass: actionlint, check-freshness, build-image, windows-free-tests, windows-setup, plus the new non-LLM unit tests (test/copilot-host.test.ts, test/host-config.test.ts).

To get real green evals (needs a maintainer with write access)

Push this PR's existing commits to an upstream branch (fork and parent share git objects — no rework), then open the PR from there so CI has the secrets:

git fetch origin pull/2323/head
git push origin FETCH_HEAD:refs/heads/copilot-host-support
# then open a PR from garrytan/gstack:copilot-host-support -> main and close this as superseded

Or, since the eval reds are purely secret-availability and not code defects, this can be merged on the strength of the gates above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@andrey-esipov