Skip to content

Commit 6cddd07

Browse files
authored
feat: add copilot runner adapter (#48)
## What is this? This PR adds GitHub Copilot CLI as a supported Skillgym runner. It lets suites run against `agent.type: "copilot"`, keeps Copilot session artifacts, and documents the new support across the README, docs, and website. ## How does it work? The new adapter launches `copilot` in non-interactive JSONL mode with the configured model, isolates Copilot state under each artifact directory, and normalizes Copilot messages, tool calls, file reads, commands, explicit skill loads, and OpenTelemetry token usage into Skillgym session reports. The runner registry, config parser, public types, max-step monitor, and examples now recognize the `copilot` agent type. ## Why is this useful? Skill authors can now compare GitHub Copilot CLI against the existing OpenCode, Codex, Claude Code, and Cursor Agent runners using the same Skillgym cases and assertions. Explicit Copilot skill-load events are captured as detected skills, so skill-selection suites can verify that Copilot actually loaded the expected skill instead of relying only on file-read inference.
1 parent 9bf9411 commit 6cddd07

19 files changed

Lines changed: 1167 additions & 11 deletions
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
"skillgym": minor
3+
---
4+
5+
Add GitHub Copilot CLI as a supported runner with JSONL session normalization, isolated Copilot runtime state, token usage capture, explicit skill detection, docs, and website coverage.

DICTIONARY.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ Before adding or changing a term, check this file and ask the user for approval.
1515
- `prompt`: Exact instruction sent to the agent.
1616
- `runner`: Configured execution target: agent type plus model.
1717
- `agent`: External coding agent doing the work.
18+
- `copilot`: GitHub Copilot CLI agent type.
1819
- `suite run`: One top-level `skillgym run ...` over one suite.
1920
- `execution`: One case x runner unit of work, including all its repetitions and retries.
2021
- `repetition`: One planned sample within an execution.

README.md

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ When you evaluate agent skills manually, it is hard to tell whether the agent ac
1616
- Codex CLI
1717
- Claude Code
1818
- Cursor Agent (Cursor CLI `agent`)
19+
- GitHub Copilot CLI (`copilot`)
1920

2021
## Quick start
2122

@@ -66,6 +67,12 @@ const config = {
6667
model: "composer-2-fast",
6768
},
6869
},
70+
"copilot-main": {
71+
agent: {
72+
type: "copilot",
73+
model: "gpt-5.4-mini",
74+
},
75+
},
6976
},
7077
} satisfies SkillgymConfig;
7178

@@ -165,7 +172,7 @@ Most important config properties:
165172
- `run.maxSteps`: best-effort limit on streamed agent steps before skillgym terminates the execution
166173
- `run.workspace`: default workspace mode for the suite
167174
- `defaults.timeoutMs`: default per-case timeout
168-
- `runners.<id>.agent.type`: which agent integration to use, currently `opencode`, `codex`, `claude-code`, or `cursor-agent`
175+
- `runners.<id>.agent.type`: which agent integration to use, currently `opencode`, `codex`, `claude-code`, `cursor-agent`, or `copilot`
169176
- `runners.<id>.agent.model`: model passed to that runner
170177
- `snapshots`: token regression baseline configuration
171178

@@ -181,7 +188,7 @@ The execution unit is one case x runner pair. `skillgym` expands the suite into
181188

182189
For concurrent schedules, `run.maxParallel` defaults to `os.availableParallelism()`. This limits how many Skillgym executions are active at once; it does not pin or limit CPU cores used by an individual agent process.
183190

184-
Concurrent schedules do not copy or isolate the workspace by themselves. Overlapping executions may still interact through the same filesystem state and live runner output unless you use isolated workspaces. OpenCode, Codex, and Claude Code runtime state are isolated per execution under each artifact directory.
191+
Concurrent schedules do not copy or isolate the workspace by themselves. Overlapping executions may still interact through the same filesystem state and live runner output unless you use isolated workspaces. OpenCode, Codex, Claude Code, and Copilot runtime state are isolated per execution under each artifact directory.
185192

186193
`run.repeat` is useful when you want stability sampling instead of a single lucky pass. Each case x runner execution keeps running until it records the requested number of successful classified results, or stops early when one repetition still fails after exhausting `run.repeatFailure` retries.
187194

docs/explain.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,7 @@ Deferred explain is implemented for:
9191
- Codex
9292
- Claude Code
9393
- Cursor Agent
94+
- GitHub Copilot CLI
9495

9596
Each runner must have a resumable session id captured during the original run.
9697

docs/readme.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ Supported in V1:
1818
- Codex CLI
1919
- Claude Code CLI
2020
- Cursor Agent CLI
21+
- GitHub Copilot CLI
2122

2223
## Core principles
2324

docs/session-report.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,11 +50,12 @@ export interface SessionReport {
5050
stderrPath?: string;
5151
sessionPath?: string;
5252
exportPath?: string;
53+
telemetryPath?: string;
5354
};
5455
}
5556
```
5657

57-
`runner.id` is the configured runner id, and `runner.agent.type` identifies the backing integration such as `opencode`, `codex`, or `claude-code`.
58+
`runner.id` is the configured runner id, and `runner.agent.type` identifies the backing integration such as `opencode`, `codex`, `claude-code`, `cursor-agent`, or `copilot`.
5859

5960
## Skill detection
6061

@@ -110,7 +111,7 @@ Preferred source order:
110111

111112
1. `inputTokens + outputTokens + reasoningTokens - cacheTokens`
112113

113-
This keeps OpenCode, Codex, and Claude Code comparable even though the raw provider fields use different cache semantics.
114+
This keeps OpenCode, Codex, Claude Code, Cursor Agent, and Copilot comparable even though the raw provider fields use different cache semantics.
114115

115116
Snapshot enforcement uses the selected token metric directly and does not fall back to character counts when token usage is unavailable.
116117

skillgym.config.ts

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,12 @@ const config = {
3535
model: "auto",
3636
},
3737
},
38+
"copilot-main": {
39+
agent: {
40+
type: "copilot",
41+
model: "gpt-5.4-mini",
42+
},
43+
},
3844
},
3945
} satisfies SkillgymConfig;
4046

0 commit comments

Comments
 (0)