feat: Prepare agent-eval for npm CLI usage by Copilot · Pull Request #42 · primer/agent-eval

Copilot · 2026-06-11T16:17:38Z

Prepares @primer/agent-eval for npm publishing as a runnable CLI, with support for running experiments from a caller-provided local folder while retaining typed experiment authoring.

CLI package metadata
- Adds agent-eval as the package binary.
- Publishes built CLI output, config types, and Node engine metadata.
- Bundles internal workspace code needed by the CLI and declares external runtime dependencies.
Local experiment loading
- Adds --experiments to load experiment files from a local directory.
- Supports experiment named exports and default exports.
- Falls back to packaged experiment lookup when no local directory is provided.
Typed config authoring
- Adds defineExperimentConfig from @primer/agent-eval/config.

import {defineExperimentConfig} from '@primer/agent-eval/config'

export const experiment = defineExperimentConfig({
  name: 'Example experiment',
  description: 'Compare treatment behavior',
  models: ['gpt-5.5'],
  evals: ['001-agent-uses-button-from-primer'],
  treatments: [],
})

COPILOT_GITHUB_TOKEN=... agent-eval --experiments ./experiments --experiment example

joshblack · 2026-06-30T15:21:54Z

@copilot resolve the merge conflicts in this pull request

Copilot · 2026-06-30T15:28:43Z

@copilot resolve the merge conflicts in this pull request

Resolved the merge conflicts in a3d2db9 and pushed the merge commit. Validation passed: pnpm run format:diff, pnpm run build, pnpm run lint, pnpm test --run, pnpm run type-check, secret scanning, and CodeQL.

joshblack · 2026-06-30T16:08:06Z

+Run local experiments with the `agent-eval` CLI:
+
+```sh
+COPILOT_GITHUB_TOKEN=... agent-eval --experiments ./experiments --experiment example


@copilot I think --experiment should support a name or path for the experiment (in cases where they don't want to specify a directory)

Implemented in b528b96. --experiment now accepts an existing local experiment file path when --experiments is not provided, while preserving the existing name lookup behavior. Validation passed: pnpm test --run packages/agent-eval/src/experiments.test.ts, pnpm run format:diff, pnpm run lint, pnpm run build, pnpm run type-check, pnpm test --run, secret scanning, and CodeQL.

joshblack · 2026-06-30T16:14:38Z

@copilot resolve the merge conflicts in this pull request

…-eval-for-publishing # Conflicts: # packages/agent-eval/package.json

Copilot · 2026-06-30T16:20:16Z

@copilot resolve the merge conflicts in this pull request

Resolved the merge conflicts in 6d6a52c and pushed the merge commit. Validation passed: pnpm run format:diff, pnpm run build, pnpm run lint, pnpm test --run, pnpm run type-check, secret scanning, and CodeQL.

joshblack · 2026-06-30T16:23:51Z

@copilot resolve the merge conflicts in this pull request

…-eval-for-publishing # Conflicts: # packages/agent-eval/src/cli.ts # packages/agent-eval/src/config.ts

Copilot · 2026-06-30T16:29:18Z

@copilot resolve the merge conflicts in this pull request

Resolved the merge conflicts in cd9a758 and pushed the merge commit. Validation passed: targeted agent-eval tests/type-check, pnpm run format:diff, pnpm run build, pnpm run lint, pnpm test --run, pnpm run type-check, secret scanning, and CodeQL.

Copilot

Pull request overview

Prepares @primer/agent-eval to be publishable as an npm-installed CLI while adding support for loading experiments from a caller-provided local directory and exporting typed config helpers for external experiment authoring.

Changes:

Adds a runnable agent-eval CLI entrypoint and updates build/bundling (Rolldown + DTS generation) to emit publishable artifacts.
Introduces local experiment discovery/loading (--experiments) with support for named (experiment) and default exports.
Exposes typed config helpers/types via @primer/agent-eval and @primer/agent-eval/config, and updates docs/CI workflow usage.

Show a summary per file

File	Description
README.md	Documents authoring typed experiments outside the repo and running them via the CLI.
pnpm-lock.yaml	Locks new/updated dependencies needed for CLI packaging and type output generation.
packages/sandbox/src/index.ts	Re-exports `McpServerConfig` type for downstream typing.
packages/agent-eval/tsconfig.build.json	Adds a build tsconfig for emitting declarations into `dist`.
packages/agent-eval/src/index.ts	Adds package entrypoint exports for runner + experiment loading + config helpers.
packages/agent-eval/src/experiments.ts	Implements local experiment directory scanning and dynamic module loading.
packages/agent-eval/src/experiments.test.ts	Adds tests covering local experiment listing and loading behaviors.
packages/agent-eval/src/config.ts	Re-exports/aliases core types and adds a typed experiment helper.
packages/agent-eval/src/cli.ts	Adds `--experiments` flag and wires CLI to local experiment loading.
packages/agent-eval/rolldown.config.ts	Switches to multi-entry output and adds `rolldown-plugin-dts` for `.d.ts` emission.
packages/agent-eval/README.md	Adds package-specific usage docs (CLI + typed authoring + programmatic API).
packages/agent-eval/package.json	Adds `bin`, `exports`, engine/files metadata, and runtime deps for npm publishing.
.github/workflows/experiment.yml	Updates CI workflow to run experiments via the built CLI and pass `--experiments`.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files not reviewed (1)

pnpm-lock.yaml: Generated file

Files reviewed: 12/13 changed files
Comments generated: 4
Review effort level: Low

+async function findExperiment(
+  id: string,
+  options: ExperimentSourceOptions = {},
+): Promise<ExperimentConfig | undefined> {
+  if (options.experimentsDirectory) {
+    const experiments = await getLocalExperimentEntries(options.experimentsDirectory)
+    return experiments.find(([name]) => name === id)?.[1]
+  }
+
+  if (existsSync(id)) {
+    return loadExperimentFile(id)
+  }
+
+  return findPackagedExperiment(id)
+}


+  test('finds an experiment from a local file path', async () => {
+    const directory = await createExperimentsDirectory()
+
+    await expect(findExperiment(path.join(directory, 'example.mjs'))).resolves.toEqual(
+      expect.objectContaining({name: 'Example'}),
+    )
+  })


+When authoring experiments outside of this repository, import the helper from
+`@primer/agent-eval/config` to keep the experiment config typed:
+
+```ts
+import {createExperiment} from '@primer/agent-eval/config'
+
+export const experiment = createExperiment({


+    "./config": {
+      "types": "./dist/config.d.ts",
+      "default": "./src/config.ts"
+    }


Copilot AI added 2 commits June 11, 2026 16:11

Add npm CLI package support

5892014

Finalize npm package metadata

37080cc

Copilot AI assigned Copilot and joshblack Jun 11, 2026

Copilot AI requested a review from joshblack June 11, 2026 16:18

Copilot created this pull request from a session on behalf of joshblack June 11, 2026 16:18 View session

Copilot finished work on behalf of joshblack June 11, 2026 16:18

joshblack reviewed Jun 30, 2026

View reviewed changes

Comment thread packages/agent-eval/src/config.ts Outdated

Comment thread packages/agent-eval/src/config.ts Outdated

Comment thread packages/agent-eval/package.json

Comment thread packages/agent-eval/package.json Outdated

Copilot started work on behalf of joshblack June 30, 2026 15:06 View session

Address agent-eval package review feedback

54c4dfa

Copilot finished work on behalf of joshblack June 30, 2026 15:15

Copilot AI requested a review from joshblack June 30, 2026 15:15

Copilot started work on behalf of joshblack June 30, 2026 15:22 View session

Merge origin/main into PR branch

a3d2db9

Copilot finished work on behalf of joshblack June 30, 2026 15:29

joshblack reviewed Jun 30, 2026

View reviewed changes

Copilot started work on behalf of joshblack June 30, 2026 16:08 View session

feat: support experiment file paths

b528b96

Copilot finished work on behalf of joshblack June 30, 2026 16:13

Copilot AI requested a review from joshblack June 30, 2026 16:13

Copilot started work on behalf of joshblack June 30, 2026 16:15 View session

Merge remote-tracking branch 'origin/main' into copilot/prepare-agent…

6d6a52c

…-eval-for-publishing # Conflicts: # packages/agent-eval/package.json

Copilot finished work on behalf of joshblack June 30, 2026 16:20

Copilot started work on behalf of joshblack June 30, 2026 16:24 View session

Merge remote-tracking branch 'origin/main' into copilot/prepare-agent…

cd9a758

…-eval-for-publishing # Conflicts: # packages/agent-eval/src/cli.ts # packages/agent-eval/src/config.ts

Copilot finished work on behalf of joshblack June 30, 2026 16:29

joshblack marked this pull request as ready for review June 30, 2026 16:31

Copilot AI review requested due to automatic review settings June 30, 2026 16:31

joshblack merged commit c163f63 into main Jun 30, 2026
7 checks passed

joshblack deleted the copilot/prepare-agent-eval-for-publishing branch June 30, 2026 16:31

Copilot started reviewing on behalf of joshblack June 30, 2026 16:32 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Prepare agent-eval for npm CLI usage#42

feat: Prepare agent-eval for npm CLI usage#42
joshblack merged 7 commits into
mainfrom
copilot/prepare-agent-eval-for-publishing

Copilot AI commented Jun 11, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joshblack commented Jun 30, 2026

Uh oh!

Copilot AI commented Jun 30, 2026

Uh oh!

joshblack Jun 30, 2026

Uh oh!

Copilot AI Jun 30, 2026

Uh oh!

joshblack commented Jun 30, 2026

Uh oh!

Copilot AI commented Jun 30, 2026

Uh oh!

joshblack commented Jun 30, 2026

Uh oh!

Copilot AI commented Jun 30, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Copilot AI commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

joshblack commented Jun 30, 2026

Uh oh!

Copilot AI commented Jun 30, 2026

Uh oh!

joshblack Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

joshblack commented Jun 30, 2026

Uh oh!

Copilot AI commented Jun 30, 2026

Uh oh!

joshblack commented Jun 30, 2026

Uh oh!

Copilot AI commented Jun 30, 2026

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jun 11, 2026 •

edited

Loading