Skip to content

feat: Prepare agent-eval for npm CLI usage#42

Draft
Copilot wants to merge 2 commits into
mainfrom
copilot/prepare-agent-eval-for-publishing
Draft

feat: Prepare agent-eval for npm CLI usage#42
Copilot wants to merge 2 commits into
mainfrom
copilot/prepare-agent-eval-for-publishing

Conversation

Copilot AI commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Prepares @primer/agent-eval for npm publishing as a runnable CLI, with support for running experiments from a caller-provided local folder while retaining typed experiment authoring.

  • CLI package metadata

    • Adds agent-eval as the package binary.
    • Publishes built CLI output, config types, and Node engine metadata.
    • Bundles internal workspace code needed by the CLI and declares external runtime dependencies.
  • Local experiment loading

    • Adds --experiments to load experiment files from a local directory.
    • Supports experiment named exports and default exports.
    • Falls back to packaged experiment lookup when no local directory is provided.
  • Typed config authoring

    • Adds defineExperimentConfig from @primer/agent-eval/config.
import {defineExperimentConfig} from '@primer/agent-eval/config'

export const experiment = defineExperimentConfig({
  name: 'Example experiment',
  description: 'Compare treatment behavior',
  models: ['gpt-5.5'],
  evals: ['001-agent-uses-button-from-primer'],
  treatments: [],
})
COPILOT_GITHUB_TOKEN=... agent-eval --experiments ./experiments --experiment example

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants