Skip to content

Add suite-level runner for scenario and trace directories #85

Description

@mertsatilmaz

Context

The harness can run one scenario at a time. Teams will quickly want a suite mode that runs a directory of scenarios against corresponding traces or targets and produces one aggregate result.

Acceptance criteria

  • Design a minimal suite command or flag that keeps single-scenario behavior intact.
  • Support trace-file based suites first.
  • Emit per-scenario result JSON and an aggregate summary.
  • Exit behavior should compose with the CI fail-on-result option.
  • Add tests for partial failures and missing trace mappings.
  • Document expected directory conventions.

Useful files

  • src/agent_harness/cli.py
  • src/agent_harness/runner.py
  • tests/test_cli.py
  • docs/ci-github-actions.md

Metadata

Metadata

Assignees

No one assigned

    Labels

    clienhancementNew feature or requesthelp wantedExtra attention is neededtestingTest coverage, fixtures, test infrastructure, and regression tests.

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions