Skip to content

Eval: overhead time metric goes negative #53

@olaservo

Description

@olaservo

Problem

The "Overhead Time" metric in eval output frequently shows negative values:

SDK Duration:           3.77s
API Call Time:          11.71s
Overhead Time:          -7938ms

The formula is sdkDurationMs - apiDurationMs (metrics.ts lines 266, 293), which assumes API time is a subset of SDK time. But the Agent SDK can make parallel API calls across multiple models, so aggregate API time exceeds wall-clock SDK time.

Suggestion

Options:

  1. Remove the overhead metric — it's not meaningful with parallel execution
  2. Rename to "API vs Wall-clock Delta" and clarify in the display that negative means parallel execution
  3. Clamp to zero with a note: `Math.max(0, sdkDuration - apiDuration)`

Files

  • `evals/lib/metrics.ts` (lines 266, 293)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions