support running X eval cases against Y traces

[peterj]

Today, for each trace the runner calls _find_expected_invocations once. That picks at either the first case whose first user message matches the trace's first user message. Or, if there's no match, it's a fallback to the first case. All metrics for that trace then run once, all against that single gold that was picked out. There's no lopp currently that would allow us to score the same trace against eval case A, then against against eval case B, etc.

Example
Eval set has two cases: Case A (gold conversation starts with user: “Book a flight to NYC”) and Case B (starts with “Cancel my order”).

You submit one trace whose first user turn is “Book a flight to NYC”. The heuristic selects Case A; tool_trajectory_avg_score / response_match_score (etc.) compare the trace to A’s gold only. Case B is not scored against this trace in that evaluation — you do not get “trace vs B” results unless you change the pipeline (separate product / likely runner changes).

Desired state
With an input of X eval cases and Y traces, we should run and score each case against each trace.

[mesutoezdil]

Hey Peter, I looked at the code. 3 points from my side:

1- add eval_case_id to TraceResult so each result knows which case it came from

2- Change _evaluate_trace to take expected_invocations directly instead of the full eval set.. move the case selection out of that function

3- In run_evaluation_from_traces, expand the task list from one task per trace to one task per (trace, case) pair

Now: Y trace x 1 case = Y task
New: Y trace x X case = X*Y task

Isnt't it?