Skip to content

[TypeSpec Authoring][Benchmark] Run evals with 2 trials #16188

Draft
haolingdong-msft wants to merge 3 commits into
mainfrom
haolingdong-msft/eval-multi-trial
Draft

[TypeSpec Authoring][Benchmark] Run evals with 2 trials #16188
haolingdong-msft wants to merge 3 commits into
mainfrom
haolingdong-msft/eval-multi-trial

Conversation

@haolingdong-msft

Copy link
Copy Markdown
Member

Summary

Adds multi-trial support to the azure-typespec-author skill evaluation suite using Vally's native config.runs mechanism, and updates the result reporter to aggregate trials with pass@k semantics and surface each trial's per-case status.

Motivation

Single-run evals are flaky: a transient miss on one stimulus drops the suite pass rate and produces noisy red builds. Running each stimulus twice and judging pass@k separates real capability failures (fail every trial) from flakiness (passes at least once).

Changes

  • config.runs: 1 → 2 across all 28 evals/*.eval.yaml and 3 suites/{forced,trigger,no-skill}.eval.yaml specs — every stimulus now runs twice per evaluation.
  • scripts/report-eval-results.sh updated to:
    • Group the now-multiple results.jsonl records per stimulus by stimulusName (Vally emits one record per trial when runs > 1).
    • Apply pass@k: a stimulus passes if any trial passes; only stimuli that fail every trial raise an ADO ##vso[task.logissue type=error].
    • Print a per-trial headline, flag [FLAKY] stimuli (pass some-but-not-all trials), and list each case's status across trials.

Sample output

--- Suite versioning trial results ---
trial 1 (10/11)
trial 2 (9/11)
fail in both trials: 1

Pass@k: 10/11 (90%), 1 flaky
  010 [PASS] trial1:PASS trial2:FAIL tool-calls:pass
  011 [FAIL] trial1:FAIL trial2:FAIL

Validation

  • All 32 eval/suite specs confirmed at runs: 2 (0 leftover runs: 1).
  • bash -n clean on the report script.
  • Report logic verified against synthetic 2-trial data: flaky → PASS (flagged), fail-both → FAILED + error, correct trial counts and pass@k rate.

⚠️ Follow-up (not in this PR)

2 trials roughly doubles runtime. The per-suite TimeoutMinutes in eng/pipelines/azure-typespec-author-benchmark.yml (versioning 50, armtemplate 40, lro 25, decorators 15, warning 25) may need bumping.

haolingdong-msft and others added 2 commits June 26, 2026 15:45
Set Vally config.runs to 2 for all azure-typespec-author eval and suite
specs so each stimulus runs twice. Update report-eval-results.sh to group
trial records by stimulus and apply pass@k: a stimulus passes if any trial
passes, flaky stimuli (some-but-not-all trials pass) are flagged, and only
stimuli that fail every trial are logged as errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@haolingdong-msft haolingdong-msft changed the title [TypeSpec Authoring][Benchmark] Run evals with 2 trials + per-trial pass@k reporting [TypeSpec Authoring][Benchmark] Run evals with 2 trials Jun 26, 2026
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant