[TypeSpec Authoring][Benchmark] Run evals with 2 trials #16188
Draft
haolingdong-msft wants to merge 3 commits into
Draft
[TypeSpec Authoring][Benchmark] Run evals with 2 trials #16188haolingdong-msft wants to merge 3 commits into
haolingdong-msft wants to merge 3 commits into
Conversation
Set Vally config.runs to 2 for all azure-typespec-author eval and suite specs so each stimulus runs twice. Update report-eval-results.sh to group trial records by stimulus and apply pass@k: a stimulus passes if any trial passes, flaky stimuli (some-but-not-all trials pass) are flagged, and only stimuli that fail every trial are logged as errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This was referenced Jul 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds multi-trial support to the
azure-typespec-authorskill evaluation suite using Vally's nativeconfig.runsmechanism, and updates the result reporter to aggregate trials with pass@k semantics and surface each trial's per-case status.Motivation
Single-run evals are flaky: a transient miss on one stimulus drops the suite pass rate and produces noisy red builds. Running each stimulus twice and judging pass@k separates real capability failures (fail every trial) from flakiness (passes at least once).
Changes
config.runs: 1 → 2across all 28evals/*.eval.yamland 3suites/{forced,trigger,no-skill}.eval.yamlspecs — every stimulus now runs twice per evaluation.scripts/report-eval-results.shupdated to:results.jsonlrecords per stimulus bystimulusName(Vally emits one record per trial whenruns > 1).##vso[task.logissue type=error].[FLAKY]stimuli (pass some-but-not-all trials), and list each case's status across trials.Sample output
Validation
runs: 2(0 leftoverruns: 1).bash -nclean on the report script.2 trials roughly doubles runtime. The per-suite
TimeoutMinutesineng/pipelines/azure-typespec-author-benchmark.yml(versioning 50, armtemplate 40, lro 25, decorators 15, warning 25) may need bumping.