Skip to content

Commit ec1d42e

Browse files
Move Vally eval suite to repo-root azsdk-evals (#48095)
Co-authored-by: helen229 <gaoh@microsoft.com>
1 parent 694c4f4 commit ec1d42e

6 files changed

Lines changed: 562 additions & 227 deletions

File tree

eng/common/pipelines/live-eval.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ extends:
2525
# Shared mock/live builder; select the live tier (real Cli MCP).
2626
mcpSetupTemplate: /eng/common/pipelines/templates/steps/eval-mcp-setup.yml
2727
TestType: live
28-
vallyRoot: tools/azsdk-cli/Azure.Sdk.Tools.Vally
28+
vallyRoot: azsdk-evals
2929
evalGlobs:
3030
- 'evals/workflow-scenarios/live/*.eval.yaml'
3131
# Run each shard under AzureCLI@2 so the real MCP's DevOps calls are authenticated.

eng/common/pipelines/workflow-eval.yml

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
# Hermetic workflow-scenario eval CI: runs the unit-tool + mock workflow-scenario evals in
2-
# tools/azsdk-cli/Azure.Sdk.Tools.Vally against the mock MCP (live tier runs in live-eval.yml).
2+
# azsdk-evals against the mock MCP (live tier runs in live-eval.yml).
33

44
trigger:
55
branches:
66
include:
77
- main
88
paths:
99
include:
10-
- tools/azsdk-cli/Azure.Sdk.Tools.Vally/**
10+
- azsdk-evals/**
1111
- tools/azsdk-cli/Azure.Sdk.Tools.Mock/**
1212
# Cli tool catalog + skills feed the eval results, so changes here must retrigger.
1313
- tools/azsdk-cli/Azure.Sdk.Tools.Cli/**
@@ -25,7 +25,7 @@ pr:
2525
- main
2626
paths:
2727
include:
28-
- tools/azsdk-cli/Azure.Sdk.Tools.Vally/**
28+
- azsdk-evals/**
2929
- tools/azsdk-cli/Azure.Sdk.Tools.Mock/**
3030
- tools/azsdk-cli/Azure.Sdk.Tools.Cli/**
3131
- .github/skills/**
@@ -48,7 +48,7 @@ extends:
4848
# Shared mock/live builder; select the mock tier.
4949
mcpSetupTemplate: /eng/common/pipelines/templates/steps/eval-mcp-setup.yml
5050
TestType: mock
51-
vallyRoot: tools/azsdk-cli/Azure.Sdk.Tools.Vally
51+
vallyRoot: azsdk-evals
5252
# Per-shard job timeout (report-only tier).
5353
shardTimeoutInMinutes: 20
5454
# Pass-rate gate for `vally eval`. This tier is report-only (failOnFailedTests defaults false),

eng/common/scripts/allow-relative-links.txt

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,5 @@
99
AGENTS.md
1010
# Allow relative links for all files under the eng folder (e.g. engineering system scripts and templates).
1111
eng/**
12+
# Allow relative links for the top-level eval suite (azsdk-evals is a cross-cutting asset at repo root).
13+
azsdk-evals/**

0 commit comments

Comments
 (0)