CI: Wire vally eval invocation to a curated medium-size suite via .vally.yaml

[wbreza]

Context

Follow-up to #1912, which migrated Waza eval references to Vally. That PR deliberately wired .github/workflows/eval.yml to a single hardcoded --eval-spec path to keep scope small and get CI green.

Goal

Replace the single --eval-spec flag with --suite <name> driven by .vally.yaml suite config, so CI runs a curated set of eval specs (not just one).

Target

• Inner loop (local dev): vally-cli eval --eval-spec <one> — fast iteration on a single spec.
• Medium loop (CI): vally-cli eval --suite pr — curated set targeting 5–10 minute wall-clock runtime.
• Outer loop (nightly / manual): vally-cli eval --suite full or individual dispatches — full coverage.

Prereqs

• Tag hygiene pass (completed in ci(eval): migrate from Waza to Vally eval framework #1912): cost: free now reflects reality, so suite filters behave correctly.
• Define pr and full suites in .vally.yaml with the right tag filters.

Acceptance

• .vally.yaml defines pr and full suites with explicit tag filters
• .github/workflows/eval.yml Run evaluations step uses --suite pr
• CI wall-clock time for the pr suite is 5–10 minutes
• Results artifact still uploads at the standard path

Related

• PR ci(eval): migrate from Waza to Vally eval framework #1912 — initial Waza migration groundwork
• Thread: ci(eval): migrate from Waza to Vally eval framework #1912 (comment) (jongio's original comment)

[github-actions]

Thanks for filing this, @wbreza! 🎯

Labeled as enhancement since this improves the CI eval pipeline, and github_actions since it involves updating the workflow file and .vally.yaml suite config. A coding agent has been assigned (assign-to-copilot) to implement the .vally.yaml pr/full suite definitions and wire --suite pr into the eval workflow step.

Generated by Issue Triage for issue #1920 · ● 123.5K · ◷