fix(evals): install YAML score dependency

Author: adibarra

.github/workflows/benchmark-tmpl.yml

@@ -337,7 +337,10 @@ jobs:
 
       - name: Verify eval scores
         if: ${{ (success() || failure()) && inputs.eval-only && (inputs.eval-limit == '' || inputs.eval-limit == 'full' || inputs.eval-limit == '0') }}
-        run: python3 utils/evals/validate_scores.py
+        run: |
+          python3 -m venv "$RUNNER_TEMP/eval-validation-venv"
+          "$RUNNER_TEMP/eval-validation-venv/bin/pip" install 'PyYAML==6.0.3'
+          "$RUNNER_TEMP/eval-validation-venv/bin/python" utils/evals/validate_scores.py
 
       - name: Cleanup eval outputs (post-upload)
         if: ${{ always() && (env.RUN_EVAL == 'true' || inputs.eval-only) }}

utils/evals/EVALS.md

@@ -156,7 +156,7 @@ cat ./evals/agg_eval_all.json | jq '[.[] | select(.hw == "B200")]'
 | `EVAL_LIMIT` | empty | Limit eval to first N instances (smoke tests); empty = full set |
 
 ### Score validation
-`utils/evals/validate_scores.py` checks full-split eval results against thresholds in `utils/evals/thresholds.json`. The JSON format keeps validation independent of optional PyYAML installations on runner hosts. First-N smoke slices skip the quality gate because their scores are not statistically representative. Validation runs after artifact upload so full-run results are preserved even if the gate fails.
+`utils/evals/validate_scores.py` checks full-split eval results against thresholds in `utils/evals/thresholds.yaml`. The workflow installs the pinned PyYAML dependency before validation. First-N smoke slices skip the quality gate because their scores are not statistically representative. Validation runs after artifact upload so full-run results are preserved even if the gate fails.
 
 ### Adding a new eval task
 
@@ -214,7 +214,7 @@ append_lm_eval_summary
   is derived from the YAML's `dataset_path` so generation and scoring can't diverge;
   `SWEBENCH_DATASET`, if set, must match it (mismatch fails fast).
 - Scoring runs on Modal remote sandboxes in CI (`SWEBENCH_USE_MODAL=true`, no Docker on the GPU
-  nodes); local Docker scoring needs ~120 GB disk. The `thresholds.json` gate is `0.50`, calibrated
+  nodes); local Docker scoring needs ~120 GB disk. The `thresholds.yaml` gate is `0.50`, calibrated
   from full-split runs (54%); historical 50-instance slices scored 62–76%.
 
 ## Task files

utils/evals/thresholds.json

@@ -0,0 +1,25 @@
+# Model thresholds override task defaults.
+default:
+  aime26: 0.30
+  gsm8k: 0.90
+  gpqa_diamond_cot_n_shot: 0.30
+  swebench_lite: 0.50
+models:
+  dsr1:
+    gsm8k: 0.91
+  dsv4:
+    aime26: 0.30
+    gpqa_diamond_cot_n_shot: 0.65
+    gsm8k: 0.91
+  glm5:
+    gsm8k: 0.94
+  glm5.1:
+    gsm8k: 0.93
+  gptoss:
+    gsm8k: 0.91
+  kimik2.5:
+    gsm8k: 0.90
+  minimaxm2.5:
+    gsm8k: 0.92
+  qwen3.5:
+    gsm8k: 0.94

utils/evals/thresholds.yaml

@@ -186,7 +186,7 @@ def main() -> int:
     )
     parser.add_argument(
         "--thresholds", default=None,
-        help="Path to thresholds config, YAML or JSON (default: utils/evals/thresholds.json)",
+        help="Path to thresholds config, YAML or JSON (default: utils/evals/thresholds.yaml)",
     )
     parser.add_argument(
         "--meta-env", default="meta_env.json",
@@ -229,7 +229,7 @@ def main() -> int:
     config = {"default": {}, "models": {}}
     thresholds_path = args.thresholds
     if thresholds_path is None:
-        default_path = Path(__file__).parent / "thresholds.json"
+        default_path = Path(__file__).parent / "thresholds.yaml"
         if default_path.exists():
             thresholds_path = str(default_path)
     if thresholds_path: