Skip to content

Commit febb7c4

Browse files
vadam5Virginia Adamsagronskiy
authored
feat: [GDPval-AA v2 Updates 1 / n] - GDPval Multi-Reference Model Support (#1663)
Adds multi-reference model support and Bradley-Terry MLE ELO computation with multiple reference models in order to directly match AA's ELO score calcualtion algorithm: [GDPval-AA v2](https://artificialanalysis.ai/methodology/intelligence-benchmarking#gdpval-aa) > After collecting pairwise rankings, we fit them to a Bradley-Terry model via maximum likelihood estimation and compute confidence intervals using the sandwich estimator to establish our final Elo metric This is one of many changes needed to match AA's v2 benchmark. --------- Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com> Co-authored-by: Virginia Adams <vadams@oci-hsg-cs-001-vscode-02.cm.cluster> Co-authored-by: Alex Gronskiy <agronskiy@nvidia.com>
1 parent fdd6130 commit febb7c4

5 files changed

Lines changed: 566 additions & 54 deletions

File tree

benchmarks/gdpval/config.yaml

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,15 @@
1010
# Comparison mode (pairwise ELO vs a reference model's deliverables):
1111
# ++gdpval_resources_server.resources_servers.gdpval.reward_mode=comparison \
1212
# ++gdpval_resources_server.resources_servers.gdpval.reference_deliverables_dir=/path/to/fork
13+
#
14+
# Multi-reference comparison mode (anchored Bradley-Terry MLE ELO over N
15+
# references — each reference is judged separately and the eval model's rating
16+
# is fit globally, with per-reference + total win stats reported):
17+
# ++gdpval_resources_server.resources_servers.gdpval.reward_mode=comparison \
18+
# ++gdpval_resources_server.resources_servers.gdpval.reference_models.kimi.deliverables_dir=/gdpval/refs/kimi \
19+
# ++gdpval_resources_server.resources_servers.gdpval.reference_models.kimi.elo=1290 \
20+
# ++gdpval_resources_server.resources_servers.gdpval.reference_models.gpt5.deliverables_dir=/gdpval/refs/gpt5 \
21+
# ++gdpval_resources_server.resources_servers.gdpval.reference_models.gpt5.elo=1320
1322

1423
# Judge model — proxy to NVIDIA inference API for Gemini 3.1 Pro.
1524
gdpval_judge_model:
@@ -28,9 +37,14 @@ gdpval_resources_server:
2837
domain: other
2938
verified: false
3039
reward_mode: rubric
40+
# Single-reference (legacy) comparison fields.
3141
reference_deliverables_dir: null
32-
num_comparison_trials: 4
3342
reference_elo: 1000.0
43+
# Multi-reference comparison: map of reference id -> {deliverables_dir, elo}.
44+
# When set, the eval model's ELO is fit via anchored Bradley-Terry MLE
45+
# across all references and per-reference win stats are reported.
46+
reference_models: {}
47+
num_comparison_trials: 4
3448
preconvert_office_to_pdf: true
3549
preconvert_max_concurrent: 1
3650
judge_model_server:

0 commit comments

Comments
 (0)