Commit febb7c4
feat: [GDPval-AA v2 Updates 1 / n] - GDPval Multi-Reference Model Support (#1663)
Adds multi-reference model support and Bradley-Terry MLE ELO computation
with multiple reference models in order to directly match AA's ELO score
calcualtion algorithm: [GDPval-AA
v2](https://artificialanalysis.ai/methodology/intelligence-benchmarking#gdpval-aa)
> After collecting pairwise rankings, we fit them to a Bradley-Terry
model via maximum likelihood estimation and compute confidence intervals
using the sandwich estimator to establish our final Elo metric
This is one of many changes needed to match AA's v2 benchmark.
---------
Signed-off-by: Alex Gronskiy <agronskiy@nvidia.com>
Co-authored-by: Virginia Adams <vadams@oci-hsg-cs-001-vscode-02.cm.cluster>
Co-authored-by: Alex Gronskiy <agronskiy@nvidia.com>1 parent fdd6130 commit febb7c4
5 files changed
Lines changed: 566 additions & 54 deletions
File tree
- benchmarks/gdpval
- resources_servers/gdpval
- configs
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
13 | 22 | | |
14 | 23 | | |
15 | 24 | | |
| |||
28 | 37 | | |
29 | 38 | | |
30 | 39 | | |
| 40 | + | |
31 | 41 | | |
32 | | - | |
33 | 42 | | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
34 | 48 | | |
35 | 49 | | |
36 | 50 | | |
| |||
0 commit comments