Proposal: a multilingual judge-ELO leaderboard

[kargibora]

This issue is related to the discussion we had, in which we want to create a (i) leaderboard that shows (ii) multilingaul performances of a pool of model in which (iii) can submit a new model quite easily.

Summary

Add a leaderboard to JudgeArena where anyone can submit a model with one command and get it ranked against a shared pool of models across many languages, judged by an LLM. The ranking uses soft-ELO calibrated against human preferences (can be changed easily with a flag), so a single judge instrument produces ratings that track human-ELO closely. The result will be published as a Hugging Face dataset and rendered by a modern Gradio Space.

Motivation

We already have the pieces — generation, LLM-as-judge, ELO estimation — but no standing, comparable ranking. Today comparing two models means running a bespoke ELO job and eyeballing the numbers; there's no frozen reference, no multilingual coverage, and no shared place to see where a new model lands. A leaderboard turns "run an eval" into "submit a model and see its rank," and makes results comparable across submissions and over time.

The idea

One command to submit a model. It is evaluated against a frozen pool of multilingual prompts (target: ≥15 languages, ~100 prompts each), battled against the existing models with an LLM judge (vLLM or OpenRouter), and the leaderboard updates with its rank, per-language breakdown, and diagnostics.

How it works

• Frozen panel. We curate a fixed set of judge-scored anchor battles (a battles.parquet + a panel.json describing it) from an arena dataset (e.g. LMArena/Comparia). This panel is the shared measuring stick — every submission is scored against the same instrument, so all submissions are comparable through the shared anchors (standard arena methodology; submissions don't battle each other by default).
• Quality gating. Per-language judge↔human agreement (Cohen's κ) decides which languages are trustworthy enough to include; low-κ languages are dropped or flagged.
• Publish + render. A publish step computes a self-contained bundle (rankings, per-language tables, calibration points, κ, win-rate and head-to-head matrices) and uploads it as a HF dataset; a thin Gradio Space renders the precomputed bundle (no eval stack imported into the Space). The rendering plots can be also a seperate package and we can push this to a seperate repo.

Scope (sub-projects)

1. Eval contract & panel — curation, frozen panel artifact, κ gating, soft-ELO calibration, ResultRecord.
2. CLIs — curate (build panel), submit (score a model, with --tag for repeated runs of the same model), board (render).
3. HF publish + Gradio Space — the bundle format, the dataset upload, and a thin, modern Space.
4. Insights & modern UI — human-ELO vs judge-ELO calibration plot with CI, per-language κ bar, win-rate-by-language heatmap, head-to-head model×model matrix, score distributions, examples viewer.

What's prototyped already

Most of is built on the feat/live-leaderboard (library + CLIs) and feat/leaderboard-hf-space (publish + Space + insights + head-to-head) branches, validated end-to-end with a small deepseek-v3.2 / OpenRouter smoke run. I will try to extend this to multilingual results as a test after getting cluster access

Open questions

• Panel source & size — which arena dataset(s), how many languages and prompts per language, and the roster selection rule (e.g. models with enough human annotations across enough languages).
• Comparability over time — how to handle panel versioning so older submissions stay comparable (panel hash / version gating).
• Direct submission-vs-submission — should we offer an opt-in compare flow that judges two submissions head-to-head (their completions already exist on the same frozen prompts; only the judging is missing), or rely solely on transitive ELO through the anchors?

Examples

(1) Submit a new model to arena

uv run judgearena-submit --panel-dir panels/deepseek-test-v1 --model OpenRouter/deepseek/deepseek-chat-v3.1 --tag "[test]"

Note: This does not submit a job from an external source. It should be done on the folder in which we keep the leaderboard (hugging-face space on later terms).

(2) Render it to the CLI

uv run judgearena-board --panel-dir panels/deepseek-test-v1 --format markdown
| Rank | Model | ELO | CI | n |
|---|---|---|---|---|
| 1 | gemini-2.5-pro | 1029.3 |  | 52 |
| 2 | gemini-2.5-flash | 1011.8 |  | 56 |
| 3 | OpenRouter/deepseek/deepseek-chat-v3.1 #[test] * | 993.3 | [983, 1002] | 100 |
| 4 | OpenRouter/deepseek/deepseek-v3.2 * | 992.6 | [983, 1002] | 100 |
| 5 | qwen3-235b-a22b-no-thinking | 989.5 |  | 46 |
| 6 | claude-opus-4-20250514 | 969.4 |  | 46 |

(3) Run the Leaderboard locally (before we push to hface)

 uv run judgearena-publish --panel-dir panels/deepseek-test-v1 --out /tmp/lb --dry-run
  uv run python space/app.py --local /tmp/lb

[geoalgo]

Thank you for opening the issue and apologies for not commenting earlier.

Panel source & size — which arena dataset(s), how many languages and prompts per language, and the roster selection rule (e.g. models with enough human annotations across enough languages).

We need to take the union of all datasets to have enough data for all languages. In addition I would suggest to have a cutoff date (like May 2026) so that we can evaluate easily fresh data (eg the data being populated in comparia which would not have been seen. That being said, we would also have lots of battles (for medium resource languages) which would be left out.

Comparability over time — how to handle panel versioning so older submissions stay comparable (panel hash / version gating).

We should version the metadata.json listing all the CLI and package version for reproducibility.

Direct submission-vs-submission — should we offer an opt-in compare flow that judges two submissions head-to-head (their completions already exist on the same frozen prompts; only the judging is missing), or rely solely on transitive ELO through the anchors?

I would first consider the simplest case to rely on transitive ELO.

Note: This does not submit a job from an external source. It should be done on the folder in which we keep the leaderboard (hugging-face space on later terms).

What do you mean here? I think having the leaderboard as another package would make sense to me.