You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+2-3Lines changed: 2 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -38,7 +38,7 @@ Leaderboard/evaluation data is split-up into files by individual model, and data
38
38
### TL;DR How to successfully submit
39
39
40
40
1. Data must conform to [`eval.schema.json`](eval.schema.json) (current version: `0.2.2`)
41
-
2.Validation runs automatically on every PR via [`.github/workflows/validate-data.yml`](.github/workflows/validate-data.yml)
41
+
2.The validation pipeline will automatically verify the data submitted in the pull request, but can also be manually triggered by typing ```/eee validate changed``` in a comment on the HF PR.
42
42
3. An EvalEval member will review and merge your submission
43
43
44
44
### PR Naming Convention
@@ -71,8 +71,7 @@ Note: Each file can contain multiple individual results related to one model. Se
71
71
2. For each model, use the Hugging Face (`developer_name/model_name`) naming convention to create a 2-tier folder structure.
72
72
3. Add a JSON file with results for each model and name it `{uuid}.json`.
73
73
4.[Optional] Include a [`utils/`](utils/) folder in your benchmark name folder with any scripts used to generate the data (see e.g. [`utils/global-mmlu-lite/adapter.py`](utils/global-mmlu-lite/adapter.py)).
74
-
5.[Validate] Validation runs automatically via [`validate-data.yml`](.github/workflows/validate-data.yml) using the package-local validator to check JSON files against the schema before merging.
75
-
6.[Submit] Two ways to submit your evaluation data:
74
+
5.[Submit] Two ways to submit your evaluation data:
76
75
-**Option A: Drag & drop via Hugging Face** — Go to [evaleval/EEE_datastore](https://huggingface.co/datasets/evaleval/EEE_datastore) → click "Files and versions" → "Contribute" → "Upload files" → drag and drop your data → select "Open as a pull request to the main branch". See [step-by-step screenshots](https://docs.google.com/document/d/1dxTQF8ncGCzaAOIj0RX7E9Hg4THmUBzezDOYUp_XdCY/edit?usp=sharing).
77
76
-**Option B: Clone & PR** — Clone the [Hugging Face repository](https://huggingface.co/datasets/evaleval/EEE_datastore), add your data under `data/`, and open a pull request
0 commit comments