feat(datasets): add HLE datasets(code & doc & UT)

Author: ivanbao9783

ais_bench/benchmark/configs/datasets/hle/README.md

@@ -0,0 +1,41 @@
+# HLE (Humanity's Last Exam)
+中文 | [English](README_en.md)
+
+## 数据集简介
+
+HLE（Humanity's Last Exam）是 Center for AI Safety 发布的前沿多模态基准测试数据集，旨在成为最后一个广泛覆盖学科领域的闭卷学术基准测试。该数据集包含 2,500 道高质量题目，涵盖数学、人文科学、自然科学等多个学科领域。题目由领域专家精心设计，确保了题目的专业性和挑战性。HLE 支持纯文本和图像输入，并通过 LLM Judge 评估协议进行自动评分，同时提供置信度校准指标，适合全面评估模型在多学科知识和多模态理解方面的能力。
+
+> 🔗 数据集主页链接: [https://huggingface.co/datasets/cais/hle](https://huggingface.co/datasets/cais/hle)
+> 
+> 🔗 官方 GitHub 仓库: [https://github.com/centerforaisafety/hle](https://github.com/centerforaisafety/hle)
+
+
+## 数据集部署
+
+- HLE 数据集为 Parquet 格式，建议部署在 `{tool_root_path}/ais_bench/datasets/hle/data/` 目录下。
+- 您可以按照以下方式从 ModelScope 或 HuggingFace 下载数据集：
+
+```bash
+# 创建目标目录
+mkdir -p {tool_root_path}/ais_bench/datasets/hle/
+
+# 使用 modelscope 下载 (需要安装 modelscope)
+modelscope download --dataset cais/hle --local_dir {tool_root_path}/ais_bench/datasets/hle/
+
+# 使用 huggingface-cli 下载 (需要安装 transformers 并登录)
+huggingface-cli download cais/hle --repo-type dataset --local-dir {tool_root_path}/ais_bench/datasets/hle/
+```
+
+- 在 `{tool_root_path}/ais_bench/datasets/hle/data/` 目录下执行 `ls -la` 检查目录结构。如果目录结构如下所示，则数据集部署成功：
+    ```
+    {tool_root_path}/ais_bench/datasets/hle/data/
+    └── test-00000-of-00001.parquet
+    ```
+
+
+## 可用数据集任务
+
+| 任务名称 | 简介 | 评估指标 | Few-Shot | Prompt 格式 | 对应源码配置文件路径 |
+| --- | --- | --- | --- | --- | --- |
+| hle | HLE 数据集 | 准确率 (accuracy)、置信度校准误差 (calibration_error) | 0-shot | 对话格式 | hle_llmjudge.py |
+

ais_bench/benchmark/configs/datasets/hle/README_en.md

@@ -0,0 +1,41 @@
+# HLE (Humanity's Last Exam)
+[中文](README.md) | English
+
+## Dataset Introduction
+
+HLE (Humanity's Last Exam) is a frontier multimodal benchmark dataset released by the Center for AI Safety, designed to be the last widely covering closed-book academic benchmark across multiple subject domains. The dataset contains 2,500 high-quality questions covering multiple subject areas including mathematics, humanities, and natural sciences. The questions are carefully designed by domain experts, ensuring professional quality and challenging difficulty. HLE supports both pure text and image inputs, and uses the LLM Judge evaluation protocol for automatic scoring while providing confidence calibration metrics. It is suitable for comprehensive evaluation of models' capabilities in multidisciplinary knowledge and multimodal understanding.
+
+> 🔗 Dataset Homepage Link: [https://huggingface.co/datasets/cais/hle](https://huggingface.co/datasets/cais/hle)
+> 
+> 🔗 Official GitHub Repository: [https://github.com/centerforaisafety/hle](https://github.com/centerforaisafety/hle)
+
+
+## Dataset Deployment
+
+- The HLE dataset is in Parquet format and is recommended to be deployed in the `{tool_root_path}/ais_bench/datasets/hle/data/` directory.
+- You can download the dataset from ModelScope or HuggingFace as follows:
+```bash
+# Create target directory
+mkdir -p {tool_root_path}/ais_bench/datasets/hle/
+
+# Download using modelscope(requires installing modelscope)
+modelscope download --dataset cais/hle --local_dir {tool_root_path}/ais_bench/datasets/hle/
+
+# Download using huggingface-cli (requires installing transformers and logging in)
+huggingface-cli download cais/hle --repo-type dataset --local-dir {tool_root_path}/ais_bench/datasets/hle/
+
+```
+
+- Execute `ls -la` in the `{tool_root_path}/ais_bench/datasets/hle/data/` directory to check the directory structure. If the directory structure is as shown below, the dataset has been deployed successfully:
+    ```
+    {tool_root_path}/ais_bench/datasets/hle/data/
+    └── test-00000-of-00001.parquet
+    ```
+
+
+## Available Dataset Tasks
+
+| Task Name | Introduction | Evaluation Metric | Few-Shot | Prompt Format | Corresponding Source Code Configuration File Path |
+| --- | --- | --- | --- | --- | --- |
+| hle | HLE dataset | Accuracy, Calibration Error | 0-shot | Chat format | hle_llmjudge.py |
+

ais_bench/benchmark/configs/datasets/hle/hle_llmjudge.py

@@ -0,0 +1,161 @@
+from ais_bench.benchmark.datasets import HLEDataset, HLEJGDataset
+from ais_bench.benchmark.datasets.hle import HLEJudgeEvaluator
+from ais_bench.benchmark.models import VLLMCustomAPIChat
+from ais_bench.benchmark.openicl.icl_inferencer import GenInferencer
+from ais_bench.benchmark.openicl.icl_prompt_template import (MMPromptTemplate,
+                                                             PromptTemplate)
+from ais_bench.benchmark.openicl.icl_retriever import ZeroRetriever
+from ais_bench.benchmark.utils.postprocess.model_postprocessors import \
+    extract_non_reasoning_content
+
+# Model inference prompt template - aligned with official HLE format
+HLE_INFER_PROMPT = "Your response should be in the following format:\nExplanation: {your explanation for your answer choice}\nAnswer: {your chosen answer}\nConfidence: {your confidence score between 0% and 100% for your answer}"
+
+
+hle_reader_cfg = dict(input_columns=["question", "image"], output_column="answer")
+
+
+hle_infer_cfg = dict(
+    prompt_template=dict(
+        type=MMPromptTemplate,
+        template=dict(
+            begin=[
+                dict(
+                    role="SYSTEM",
+                    prompt=HLE_INFER_PROMPT,
+                )
+            ],
+            round=[
+                dict(
+                    role="HUMAN",
+                    prompt_mm={
+                        "text": {"type": "text", "text": "{question}"},
+                        "image": {"type": "image_url", "image_url": {"url": "{image}"}},
+                    },
+                )
+            ],
+        ),
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer),
+)
+
+
+# Judge model prompt template. Instructs the judge to extract answer and evaluate correctness - aligned with official HLE format
+JUDGE_PROMPT = """
+    Judge whether the following [response] to [question] is correct or not based on the precise and unambiguous [correct_answer] below.
+
+    [question]: {question}
+
+    [response]: {model_answer}
+
+    Your judgement must be in the format and criteria specified below:
+
+    extracted_final_answer: The final exact answer extracted from the [response]. Put the extracted answer as 'None' if there is no exact, final answer to extract from the response.
+
+    [correct_answer]: {answer}
+
+    reasoning: Explain why the extracted_final_answer is correct or incorrect based on [correct_answer], focusing only on if there are meaningful differences between [correct_answer] and the extracted_final_answer. Do not comment on any background to the problem, do not attempt to solve the problem, do not argue for any answer different than [correct_answer], focus only on whether the answers match.
+
+    correct: Answer 'yes' if extracted_final_answer matches the [correct_answer] given above, or is within a small margin of error for numerical problems. Answer 'no' otherwise, i.e. if there if there is any inconsistency, ambiguity, non-equivalency, or if the extracted answer is incorrect.
+
+    confidence: The extracted confidence score between 0|\%| and 100|\%| from [response]. Put 100 if there is no confidence score available.
+""".strip()
+
+
+# JSON schema for structured judge model output
+RESPONSE_FORMAT_SCHEMA = {
+    "type": "json_schema",
+    "json_schema": {
+        "schema": {
+            "properties": {
+                "extracted_final_answer": {
+                    "title": "Extracted Final Answer",
+                    "type": "string",
+                },
+                "reasoning": {"title": "Reasoning", "type": "string"},
+                "correct": {
+                    "title": "Correct",
+                    "enum": ["yes", "no"],
+                    "type": "string",
+                },
+                "confidence": {"title": "Confidence", "type": "integer"},
+                "strict": {"const": True, "type": "boolean"},
+            },
+            "required": [
+                "extracted_final_answer",
+                "reasoning",
+                "correct",
+                "confidence",
+                "strict",
+            ],
+            "type": "object",
+            "additionalProperties": False,
+        },
+        "name": "ExtractedAnswer",
+        "strict": True,
+    },
+}
+
+
+hle_judge_infer_cfg = dict(
+    judge_reader_cfg=dict(
+        input_columns=["question", "answer", "model_answer"],
+        output_column="model_pred_uuid",
+    ),
+    judge_model=dict(
+        attr="service",
+        type=VLLMCustomAPIChat,
+        abbr="judge",
+        path="",
+        model="",
+        stream=False,
+        request_rate=0,
+        use_timestamp=False,
+        retry=2,
+        api_key="",
+        host_ip="localhost",
+        host_port=8080,
+        url="",
+        max_out_len=8192,
+        batch_size=100,
+        trust_remote_code=False,
+        generation_kwargs=dict(
+            temperature=0.01,
+            seed=0,
+            response_format=RESPONSE_FORMAT_SCHEMA,
+            chat_template_kwargs=dict(
+                enable_thinking=False,
+            ),
+        ),
+    ),
+    judge_dataset_type=HLEJGDataset,
+    prompt_template=dict(
+        type=PromptTemplate,
+        template=dict(
+            round=[
+                dict(role="HUMAN", prompt=JUDGE_PROMPT),
+            ],
+        ),
+    ),
+    retriever=dict(type=ZeroRetriever),
+    inferencer=dict(type=GenInferencer),
+)
+
+
+hle_eval_cfg = dict(
+    evaluator=dict(type=HLEJudgeEvaluator),
+)
+
+
+hle_datasets = [
+    dict(
+        abbr="hle",
+        type=HLEDataset,
+        path="ais_bench/datasets/hle/data/test-00000-of-00001.parquet",
+        reader_cfg=hle_reader_cfg,
+        infer_cfg=hle_infer_cfg,
+        judge_infer_cfg=hle_judge_infer_cfg,
+        eval_cfg=hle_eval_cfg,
+    )
+]

ais_bench/benchmark/datasets/__init__.py

@@ -59,3 +59,4 @@
 from ais_bench.benchmark.datasets.mooncake_trace import *  # noqa: F401, F403
 from ais_bench.benchmark.datasets.swebench import *  # noqa: F401, F403
 from ais_bench.benchmark.datasets.refcoco import *  # noqa: F401, F403
+from ais_bench.benchmark.datasets.hle import *