Skip to content

[Docs] Add recommand instruction for custom configs#349

Open
SJTUyh wants to merge 37 commits into
AISBench:masterfrom
SJTUyh:custom_cfg
Open

[Docs] Add recommand instruction for custom configs#349
SJTUyh wants to merge 37 commits into
AISBench:masterfrom
SJTUyh:custom_cfg

Conversation

@SJTUyh

@SJTUyh SJTUyh commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Thanks for your contribution; we appreciate it a lot. The following instructions will make your pull request healthier and help you get feedback more easily. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.
感谢您的贡献,我们非常重视。以下说明将使您的拉取请求更健康,更易于获得反馈。如果您不理解某些项目,请不要担心,只需提交拉取请求并从维护人员那里寻求帮助即可。

PR Type / PR类型

  • Feature(功能新增)
  • Bugfix(Bug 修复)
  • Docs(文档更新)
  • CI/CD(持续集成/持续部署)
  • Refactor(代码重构)
  • Perf(性能优化)
  • Dependency(依赖项更新)
  • Test-Cases(测试用例更新)
  • Other(其他)

Related Issue | 关联 Issue
Fixes #(issue ID / issue 编号) / Relates to #(issue ID / issue 编号)
Fixes #294

🔍 Motivation / 变更动机

Please describe the motivation of this PR and the goal you want to achieve through this PR.
请描述您的拉取请求的动机和您希望通过此拉取请求实现的目标。

📝 Modification / 修改内容

Please briefly describe what modification is made in this PR.
请简要描述此拉取请求中进行的修改。

📐 Associated Test Results / 关联测试结果

Please provide links to the related test results, such as CI pipelines, test reports, etc.
请提供相关测试结果的链接,例如 CI 管道、测试报告等。

⚠️ BC-breaking (Optional) / 向后不兼容变更(可选)

Does the modification introduce changes that break the backward compatibility of the downstream repositories? If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.
是否引入了会破坏下游存储库向后兼容性的更改?如果是,请描述它如何破坏兼容性,以及下游项目应该如何修改其代码以保持与此 PR 的兼容性。

⚠️ Performance degradation (Optional) / 性能下降(可选)

If the modification introduces performance degradation, please describe the impact of the performance degradation and the expected performance improvement.
如果引入了性能下降,请描述性能下降的影响和预期的性能改进。

🌟 Use cases (Optional) / 使用案例(可选)

If this PR introduces a new feature, it is better to list some use cases here and update the documentation.
如果此拉取请求引入了新功能,最好在此处列出一些用例并更新文档。

✅ Checklist / 检查列表

Before PR:

  • Pre-commit or other linting tools are used to fix the potential lint issues. / 使用预提交或其他 linting 工具来修复潜在的 lint 问题。
  • Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests. / 修复的 Bug 已完全由单元测试覆盖,导致 Bug 的情况应在单元测试中添加。
  • The modification is covered by complete unit tests. If not, please add more unit tests to ensure the correctness. / 此拉取请求中的修改已完全由单元测试覆盖。如果不是,请添加更多单元测试以确保正确性。
  • All relevant documentation (API docs, docstrings, example tutorials) has been updated to reflect these changes. / 所有相关文档(API 文档、文档字符串、示例教程)已更新以反映这些更改。

After PR:

  • If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects. / 如果此拉取请求对下游或其他相关项目有潜在影响,应在那些项目中测试此 PR。
  • CLA has been signed and all committers have signed the CLA in this PR. / CLA 已签署,且本 PR 中的所有提交者均已签署 CLA。

👥 Collaboration Info / 协作信息

  • Suggested Reviewers / 建议审核人: @xxx
  • Relevant Module Owners / 相关模块负责人: @xxx
  • Other Collaboration Notes / 其他协作说明:

🌟 Useful CI Command / 实用的CI命令

Command / 命令 Introduction / 介绍
/gemini review Performs a code review for the current pull request in its current state by Gemini. / 对当前拉取请求在当前状态下由 Gemini 执行代码审核。
/gemini summary Provides a summary of the current pull request in its current state by Gemini. / 对当前拉取请求在当前状态下由 Gemini 提供摘要。
/gemini help Displays a list of available commands of Gemini. / 显示 Gemini 可用命令的列表。
/readthedocs build Triggers a build of the documentation for the current pull request in its current state by Read the Docs. / 触发当前拉取请求在当前状态下由 Read the Docs 构建文档。

@SJTUyh SJTUyh temporarily deployed to smoke-test-approval June 22, 2026 06:28 — with GitHub Actions Inactive
@github-actions github-actions Bot added the docs label Jun 22, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request significantly enhances the visibility and completeness of the 'custom configuration file' feature in AISBench. It introduces comprehensive documentation updates, including a detailed comparison with CLI usage, step-by-step guides, and cross-references across various tutorials and best practices. Additionally, multiple preset configuration examples for different evaluation scenarios (such as multi-model, multi-dataset, LLM judge, and performance benchmarks) have been added, and dataset READMEs have been updated with direct import statements. The reviewer feedback highlights a few areas for improvement: removing commented-out code in config_manager.py and model_api_test_zh_cn.py, and correcting a widespread Markdown table formatting issue in the dataset READMEs where column counts are mismatched.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines 23 to +26
def check(self):
self._check_models_config()
self._check_datasets_config()
self._check_summarizer_config()
# self._check_summarizer_config() # summarizer has default value, so it is not required

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

不建议在代码中保留注释掉的废弃代码(Line 26)。既然 summarizer 已经有了默认值且不再强制要求,建议直接删除该行,以保持代码的整洁和可维护性。

Suggested change
def check(self):
self._check_models_config()
self._check_datasets_config()
self._check_summarizer_config()
# self._check_summarizer_config() # summarizer has default value, so it is not required
def check(self):
self._check_models_config()
self._check_datasets_config()

Comment on lines +15 to +17

# datasets =
models = vllm_api_general_chat

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

配置文件中存在残留的注释行 # datasets =(Line 16),建议将其删除以保持配置文件的干净整洁。

Suggested change
# datasets =
models = vllm_api_general_chat
models = vllm_api_general_chat

Comment on lines +31 to +35
|任务名称|简介|评估指标|few-shot|prompt格式|配套文件导入方式|对应源码配置文件路径|
| --- | --- | --- | --- | --- | --- | --- | --- |
||ARC_c_gen_0_shot_chat_prompt|ARC Challenge Set数据集生成式任务|accuracy|0-shot|对话格式|`from ais_bench.benchmark.configs.datasets.ARC_c.ARC_c_gen_0_shot_chat_prompt import ARC_c_datasets as datasets`|[ARC_c_gen_0_shot_chat_prompt.py](ARC_c_gen_0_shot_chat_prompt.py)|
||ARC_c_gen_25_shot_chat_prompt|ARC Challenge Set数据集生成式任务|accuracy|25-shot|对话格式|`from ais_bench.benchmark.configs.datasets.ARC_c.ARC_c_gen_25_shot_chat_prompt import ARC_c_datasets as datasets`|[ARC_c_gen_25_shot_chat_prompt.py](ARC_c_gen_25_shot_chat_prompt.py)|
||ARC_c_ppl_0_shot_str|ARC Challenge Set数据集PPL任务|accuracy|0-shot|字符串格式|`from ais_bench.benchmark.configs.datasets.ARC_c.ARC_c_ppl_0_shot_str import ARC_c_datasets as datasets`|[ARC_c_ppl_0_shot_str.py](ARC_c_ppl_0_shot_str.py)|

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

在修改数据集 README 中的表格时,新增了 配套文件导入方式 列,但表格的列数和对齐存在不一致的问题,这会导致 Markdown 表格在渲染时错乱:

  1. 表头(Line 31):包含 7 列。
  2. 分隔符(Line 32):包含 8 列(有 8 个 ---)。
  3. 数据行(Line 33-35):以 || 开头,导致第一列为空,整行变成了 8 列。

建议修改方案
将数据行开头的 || 改为 |,并修正分隔符的列数,使其与表头一致(均为 7 列)。修改后的表格结构如下:

|任务名称|简介|评估指标|few-shot|prompt格式|配套文件导入方式|对应源码配置文件路径|
| --- | --- | --- | --- | --- | --- | --- |
|ARC_c_gen_0_shot_chat_prompt|ARC Challenge Set数据集生成式任务|accuracy|0-shot|对话格式|`from ais_bench.benchmark.configs.datasets.ARC_c.ARC_c_gen_0_shot_chat_prompt import ARC_c_datasets as datasets`|[ARC_c_gen_0_shot_chat_prompt.py](ARC_c_gen_0_shot_chat_prompt.py)|
|ARC_c_gen_25_shot_chat_prompt|ARC Challenge Set数据集生成式任务|accuracy|25-shot|对话格式|`from ais_bench.benchmark.configs.datasets.ARC_c.ARC_c_gen_25_shot_chat_prompt import ARC_c_datasets as datasets`|[ARC_c_gen_25_shot_chat_prompt.py](ARC_c_gen_25_shot_chat_prompt.py)|
|ARC_c_ppl_0_shot_str|ARC Challenge Set数据集PPL任务|accuracy|0-shot|字符串格式|`from ais_bench.benchmark.configs.datasets.ARC_c.ARC_c_ppl_0_shot_str import ARC_c_datasets as datasets`|[ARC_c_ppl_0_shot_str.py](ARC_c_ppl_0_shot_str.py)|

注:此列数不一致的问题在本次 PR 修改的所有其他数据集 README(包括英文版 README_en.md)中也同样存在,请一并予以修正。

@SJTUyh SJTUyh temporarily deployed to smoke-test-approval June 22, 2026 06:57 — with GitHub Actions Inactive
@SJTUyh SJTUyh temporarily deployed to smoke-test-approval June 22, 2026 07:20 — with GitHub Actions Inactive
@SJTUyh SJTUyh had a problem deploying to smoke-test-approval June 22, 2026 07:42 — with GitHub Actions Error
@SJTUyh SJTUyh had a problem deploying to smoke-test-approval June 22, 2026 07:51 — with GitHub Actions Error
@SJTUyh SJTUyh temporarily deployed to smoke-test-approval June 22, 2026 07:56 — with GitHub Actions Inactive
@SJTUyh SJTUyh temporarily deployed to smoke-test-approval June 22, 2026 08:19 — with GitHub Actions Inactive
@SJTUyh SJTUyh had a problem deploying to smoke-test-approval June 22, 2026 08:48 — with GitHub Actions Error
@SJTUyh SJTUyh temporarily deployed to smoke-test-approval June 22, 2026 08:52 — with GitHub Actions Inactive
@SJTUyh SJTUyh had a problem deploying to smoke-test-approval June 22, 2026 10:22 — with GitHub Actions Failure
@SJTUyh SJTUyh temporarily deployed to smoke-test-approval June 22, 2026 12:21 — with GitHub Actions Inactive
@SJTUyh SJTUyh temporarily deployed to smoke-test-approval June 23, 2026 08:05 — with GitHub Actions Inactive
@SJTUyh SJTUyh had a problem deploying to smoke-test-approval June 23, 2026 09:27 — with GitHub Actions Failure
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[需求] AISBench 压测工具增加像 Evalscope 中的SLA 自动调优设置

1 participant