Skip to content

[Bug] ShareGPT 数据集多轮对话，一个会话中每个轮次max_out_len被设置为相同值 #242

Open
#243
Open
[Bug] ShareGPT 数据集多轮对话，一个会话中每个轮次max_out_len被设置为相同值#242
#243
Assignees
wenba0
Labels
content_check_passedissue content check passedfeature

Description

@Shadowless-ly
Shadowless-ly
opened on Apr 12, 2026

操作系统及版本

ubuntu 24.04

安装工具的python环境

在anaconda/miniconda创建的python虚拟环境

python版本

3.11

AISBench工具版本

3.1.0

AISBench执行命令

ais_bench --models vllm_api_stream_chat_multiturn --datasets sharegpt_gen --mode perf --debug --num-prompts 5 --num-warmups 0

模型配置文件或自定义配置文件内容

from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content

models = [
    dict(
        attr="service",
        type=VLLMCustomAPIChat,
        abbr="vllm-multiturn-api-chat-stream",
        path="/opt/models/MiniMax-M2.5/",
        model="",
        stream=True,
        request_rate=0,
        retry=2,
        api_key="",
        host_ip="localhost",
        host_port=8000,
        url="",
        max_out_len=512,
        batch_size=1,
        trust_remote_code=False,
        generation_kwargs=dict(
            temperature=0.01,
            ignore_eos=True,
        ),
        pred_postprocessor=dict(type=extract_non_reasoning_content),
    )
]

预期行为

基于ShareGPT数据集的多轮对话测试(every模式)，每个轮次需要按照数据集的output长度动态指定max_out_len。

预期行为：

  1. 数据集中每个对话轮次可以单独指定 max_out_len
  2. inferencer 在生成时能正确读取对应轮次的 max_out_len 值
  3. 多轮对话的每一轮使用各自配置的长度限制进行生成

示例：

{
"conversations": [...],
"max_out_len": [100, 200, 150]  // 第1轮100，第2轮200，第3轮150
}

实际行为

从实际发出的请求看，每个conversion的output_tokens字段被设置为相同的值。

见下文sharegpt_details.jsonl :

  • id=0的对话,output_tokens均为242
  • id=3的对话,output_tokens均为117
  • ...
{"data_abbr": "sharegpt", "id": 0, "success": true, "error_info": "", "time_points": {"__db_ref__": 1}, "input_tokens": 70, "output_tokens": 242, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content": ...
{"data_abbr": "sharegpt", "id": 0, "success": true, "error_info": "", "time_points": {"__db_ref__": 2}, "input_tokens": 153, "output_tokens": 242, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content":...
{"data_abbr": "sharegpt", "id": 0, "success": true, "error_info": "", "time_points": {"__db_ref__": 3}, "input_tokens": 285, "output_tokens": 242, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content":...
{"data_abbr": "sharegpt", "id": 0, "success": true, "error_info": "", "time_points": {"__db_ref__": 4}, "input_tokens": 571, "output_tokens": 242, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content":...
{"data_abbr": "sharegpt", "id": 0, "success": true, "error_info": "", "time_points": {"__db_ref__": 5}, "input_tokens": 715, "output_tokens": 242, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content":...
{"data_abbr": "sharegpt", "id": 0, "success": true, "error_info": "", "time_points": {"__db_ref__": 6}, "input_tokens": 787, "output_tokens": 242, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content":...
{"data_abbr": "sharegpt", "id": 1, "success": true, "error_info": "", "time_points": {"__db_ref__": 7}, "input_tokens": 55, "output_tokens": 70, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content": "...
{"data_abbr": "sharegpt", "id": 2, "success": true, "error_info": "", "time_points": {"__db_ref__": 8}, "input_tokens": 100, "output_tokens": 405, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content":...
{"data_abbr": "sharegpt", "id": 3, "success": true, "error_info": "", "time_points": {"__db_ref__": 9}, "input_tokens": 129, "output_tokens": 117, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content":...
{"data_abbr": "sharegpt", "id": 3, "success": true, "error_info": "", "time_points": {"__db_ref__": 10}, "input_tokens": 260, "output_tokens": 117, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content"...
{"data_abbr": "sharegpt", "id": 3, "success": true, "error_info": "", "time_points": {"__db_ref__": 11}, "input_tokens": 391, "output_tokens": 117, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content"...
{"data_abbr": "sharegpt", "id": 3, "success": true, "error_info": "", "time_points": {"__db_ref__": 12}, "input_tokens": 522, "output_tokens": 117, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content"...
{"data_abbr": "sharegpt", "id": 3, "success": true, "error_info": "", "time_points": {"__db_ref__": 13}, "input_tokens": 656, "output_tokens": 117, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content"...
{"data_abbr": "sharegpt", "id": 3, "success": true, "error_info": "", "time_points": {"__db_ref__": 14}, "input_tokens": 936, "output_tokens": 117, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content"...
{"data_abbr": "sharegpt", "id": 4, "success": true, "error_info": "", "time_points": {"__db_ref__": 15}, "input_tokens": 323, "output_tokens": 193, "extra_perf_data": {}, "extra_details_data": {}, "input": [{"content"...

前置检查

  • 我已读懂主页文档的快速入门，无法解决问题
  • 我已检索过FAQ，无重复问题
  • 我已搜索过现有Issue，无重复问题
  • 我已更新到最新版本，问题仍存在

Metadata

Metadata

Assignees

Labels

content_check_passedissue content check passedfeature

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions