操作系统及版本
openEuler 22.03 (LTS-SP2)
安装工具的python环境
在anaconda/miniconda创建的python虚拟环境
python版本
3.11
AISBench工具版本
3.1.20260630
AISBench执行命令
ais_bench --models vllm_api_stream_chat --datasets textvqa_gen_base64 --summarizer default_perf --mode perf --debug
模型配置文件或自定义配置文件内容
models = [
dict(
attr="service",
type=VLLMCustomAPIChat,
abbr="vllm-api-stream-chat",
path="/opt/quanzhong/Qwen3.6-35B-A3B-w8a8",
model="Qwen3.6-35B-A3B-w8a8",
stream=True,
request_rate=0,
use_timestamp=False,
retry=2,
api_key="2001****",
#host_ip="10.80.28.248",
#host_port=38444,
url="http://10.80.28.248:38444/apiaccess/1783668040513fb71a803_qwen36-35b-w8a8/v1/chat/completions",
max_out_len=512,
batch_size=1,
trust_remote_code=True,
generation_kwargs=dict(
temperature=0.01,
ignore_eos=True,
),
pred_postprocessor=dict(type=extract_non_reasoning_content),
)
]
预期行为
正常完成性能测试
实际行为
[2026-07-16 14:10:26,622] [ais_bench] [INFO] Loading vllm_api_stream_chat: /root/test/benchmark/ais_bench/benchmark/configs/./models/vllm_api/vllm_api_stream_chat.py
[2026-07-16 14:10:26,627] [ais_bench] [INFO] Loading textvqa_gen_base64: /root/test/benchmark/ais_bench/benchmark/configs/./datasets/textvqa/textvqa_gen_base64.py
[2026-07-16 14:10:26,629] [ais_bench] [INFO] Loading default_perf: /root/test/benchmark/ais_bench/benchmark/configs/./summarizers/perf/default_perf.py
[2026-07-16 14:10:26,630] [ais_bench] [INFO] Current exp folder: outputs/default/20260716_141013
[2026-07-16 14:10:26,641] [ais_bench] [INFO] Starting inference tasks...
[2026-07-16 14:10:26,642] [ais_bench] [INFO] Partitioned into 1 tasks.
[2026-07-16 14:10:26,642] [ais_bench] [INFO] Merging datasets with the same model and inferencer...
[2026-07-16 14:10:26,693] [ais_bench] [INFO] Launch TasksMonitor, PID: 37466, Refresh interval: 0.5, Run in background: True
[2026-07-16 14:10:41,443] [ais_bench] [INFO] Debug mode, print progress directly
[2026-07-16 14:10:41,445] [ais_bench] [INFO] Task [vllm-api-stream-chat/textvqa]
[2026-07-16 14:11:17,146] [ais_bench] [INFO] Zero Retriever initialized, returning empty shot case for all queries
[2026-07-16 14:11:25,025] [ais_bench] [INFO] Apply ice template finished
[2026-07-16 14:11:25,538] [ais_bench] [INFO] Using custom URL: [http://10.80.28.248:38444/apiaccess/1783668040513fb71a803_qwen36-35b-w8a8/v1/chat/completions], [host_ip: localhost] and [host_port: 8080] will be ignored
[2026-07-16 14:11:25,538] [ais_bench] [INFO] API key is set
[2026-07-16 14:11:25,540] [ais_bench] [INFO] Start warmup, run with concurrency: 1
Warmup: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 64.44case/s]
[2026-07-16 14:11:25,558] [ais_bench] [INFO] Warmup finished Total Count: 1 Success Count: 0 Failed Count: 1
Failed Reasons:
+---------------+-------+
| Failed Reason | Count |
+---------------+-------+
| Not Found | 1 |
+---------------+-------+
Traceback (most recent call last):
File "/root/test/benchmark/ais_bench/benchmark/tasks/openicl_api_infer.py", line 704, in
raise e
File "/root/test/benchmark/ais_bench/benchmark/tasks/openicl_api_infer.py", line 701, in
inferencer.run(task_state_manager)
File "/root/test/benchmark/ais_bench/benchmark/tasks/openicl_api_infer.py", line 488, in run
self.warm_up(data_list, task_state_manager)
File "/root/test/benchmark/ais_bench/benchmark/tasks/openicl_api_infer.py", line 467, in warm_up
raise AISBenchRuntimeError(
ais_bench.benchmark.utils.logging.exceptions.AISBenchRuntimeError: [TINFER-RUNTIME-001]warmup failed. Exit task because all warmup requests failed, failed reasons: {'Not Found': 1} | Visit https://ais-bench-benchmark.readthedocs.io/zh-cn/latest/faqs/error_codes.html#tinfer-runtime-001 for further help.
[2026-07-16 14:11:30,101] [ais_bench] [INFO] Inference tasks completed.
[2026-07-16 14:11:30,103] [ais_bench] [INFO] Summarizing performance results...
[2026-07-16 14:11:30,103] [ais_bench] [WARNING] Can't find details perf data of [vllm-api-stream-chat/textvqa] in outputs/default/20260716_141013, use tmp cache data.
Traceback (most recent call last):
File "/root/test/.venv/bin/ais_bench", line 10, in
sys.exit(main())
^^^^^^
File "/root/test/benchmark/ais_bench/benchmark/cli/main.py", line 6, in main
task_manager.run()
File "/root/test/benchmark/ais_bench/benchmark/cli/task_manager.py", line 39, in run
workflow_executor.execute()
File "/root/test/benchmark/ais_bench/benchmark/cli/workers.py", line 492, in execute
worker.do_work(cfg)
File "/root/test/benchmark/ais_bench/benchmark/cli/workers.py", line 469, in do_work
summarizer.summarize()
File "/root/test/benchmark/ais_bench/benchmark/summarizers/default_perf.py", line 429, in summarize
details_perf_datas = self._load_details_perf_data(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/test/benchmark/ais_bench/benchmark/summarizers/default_perf.py", line 253, in _load_details_perf_data
raise FileMatchError(
ais_bench.benchmark.utils.logging.exceptions.FileMatchError: [SUMM-FILE-001]can't find detail perf data file. Can't find any details perf data file in work_dir, please check outputs/default/20260716_141013. | Visit https://ais-bench-benchmark.readthedocs.io/zh-cn/latest/faqs/error_codes.html#summ-file-001 for further help.
前置检查
操作系统及版本
openEuler 22.03 (LTS-SP2)
安装工具的python环境
在anaconda/miniconda创建的python虚拟环境
python版本
3.11
AISBench工具版本
3.1.20260630
AISBench执行命令
ais_bench --models vllm_api_stream_chat --datasets textvqa_gen_base64 --summarizer default_perf --mode perf --debug
模型配置文件或自定义配置文件内容
models = [
dict(
attr="service",
type=VLLMCustomAPIChat,
abbr="vllm-api-stream-chat",
path="/opt/quanzhong/Qwen3.6-35B-A3B-w8a8",
model="Qwen3.6-35B-A3B-w8a8",
stream=True,
request_rate=0,
use_timestamp=False,
retry=2,
api_key="2001****",
#host_ip="10.80.28.248",
#host_port=38444,
url="http://10.80.28.248:38444/apiaccess/1783668040513fb71a803_qwen36-35b-w8a8/v1/chat/completions",
max_out_len=512,
batch_size=1,
trust_remote_code=True,
generation_kwargs=dict(
temperature=0.01,
ignore_eos=True,
),
pred_postprocessor=dict(type=extract_non_reasoning_content),
)
]
预期行为
正常完成性能测试
实际行为
[2026-07-16 14:10:26,622] [ais_bench] [INFO] Loading vllm_api_stream_chat: /root/test/benchmark/ais_bench/benchmark/configs/./models/vllm_api/vllm_api_stream_chat.py
[2026-07-16 14:10:26,627] [ais_bench] [INFO] Loading textvqa_gen_base64: /root/test/benchmark/ais_bench/benchmark/configs/./datasets/textvqa/textvqa_gen_base64.py
[2026-07-16 14:10:26,629] [ais_bench] [INFO] Loading default_perf: /root/test/benchmark/ais_bench/benchmark/configs/./summarizers/perf/default_perf.py
[2026-07-16 14:10:26,630] [ais_bench] [INFO] Current exp folder: outputs/default/20260716_141013
[2026-07-16 14:10:26,641] [ais_bench] [INFO] Starting inference tasks...
[2026-07-16 14:10:26,642] [ais_bench] [INFO] Partitioned into 1 tasks.
[2026-07-16 14:10:26,642] [ais_bench] [INFO] Merging datasets with the same model and inferencer...
[2026-07-16 14:10:26,693] [ais_bench] [INFO] Launch TasksMonitor, PID: 37466, Refresh interval: 0.5, Run in background: True
[2026-07-16 14:10:41,443] [ais_bench] [INFO] Debug mode, print progress directly
[2026-07-16 14:10:41,445] [ais_bench] [INFO] Task [vllm-api-stream-chat/textvqa]
[2026-07-16 14:11:17,146] [ais_bench] [INFO] Zero Retriever initialized, returning empty shot case for all queries
[2026-07-16 14:11:25,025] [ais_bench] [INFO] Apply ice template finished
[2026-07-16 14:11:25,538] [ais_bench] [INFO] Using custom URL: [http://10.80.28.248:38444/apiaccess/1783668040513fb71a803_qwen36-35b-w8a8/v1/chat/completions], [host_ip: localhost] and [host_port: 8080] will be ignored
[2026-07-16 14:11:25,538] [ais_bench] [INFO] API key is set
[2026-07-16 14:11:25,540] [ais_bench] [INFO] Start warmup, run with concurrency: 1
Warmup: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 64.44case/s]
[2026-07-16 14:11:25,558] [ais_bench] [INFO] Warmup finished Total Count: 1 Success Count: 0 Failed Count: 1
Failed Reasons:
+---------------+-------+
| Failed Reason | Count |
+---------------+-------+
| Not Found | 1 |
+---------------+-------+
Traceback (most recent call last):
File "/root/test/benchmark/ais_bench/benchmark/tasks/openicl_api_infer.py", line 704, in
raise e
File "/root/test/benchmark/ais_bench/benchmark/tasks/openicl_api_infer.py", line 701, in
inferencer.run(task_state_manager)
File "/root/test/benchmark/ais_bench/benchmark/tasks/openicl_api_infer.py", line 488, in run
self.warm_up(data_list, task_state_manager)
File "/root/test/benchmark/ais_bench/benchmark/tasks/openicl_api_infer.py", line 467, in warm_up
raise AISBenchRuntimeError(
ais_bench.benchmark.utils.logging.exceptions.AISBenchRuntimeError: [TINFER-RUNTIME-001]warmup failed. Exit task because all warmup requests failed, failed reasons: {'Not Found': 1} | Visit https://ais-bench-benchmark.readthedocs.io/zh-cn/latest/faqs/error_codes.html#tinfer-runtime-001 for further help.
[2026-07-16 14:11:30,101] [ais_bench] [INFO] Inference tasks completed.
[2026-07-16 14:11:30,103] [ais_bench] [INFO] Summarizing performance results...
[2026-07-16 14:11:30,103] [ais_bench] [WARNING] Can't find details perf data of [vllm-api-stream-chat/textvqa] in outputs/default/20260716_141013, use tmp cache data.
Traceback (most recent call last):
File "/root/test/.venv/bin/ais_bench", line 10, in
sys.exit(main())
^^^^^^
File "/root/test/benchmark/ais_bench/benchmark/cli/main.py", line 6, in main
task_manager.run()
File "/root/test/benchmark/ais_bench/benchmark/cli/task_manager.py", line 39, in run
workflow_executor.execute()
File "/root/test/benchmark/ais_bench/benchmark/cli/workers.py", line 492, in execute
worker.do_work(cfg)
File "/root/test/benchmark/ais_bench/benchmark/cli/workers.py", line 469, in do_work
summarizer.summarize()
File "/root/test/benchmark/ais_bench/benchmark/summarizers/default_perf.py", line 429, in summarize
details_perf_datas = self._load_details_perf_data(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/test/benchmark/ais_bench/benchmark/summarizers/default_perf.py", line 253, in _load_details_perf_data
raise FileMatchError(
ais_bench.benchmark.utils.logging.exceptions.FileMatchError: [SUMM-FILE-001]can't find detail perf data file. Can't find any details perf data file in work_dir, please check outputs/default/20260716_141013. | Visit https://ais-bench-benchmark.readthedocs.io/zh-cn/latest/faqs/error_codes.html#summ-file-001 for further help.
前置检查