Skip to content

Commit 60a59e4

Browse files
KemingWuKemingWu
authored
add kris_bench task (#1017)
* add kris_bench task * style: format prepare_dataset.py with black * kris update * kris_update * update kris_bench
1 parent 7420e68 commit 60a59e4

5 files changed

Lines changed: 1346 additions & 0 deletions

File tree

‎lmms_eval/tasks/kris_bench/README.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# KRIS-Bench
2+
3+
KRIS-Bench is an image editing benchmark that evaluates:
4+
- **Consistency** (non-instructed content preserved)
5+
- **Instruction following** (requested edit is performed correctly)
6+
- **Quality** (visual quality / artifacts)
7+
- **Knowledge plausibility** (only for knowledge categories)
8+
9+
This lmms-eval task:
10+
1. Generates an image per sample (conditioned on 1+ source images).
11+
2. Scores the generated image with an **OpenAI-compatible multimodal judge** inside `process_results`.
12+
13+
## Data Source
14+
15+
This task loads data from Hugging Face Hub:
16+
- `wukeming11/kris-bench` (images embedded in dataset)
17+
18+
Images are loaded automatically via `--process_with_media` flag. No local paths needed!
19+
20+
## Judge Server
21+
22+
You must provide an OpenAI-compatible API server that supports multimodal chat completions (e.g., vLLM serving Qwen2.5-VL).
23+
24+
Required:
25+
26+
```bash
27+
export KRIS_BENCH_API_KEY="EMPTY"
28+
export KRIS_BENCH_BASE_URL="http://localhost:8000/v1"
29+
export KRIS_BENCH_EVAL_MODEL_NAME="Qwen/Qwen2.5-VL-72B-Instruct-AWQ"
30+
```
31+
32+
Optional:
33+
- `KRIS_BENCH_JUDGE_MODEL_NAME`: use a separate model for judging
34+
- `KRIS_BENCH_TIMEOUT` (default: 180)
35+
- `KRIS_BENCH_MAX_RETRIES` (default: 3)
36+
- `KRIS_BENCH_CALL_DELAY` (default: 0.5)
37+
38+
## Important Runtime Flag
39+
40+
**Required**: `--process_with_media` flag to load images from HF dataset and edited images from model output.
41+
42+
## Usage Example
43+
44+
```bash
45+
export KRIS_BENCH_API_KEY="EMPTY"
46+
export KRIS_BENCH_BASE_URL="http://localhost:8000/v1"
47+
export KRIS_BENCH_EVAL_MODEL_NAME="Qwen/Qwen2.5-VL-72B-Instruct-AWQ"
48+
49+
accelerate launch --num_processes=8 -m lmms_eval \
50+
--model bagel_lmms_engine \
51+
--model_args pretrained=your_model_path,device_map=cuda,output_image_dir=./logs/kris_bench_images \
52+
--tasks kris_bench \
53+
--batch_size 1 \
54+
--output_path ./logs/ \
55+
--log_samples \
56+
--process_with_media
57+
```
58+
59+
## How It Works
60+
61+
1. **Input images**: Loaded from HF dataset `doc["ori_images"]` via `--process_with_media`
62+
2. **Model generation**: Model saves edited images to `output_image_dir`
63+
3. **Evaluation**: Task reads edited images from `pred["images"][0]` path, sends to VLM judge
64+
4. **Scoring**: Judge returns consistency, instruction, quality (and knowledge) scores

‎lmms_eval/tasks/kris_bench/__init__.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# KRIS-Bench task
2+
# Reference: github_repo/Kris_Bench

‎lmms_eval/tasks/kris_bench/kris_bench.yaml‎

Lines changed: 208 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,208 @@
1+
# KRIS-Bench task configuration for lmms-eval
2+
#
3+
# This task loads data from Hugging Face Hub (images embedded in dataset).
4+
# No local paths or environment variables needed for data loading.
5+
#
6+
# Scoring uses a multimodal judge (OpenAI-compatible VLM server) inside process_results.
7+
#
8+
# To upload your own dataset:
9+
# python lmms_eval/tasks/kris_bench/upload_to_hf.py \
10+
# --bench_root /path/to/KRIS_Bench --repo_id your-username/kris-bench
11+
12+
dataset_path: wukeming11/kris-bench
13+
task: "kris_bench"
14+
test_split: test
15+
output_type: generate_until
16+
17+
doc_to_visual: !function utils.kris_bench_doc_to_visual
18+
doc_to_text: !function utils.kris_bench_doc_to_text
19+
doc_to_target: !function utils.kris_bench_doc_to_target
20+
21+
generation_kwargs:
22+
max_new_tokens: 64
23+
temperature: 0
24+
top_p: 1.0
25+
num_beams: 1
26+
do_sample: false
27+
28+
process_results: !function utils.kris_bench_process_results
29+
30+
metric_list:
31+
- metric: kris_bench_consistency_score
32+
aggregation: !function utils.kris_bench_aggregate_consistency
33+
higher_is_better: true
34+
- metric: kris_bench_instruction_score
35+
aggregation: !function utils.kris_bench_aggregate_instruction
36+
higher_is_better: true
37+
- metric: kris_bench_quality_score
38+
aggregation: !function utils.kris_bench_aggregate_quality
39+
higher_is_better: true
40+
- metric: kris_bench_knowledge_score
41+
aggregation: !function utils.kris_bench_aggregate_knowledge
42+
higher_is_better: true
43+
- metric: kris_bench_overall_avg
44+
aggregation: !function utils.kris_bench_aggregate_mean
45+
higher_is_better: true
46+
47+
# ---------------------------------------------------------------------------
48+
# Dimension-level reporting (per requested taxonomy)
49+
# Each metric is averaged over samples that belong to that dimension.
50+
# "avg" is the per-sample mean over available metric dimensions:
51+
# - Non-knowledge: mean(consistency, instruction, quality)
52+
# - Knowledge: mean(consistency, instruction, quality, knowledge)
53+
# ---------------------------------------------------------------------------
54+
55+
# Factual Knowledge -> Attribute Perception
56+
- metric: kris_bench_factual_attribute_perception_consistency_score
57+
aggregation: !function utils.kris_bench_aggregate_mean
58+
higher_is_better: true
59+
- metric: kris_bench_factual_attribute_perception_instruction_score
60+
aggregation: !function utils.kris_bench_aggregate_mean
61+
higher_is_better: true
62+
- metric: kris_bench_factual_attribute_perception_quality_score
63+
aggregation: !function utils.kris_bench_aggregate_mean
64+
higher_is_better: true
65+
- metric: kris_bench_factual_attribute_perception_avg
66+
aggregation: !function utils.kris_bench_aggregate_mean
67+
higher_is_better: true
68+
69+
# Factual Knowledge -> Spatial Perception
70+
- metric: kris_bench_factual_spatial_perception_consistency_score
71+
aggregation: !function utils.kris_bench_aggregate_mean
72+
higher_is_better: true
73+
- metric: kris_bench_factual_spatial_perception_instruction_score
74+
aggregation: !function utils.kris_bench_aggregate_mean
75+
higher_is_better: true
76+
- metric: kris_bench_factual_spatial_perception_quality_score
77+
aggregation: !function utils.kris_bench_aggregate_mean
78+
higher_is_better: true
79+
- metric: kris_bench_factual_spatial_perception_avg
80+
aggregation: !function utils.kris_bench_aggregate_mean
81+
higher_is_better: true
82+
83+
# Factual Knowledge -> Temporal Prediction (Reverse / Intermediate / Forward)
84+
- metric: kris_bench_factual_temporal_reverse_prediction_consistency_score
85+
aggregation: !function utils.kris_bench_aggregate_mean
86+
higher_is_better: true
87+
- metric: kris_bench_factual_temporal_reverse_prediction_instruction_score
88+
aggregation: !function utils.kris_bench_aggregate_mean
89+
higher_is_better: true
90+
- metric: kris_bench_factual_temporal_reverse_prediction_quality_score
91+
aggregation: !function utils.kris_bench_aggregate_mean
92+
higher_is_better: true
93+
- metric: kris_bench_factual_temporal_reverse_prediction_avg
94+
aggregation: !function utils.kris_bench_aggregate_mean
95+
higher_is_better: true
96+
97+
- metric: kris_bench_factual_temporal_intermediate_prediction_consistency_score
98+
aggregation: !function utils.kris_bench_aggregate_mean
99+
higher_is_better: true
100+
- metric: kris_bench_factual_temporal_intermediate_prediction_instruction_score
101+
aggregation: !function utils.kris_bench_aggregate_mean
102+
higher_is_better: true
103+
- metric: kris_bench_factual_temporal_intermediate_prediction_quality_score
104+
aggregation: !function utils.kris_bench_aggregate_mean
105+
higher_is_better: true
106+
- metric: kris_bench_factual_temporal_intermediate_prediction_avg
107+
aggregation: !function utils.kris_bench_aggregate_mean
108+
higher_is_better: true
109+
110+
- metric: kris_bench_factual_temporal_forward_prediction_consistency_score
111+
aggregation: !function utils.kris_bench_aggregate_mean
112+
higher_is_better: true
113+
- metric: kris_bench_factual_temporal_forward_prediction_instruction_score
114+
aggregation: !function utils.kris_bench_aggregate_mean
115+
higher_is_better: true
116+
- metric: kris_bench_factual_temporal_forward_prediction_quality_score
117+
aggregation: !function utils.kris_bench_aggregate_mean
118+
higher_is_better: true
119+
- metric: kris_bench_factual_temporal_forward_prediction_avg
120+
aggregation: !function utils.kris_bench_aggregate_mean
121+
higher_is_better: true
122+
123+
# Conceptual Knowledge -> Social Science
124+
- metric: kris_bench_conceptual_social_science_consistency_score
125+
aggregation: !function utils.kris_bench_aggregate_mean
126+
higher_is_better: true
127+
- metric: kris_bench_conceptual_social_science_instruction_score
128+
aggregation: !function utils.kris_bench_aggregate_mean
129+
higher_is_better: true
130+
- metric: kris_bench_conceptual_social_science_quality_score
131+
aggregation: !function utils.kris_bench_aggregate_mean
132+
higher_is_better: true
133+
- metric: kris_bench_conceptual_social_science_knowledge_score
134+
aggregation: !function utils.kris_bench_aggregate_mean
135+
higher_is_better: true
136+
- metric: kris_bench_conceptual_social_science_avg
137+
aggregation: !function utils.kris_bench_aggregate_mean
138+
higher_is_better: true
139+
140+
# Conceptual Knowledge -> Natural Science
141+
- metric: kris_bench_conceptual_natural_science_consistency_score
142+
aggregation: !function utils.kris_bench_aggregate_mean
143+
higher_is_better: true
144+
- metric: kris_bench_conceptual_natural_science_instruction_score
145+
aggregation: !function utils.kris_bench_aggregate_mean
146+
higher_is_better: true
147+
- metric: kris_bench_conceptual_natural_science_quality_score
148+
aggregation: !function utils.kris_bench_aggregate_mean
149+
higher_is_better: true
150+
- metric: kris_bench_conceptual_natural_science_knowledge_score
151+
aggregation: !function utils.kris_bench_aggregate_mean
152+
higher_is_better: true
153+
- metric: kris_bench_conceptual_natural_science_avg
154+
aggregation: !function utils.kris_bench_aggregate_mean
155+
higher_is_better: true
156+
157+
# Procedural Knowledge -> Logical Reasoning
158+
- metric: kris_bench_procedural_logical_reasoning_consistency_score
159+
aggregation: !function utils.kris_bench_aggregate_mean
160+
higher_is_better: true
161+
- metric: kris_bench_procedural_logical_reasoning_instruction_score
162+
aggregation: !function utils.kris_bench_aggregate_mean
163+
higher_is_better: true
164+
- metric: kris_bench_procedural_logical_reasoning_quality_score
165+
aggregation: !function utils.kris_bench_aggregate_mean
166+
higher_is_better: true
167+
- metric: kris_bench_procedural_logical_reasoning_knowledge_score
168+
aggregation: !function utils.kris_bench_aggregate_mean
169+
higher_is_better: true
170+
- metric: kris_bench_procedural_logical_reasoning_avg
171+
aggregation: !function utils.kris_bench_aggregate_mean
172+
higher_is_better: true
173+
174+
# Procedural Knowledge -> Instruction Decomposition
175+
- metric: kris_bench_procedural_instruction_decomposition_consistency_score
176+
aggregation: !function utils.kris_bench_aggregate_mean
177+
higher_is_better: true
178+
- metric: kris_bench_procedural_instruction_decomposition_instruction_score
179+
aggregation: !function utils.kris_bench_aggregate_mean
180+
higher_is_better: true
181+
- metric: kris_bench_procedural_instruction_decomposition_quality_score
182+
aggregation: !function utils.kris_bench_aggregate_mean
183+
higher_is_better: true
184+
- metric: kris_bench_procedural_instruction_decomposition_avg
185+
aggregation: !function utils.kris_bench_aggregate_mean
186+
higher_is_better: true
187+
188+
# Big-class averages (3 avgs)
189+
- metric: kris_bench_factual_knowledge_avg
190+
aggregation: !function utils.kris_bench_aggregate_mean
191+
higher_is_better: true
192+
- metric: kris_bench_conceptual_knowledge_avg
193+
aggregation: !function utils.kris_bench_aggregate_mean
194+
higher_is_better: true
195+
- metric: kris_bench_procedural_knowledge_avg
196+
aggregation: !function utils.kris_bench_aggregate_mean
197+
higher_is_better: true
198+
199+
lmms_eval_specific_kwargs:
200+
default:
201+
pre_prompt: ""
202+
post_prompt: ""
203+
204+
metadata:
205+
- version: 0.1
206+
description: "KRIS-Bench (image editing & knowledge plausibility) with OpenAI-compatible VLM judge"
207+
208+

0 commit comments

Comments
 (0)