Add UniG2U benchmark task with model support (#1297)

* Add UniG2U benchmark task with model support

Add UniG2U (Unified Generation-to-Understanding) benchmark covering
11 sub-tasks across chart understanding, geometry, physics, spatial
planning, and visual puzzles.

Two evaluation modes:
- unig2u: standard understanding (single-stage)
- unig2u_GtA: Visual CoT two-stage (generate auxiliary image, then answer)

GtA triggered explicitly via generation_kwargs.visual_cot: true.

New model implementations: Ovis-U1, ILLUME+, MMaDa, Qwen-Image-Edit.

via [HAPI](https://hapi.run)

Co-Authored-By: HAPI <noreply@hapi.run>

* Fix qwen_image_edit device_map for diffusion pipeline

DiffusionPipeline does not support device_map="auto", use "balanced" instead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Add Visual CoT (GtA) support check for models

Models without GtA implementation now raise a clear error when run on
visual_cot tasks, instead of silently degrading with garbled prompts.

- Add `supports_visual_cot` class attribute to lmms base (default False)
- Add `_check_visual_cot_support()` guard in base class
- Call the check in evaluator before generate_until
- Set `supports_visual_cot = True` on ovis_u1, bagel_unig2u, illume_plus, qwen_image_edit

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Revert "Add Visual CoT (GtA) support check for models"

This reverts commit 2bc4f01.

* Add generate_visual_cot as dedicated output type for GtA tasks

Visual CoT (GtA) tasks now use output_type: generate_visual_cot instead
of generate_until. Models must implement generate_visual_cot() to run
these tasks — models without it get a clear NotImplementedError.

- Add generate_visual_cot() default in lmms base class (raises NotImplementedError)
- Register generate_visual_cot in ALL_OUTPUT_TYPES and construct_requests
- All 31 visual_cot yaml configs: output_type → generate_visual_cot
- 4 GtA models (ovis_u1, bagel_unig2u, illume_plus, qwen_image_edit):
  implement generate_visual_cot() delegating to generate_until()

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Update README with generate_visual_cot documentation

Reflect the new GtA mechanism: models must implement
generate_visual_cot() to run visual_cot tasks.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Remove duplicate entries in .gitignore

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Organize unig2u task folder by sub-task

Move yaml configs into per-task subdirectories for clarity.
Symlink utils.py into each subdirectory for !function resolution.

tasks/unig2u/
├── chartqa100/          ├── geometry3k/
├── auxsolidmath/        ├── babyvision/
├── illusionbench/       ├── mmsi/
├── phyx/                ├── realunify/
├── uni_mmmu/            ├── vsp/
├── visualpuzzles/
├── unig2u.yaml          # top-level standard group
├── unig2u_GtA.yaml      # top-level GtA group
├── utils.py             # shared utilities
└── README.md

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Address PR review feedback

- Fix function name collisions: rename generic doc_to_visual/doc_to_text/
  process_results/aggregate_results to babyvision_* and realunify_* prefixed
  versions, update 10 yaml files accordingly
- Fix Python 3.9 compat: _JudgeClient | None → Optional[_JudgeClient]
- Remove eval_logger reassignment from loguru to stdlib logging
- Remove parse_response random fallback (return None instead)
- Remove gradient_checkpointing_enable() during eval in illume_plus
- Add OOM warning for dual model loading in qwen_image_edit
- Fix bagel_umm registration to use short class name
- Fix != None → is not None
- Fix GPT-5.1 → GPT-4o in docstrings
- Fix bare except → except (json.JSONDecodeError, ValueError, KeyError)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Split merged utils.py into per-task independent utils

Replace the single 2900-line merged utils.py with the original
per-task utils.py files from the reference implementation. Each
sub-task folder now has its own independent utils.py, eliminating
function name collisions entirely.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: run black + isort formatting

* Fix code quality issues in per-task utils

- Remove eval_logger reassignment in mmsi, visualpuzzles utils
- Remove random fallback in visualpuzzles parse_response (return None)
- Fix != None → is not None in visualpuzzles
- Fix GPT-5.1 → GPT-4o in auxsolidmath, geometry3k docstrings
- Fix bare except → specific exceptions in uni_mmmu

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix lint: run black + isort on mmsi and visualpuzzles utils

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Ubuntu <xinjiezhang@RobustDNN-A100-47.anzplpv4vzne3ajfbcfpzr1dkd.ix.internal.cloudapp.net>
Co-authored-by: HAPI <noreply@hapi.run>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: mwxely <yang0756@e.ntu.edu.sg>

Author: 5 people

.gitignore

@@ -81,3 +81,5 @@ CLAUDE.md
 .opencode
 .ignored/
 .worktrees/
+Bagel/
+MMaDA/

lmms_eval/api/model.py

@@ -118,6 +118,17 @@ def generate_until_multi_round(self, requests) -> List[str]:
         """
         pass
 
+    def generate_visual_cot(self, requests) -> List[str]:
+        """Visual CoT (GtA) generation: two-stage pipeline that generates an
+        auxiliary visualization image (Stage 1) and then answers using both
+        the original and generated images (Stage 2).
+
+        Models that support GtA must override this method.
+        """
+        raise NotImplementedError(
+            f"{type(self).__name__} does not support Visual CoT (GtA). " f"To run visual_cot tasks, the model must implement generate_visual_cot(). " f"Supported models: ovis_u1, bagel_unig2u, illume_plus, qwen_image_edit"
+        )
+
     @classmethod
     def create_from_arg_string(cls: Type[T], arg_string: str, additional_config: Optional[dict] = None) -> T:
         """

lmms_eval/api/task.py

@@ -59,6 +59,7 @@
     "generate_until",
     "generate_until_multi_round",
     "generate_until_agentic",
+    "generate_visual_cot",
 ]
 
 
@@ -1563,6 +1564,8 @@ def construct_requests(self, doc_id: int, ctx: str, **kwargs) -> Union[List[Inst
 
         elif self.OUTPUT_TYPE == "generate_until":
             arguments = (ctx, copy.deepcopy(self.config.generation_kwargs), self.doc_to_visual, doc_id, self.config.task, split)
+        elif self.OUTPUT_TYPE == "generate_visual_cot":
+            arguments = (ctx, copy.deepcopy(self.config.generation_kwargs), self.doc_to_visual, doc_id, self.config.task, split)
         elif self.OUTPUT_TYPE == "generate_until_multi_round":
             arguments = (ctx, copy.deepcopy(self.config.generation_kwargs), self.doc_to_visual, partial(self.config.doc_to_text, lmms_eval_specific_kwargs=self.lmms_eval_specific_kwargs), doc_id, self.config.task, split)
         elif self.OUTPUT_TYPE == "generate_until_agentic":
@@ -1572,7 +1575,7 @@ def construct_requests(self, doc_id: int, ctx: str, **kwargs) -> Union[List[Inst
     # TODO: we add a full_docs interface here for some evaluations that needs to access the full datasets during process_results function. we may have better ways to handle this.
     @retry(stop=(stop_after_attempt(5) | stop_after_delay(1200)), wait=wait_fixed(2))
     def process_results(self, doc, results, full_docs=None):
-        if self.OUTPUT_TYPE == "generate_until":
+        if self.OUTPUT_TYPE in ("generate_until", "generate_visual_cot"):
             if isinstance(results, list) and isinstance(results[0], list):
                 results = [res.strip() for res in results[0]]
             else:

lmms_eval/models/__init__.py

@@ -23,6 +23,7 @@
     "auroracap": "AuroraCap",
     "bagel": "Bagel",
     "bagel_umm": "BagelUMM",
+    "bagel_unig2u": "BagelUniG2U",
     "baichuan_omni": "BaichuanOmni",
     "batch_gpt4": "BatchGPT4",
     "claude": "Claude",
@@ -40,6 +41,7 @@
     "gemma3": "Gemma3",
     "gpt4v": "GPT4V",
     "idefics2": "Idefics2",
+    "illume_plus": "ILLUMEPlus",
     "instructblip": "InstructBLIP",
     "internvideo2_5": "InternVideo2_5",
     "internvideo2": "InternVideo2",
@@ -63,12 +65,14 @@
     "minicpm_o": "MiniCPM_O",
     "minicpm_v": "MiniCPM_V",
     "minimonkey": "MiniMonkey",
+    "mmada": "MMaDA",
     "moviechat": "MovieChat",
     "mplug_owl_video": "mplug_Owl",
     "ola": "Ola",
     "omnivinci": "OmniVinci",
     "openai": "OpenAICompatible",
     "oryx": "Oryx",
+    "ovis_u1": "OvisU1",
     "penguinvl": "PenguinVL",
     "phi3v": "Phi3v",
     "phi4_multimodal": "Phi4",
@@ -79,6 +83,7 @@
     "qwen2_5_vl": "Qwen2_5_VL",
     "qwen2_audio": "Qwen2_Audio",
     "qwen2_vl": "Qwen2_VL",
+    "qwen_image_edit": "QwenImageEdit",
     "qwen3_omni": "Qwen3_Omni",
     "qwen3_vl": "Qwen3_VL",
     "qwen3_5": "Qwen3_5",