Skip to content

refactor(tasks): migrate 10 benchmarks from llm_judge to the verifiers pipeline#1388

Open
Luodian wants to merge 3 commits into
feat/verifiers-modulefrom
feat/verifiers-task-migrations
Open

refactor(tasks): migrate 10 benchmarks from llm_judge to the verifiers pipeline#1388
Luodian wants to merge 3 commits into
feat/verifiers-modulefrom
feat/verifiers-task-migrations

Conversation

@Luodian

@Luodian Luodian commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Migrates 10 benchmarks from the legacy lmms_eval.llm_judge server API to the new lmms_eval.verifiers pipeline introduced by the stacked feat/verifiers-module PR.

Stacked on #1379. Review against feat/verifiers-module; merge after it lands (this branch then gets rebased/retargeted so the duplicated base commits drop out).

Each task's LLM-judge call is replaced by a lazily-constructed VerificationPipeline (threaded double-checked singleton) built from the package's OpenAIVerifier provider, with per-task extractors/prompts/response-parsers preserved. Scoring behavior is preserved with three deliberate exceptions, listed under Behavior notes below.

Tasks migrated

Task Verifier Notes
mmmu OpenAIVerifier(binary) reasoning process_results
mmvu StripReasoning + OpenAIVerifier(binary) open-ended GPT rescue now actually grades (see Behavior notes)
mathvision StripReasoning + OpenAIVerifier(binary) gpt_eval process_results
videoevalpro StripReasoning + OpenAIVerifier(custom prompt, A/B/C parser) full judge rewrite; drops raw openai client + GPT4oJudge
llava-in-the-wild OpenAIVerifier(comparative parser) replaces evaluate_comparative
llava-bench-coco OpenAIVerifier(comparative parser) preserves system-instruction prefix
browsecomp StripReasoning + OpenAIVerifier(binary) gated by metadata use_lmms_judge; exact-match fallback on judge failure
corecognition StripReasoning + OpenAIVerifier(binary) gated by metadata use_lmms_judge; direct-comparison fallback on judge failure
voicebench OpenAIVerifier(1-5 score parser) partial: only the open-ended judge migrated
ami OpenAIVerifier(1-5 score parser) edinburghcstr AMI meeting corpus

All migrations use only the openai provider (present in the verifiers package).

Behavior notes (deliberate changes)

  1. mmvu (open-ended): the legacy path passed the judge a rubric-only custom_prompt with no question/answer/prediction substituted — build_binary_prompt returns such prompts unchanged and the provider ignores the request fields, so the GPT rescue never actually graded anything (a latent no-op). The migrated prompt substitutes the real fields, so open-ended mmvu scores will move. This is a bug fix, not drift.
  2. browsecomp / corecognition now run StripReasoningExtractor before judging — reasoning blocks no longer reach the judge.
  3. ami / voicebench: the legacy judge system messages are preserved as prompt prefixes (same approach as llava-bench-coco's migration).

Validation

  • py_compile passes on all 10 files.
  • All 139 yaml !function utils.* references resolve.
  • ruff: 6 findings, all pre-existing (identical to base, line-shifted only); migration introduces zero new lint.
  • Verifiers imports resolve; 5 representative modules load against the real verifiers package.
  • Hermetic mocked-judge tests: corecognition/browsecomp fall back correctly on judge failure; ami/voicebench prompts include the original system text.

Notes

  • videoevalpro carries two small aggregate fixes also present in fix/task-utils-improvements; a trivial re-merge may be needed depending on merge order.
  • voicebench intentionally keeps its llm_judge import: a second, un-migrated judge path still uses it.

Review pass (2026-07-06)

  • Merged the verifiers-module review fixes (judge-failure judge_failed metadata, composite confident contract) into this branch.
  • Judge infra failures are no longer scored as wrong answers: corecognition falls back to its direct string comparison and browsecomp to its documented exact-match path when the verifier reports judge_failed (the pipeline swallows judge exceptions, so the old except-based fallbacks had become unreachable).
  • Restored the ami/voicebench judge system messages (Behavior note 3) and removed an orphaned _create_judge_prompt helper.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant