Skip to content

fix(tasks): utils improvements across 7 benchmarks#1386

Open
Luodian wants to merge 2 commits into
mainfrom
fix/task-utils-improvements
Open

fix(tasks): utils improvements across 7 benchmarks#1386
Luodian wants to merge 2 commits into
mainfrom
fix/task-utils-improvements

Conversation

@Luodian

@Luodian Luodian commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Summary

Generic, self-contained improvements to seven evaluation task utilities. No new dependencies; all data still loads from the existing public Hugging Face datasets.

Changes

  • egoschema: Fix the HF_HOME cache path (~/.cache/huggingface, not .../hub). Replace the unbounded submission retry loop with a bounded, backed-off POST to EgoSchema's public validation server that saves the submission first and fails loudly on persistent failure instead of spinning forever or recording a bogus 0. Return the parsed accuracy from the MC / MC_PPL aggregators. Add deterministic generation_kwargs. README now spells out which tasks score offline (the N=500 subset) vs. which are submission-only (the full set) — and the offline subset tasks (egoschema_subset, egoschema_subset_mcppl) no longer declare the server submission metric at all: they have real answers and score locally, so an offline/no-egress node can't crash on the (now loud) server POST. The full-set tasks keep server submission.
  • erqa: Use the shared extract_mcq_answer helper, add a doc_to_messages builder that interleaves images before the question text, and remove the dead local extractor / unused template load. Adds unit tests.
  • vsibench: Robust numeric-answer extraction for NA-type questions — handles bare digits, digits embedded in a sentence, and English number words (e.g. twenty-one, three hundred and twenty-one), with the first number token winning.
  • vsisuper: Harden the count_streaming metrics — _abs_dist_norm now coerces to float and guards divide-by-zero, and _parse_prediction_list tolerates malformed / non-list JSON. Scoring note: this corrects a bug where malformed predictions and zero-valued targets scored a perfect MRA (the old except: return 0.0 fed the "distance" side); they now score worst-case, so affected historical count_streaming numbers will drop.
  • mmsi_video: Add a single-visual-input subset (mmsi_video_single_input) matching Cambrian-P's reported MMSI-Video protocol, and drop unused imports.
  • tomato: Make the OpenAI import lazy (only needed for the disabled GPT parser) and stop hardcoding a 16-frame assumption in the prompt so the model's own num_frames governs the frame budget.
  • videoevalpro: Guard the score aggregator against an unexpected task_type (previously a KeyError) and use identity comparison for the judge flag.

Validation

  • py_compile, ruff check, black --check (line-length 240), and isort --profile black --check all pass on the changed files.
  • YAML task configs verified; all referenced !function utilities exist.
  • New erqa/tests/test_erqa_utils.py covers answer extraction, message interleaving, and aggregate accuracy.
  • Offline verification of the egoschema metric change: TaskManager resolves egoschema_subset/egoschema_subset_mcppl with metric_list=['score'] and the full egoschema/egoschema_mcppl still with ['submission'], no network access.

Review pass (2026-07-06)

Self-review follow-up commit removes the submission metric from the two offline subset yamls (see egoschema bullet) — previously the README recommended the subset for offline eval while its submission metric still POSTed to the validation server and, after this PR's own retry hardening, would have raised on a no-egress node before the offline score was emitted.

Luodian added 2 commits July 6, 2026 19:34
Port generic, self-contained task-utils improvements:

- egoschema: fix the HF_HOME cache path; replace the unbounded submission
  retry loop with a bounded, backed-off POST to EgoSchema's public
  validation server that fails loudly on persistent failure; return the
  accuracy from the aggregators; add deterministic generation_kwargs; and
  document offline (subset) vs submission-only (full set) scoring.
- erqa: use the shared extract_mcq_answer helper, add a doc_to_messages
  builder that interleaves images before text, drop the dead local answer
  extractor and unused config load; add unit tests.
- vsibench: robust numeric-answer extraction (bare digits, digits in a
  sentence, and English number words, first token wins) for NA questions.
- vsisuper: harden _abs_dist_norm (float coercion, divide-by-zero guard)
  and _parse_prediction_list (safe JSON parse, list guard) in the
  count_streaming task.
- mmsi_video: add a single-visual-input subset (Cambrian-P MMSI-Video
  protocol) via process_docs_single_input_only; drop unused imports.
- tomato: make the OpenAI import lazy so it is only needed for the
  (disabled) GPT parser, and stop hardcoding a 16-frame assumption in the
  prompt so the model's own num_frames governs frame count.
- videoevalpro: guard the score aggregator against an unknown task_type
  (previously a KeyError) and use identity comparison for the judge flag.
@Luodian Luodian marked this pull request as ready for review July 6, 2026 13:56
@Luodian Luodian requested review from kcz358 and pufanyi July 6, 2026 13:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant