Skip to content

feat: add VideoNet benchmark#1308

Merged
kcz358 merged 2 commits intoEvolvingLMMs-Lab:mainfrom
yadavta:feat/videonet
May 6, 2026
Merged

feat: add VideoNet benchmark#1308
kcz358 merged 2 commits intoEvolvingLMMs-Lab:mainfrom
yadavta:feat/videonet

Conversation

@yadavta
Copy link
Copy Markdown
Contributor

@yadavta yadavta commented Apr 26, 2026

Summary

  • Add VideoNet benchmark (5k videos, 4k test + 1k val multiple-choice questions, 4k binary questions)

Tasks added

Task Description
videonet_mcq_test test set for multiple-choice style of benchmark
videonet_mcq_val validation set for multiple-choice style of benchmark
videonet_binary_0shot binary style of benchmark with 0 in-context examples
videonet_binary_1shot binary style of benchmark with 1 in-context examples
videonet_binary_2shot binary style of benchmark with 2 in-context examples
videonet_binary_3shot binary style of benchmark with 3 in-context examples

Details

  • Data for CVPR 2026 Highlight paper titled "VideoNet: Domain-Specific Action Recognition in the VLM Era"
  • Paper, HF Dataset, Website
  • Please note that our benchmark has two different versions: a MCQ setup and a binary setup. I have separated the logic for these two setups, but am happy to refactor/combine if preferred.

Validation

  • I've tested all tasks end-to-end on a few different servers that I have access to

Type of Change

  • New benchmark/task

yadavta added 2 commits April 25, 2026 17:58
A large portion of the HF Dataset is not required
to run evals on our benchmark. Updated the code
so that only files required for evals are downloaded.
@kcz358 kcz358 merged commit ed09b9a into EvolvingLMMs-Lab:main May 6, 2026
2 of 3 checks passed
@yadavta
Copy link
Copy Markdown
Contributor Author

yadavta commented May 6, 2026

Thanks for merging!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants