MVEB Overview

[AdnanElAssadi56]

Classification/ZeroshotClassifcation:

• mteb/kinetics-400
• mteb/HMDB51
• mteb/Breakfast
• mteb/kinetics-700-2020
• mteb/UCF101-51VA Task implemented. Pending: WIP: Added video classification and zero-shot classification tasks for the UCF-101 dataset #4520
• mteb/SomethingSomethingV2 Implemented. Pending: [MVEB] Added SomethingSomething zeroshot #4542
• mteb/kinetics-600
• mteb/VGGSound Implemented. Pending: [MVEB] Add vggsound zeroshot task #4589
• mteb/AVE-Dataset
• mteb/Human-Animal-Cartoon
• mteb/RAVDESS_AV
• mteb/MUSIC-AVQA_cls-preprocessed
• mteb/MELD
• mteb/AVMeme-Exam
• mteb/WorldSense_1min

Pair Classification:

• VideoConPairClassification
• VinogroundPairClassification
• AVSpeakerBenchPairClassification

---

• mteb/Human-Animal-Cartoon
• mteb/AVE-Dataset
• mteb/RAVDESS_AV
• mteb/MELD
• mteb/MUSIC-AVQA_cls-preprocessed

Clustering:

• [MVEB] Adding WorldSense_1min as Clustering Task #4382
• [MVEB] Adding AVE-Dataset as Clustering Task #4409
• mteb/RAVDESS_AV
• [MVEB] Adding MELD Task (Clustering) #4498 [MVEB] MELD Clustering Task #4534
• [MVEB] Adding MUSIC-AVQA as Clustering Task #4422
• [MVEB] Adding UCF101 as Clustering Task #4447
• [MVEB] Adding HMDB51 as Clustering Task #4468
• VideoMME by domain?

Retrieval (VA2T, V2T, T2VA, T2V):

• mteb/MSR-VTT Task Implemented. Pending: Reupload MSR-VTT dataset in standard retrieval format #4375
• mteb/MSVD
• mteb/DiDeMo
• mteb/TUNA-Bench_1K
• mteb/ActivityNet_Captions_val2
• mteb/YouCook2_val
• mteb/VATEX_test_1k
• mteb/Shot2Story20K_test
• mteb/VGGSound_AV_RETRIEVAL
• mteb/VALOR-32K
• mteb/AudioCaps_AV
• mteb/panda-70m
• mteb/AVMeme-Exam

Retrieval (V2A, A2V):

• ... (from above datasets)

Video Question Answering:

• mteb/worldqa
• mteb/EgoSchema_subset
• mteb/NExT-QA
• mteb/PerceptionTest_val
• mteb/star_bench_val (blocked?)
• mteb/AV-SpeakerBench
• mteb/WorldSense_1min
• mteb/Daily-Omni
• mteb/Video-MME_short (blocked?)
• mteb/OmniVideoBench_subset
• mteb/AVQA_val
• mteb/AVMeme-Exam
• mteb/MVBench (blocked?)
• MME-Benchmarks/Video-MME-v2 (new release; not processed like others)

Models:

• LCO
• PE-AV (Facebook) Add model: PE-AV #3797
• Omni-Embed-Nemotron #4385
• Tevatron/OmniEmbed-v0.1 #4386
• Qwen2.5-Omni
• e5-omni
• nvidia/omnivinci #4387
• [MVEB] Add EBIND Model #4414

---

• Seed-1.6-Embedding
• ImageBind
• LanguageBind
• ONE-PEACE
• OmniBind
• Add model: v-jepa2 #4512
• Add model: vlm2vec2 #4513
• Add model: ume-r1 #4514

[Muennighoff]

is this still the best overview issue? ; cc @shriyasudhakar who's interested in joining!

[AdnanElAssadi56]

is this still the best overview issue? ; cc @shriyasudhakar who's interested in joining!

Great! @KennethEnevoldsen can you add her to the slack channel?

[shriyasudhakar]

Hi! Super excited to get started! Niklas sent me an invite but I seem to be having issues joining with my Stanford email (I'm an incoming student so it's possible my email doesn't have full privileges, submitted a ticket regarding this). Please do try resending the invite if possible. I can also provide my personal email if it's possible to invite external guests

[KennethEnevoldsen]

hi @shriyasudhakar! I sadly do not have you stanford mail - if you send me a mail on kenevoldsen@pm.me then I will try to add you

[manandey]

@Muennighoff / @KennethEnevoldsen - are there any contribution guidelines or docs available? I’d be happy to contribute. Thanks!

[Samoed]

We have a documentation about how to add new dataset here https://embeddings-benchmark.github.io/mteb/contributing/adding_a_dataset/

[manandey]

@Samoed I meant about the MVEB initiative. Is it the same?

[Samoed]

Yes, the same

[KennethEnevoldsen]

@manandey do be aware that the code that allows you to add a video dataset is being added in #4199

[dschechter27875]

Hi! I’ve started experimenting a bit with the MSR-VTT retrieval direction and built a small prototype (CLIP + frame sampling + mean pooling) that seems to work on small subsets.

I saw MSR-VTT is listed here and also mentioned in #4375 — would that be the best place to share more detailed experiments / contribute, or is there another thread I should follow for the retrieval side?

[wissam-sib]

Hi there, would love to be part of this! What is the contribution model? Are you meeting on a regular basis?

[Samoed]

You can start implementing these tasks and models