Skip to content

Add Zeroshot classification for Worldsense classification datatset#4538

Merged
AdnanElAssadi56 merged 3 commits intoembeddings-benchmark:mainfrom
deep9539:worldsense_dataset
Apr 29, 2026
Merged

Add Zeroshot classification for Worldsense classification datatset#4538
AdnanElAssadi56 merged 3 commits intoembeddings-benchmark:mainfrom
deep9539:worldsense_dataset

Conversation

@deep9539
Copy link
Copy Markdown
Contributor

@deep9539 deep9539 commented Apr 28, 2026

  • I have outlined why this dataset is filling an existing gap in mteb
  • I have tested that the dataset runs with the mteb package.
  • I have run the following models on the task (adding the results to the pr). These can be run using the mteb run -m {model_name} -t {task_name} command.
    • mteb/baseline-random encoder
    • facebook/pe-av-small-16-frame or another small model (Takes too long, maybe a day)
  • I have checked that the performance is neither trivial (close to perfect scores) nor random.
  • I have considered the size of the dataset and reduced it if it is too big (e.g. 2048 examples for binary classification)
mteb/baseline-random encoder facebook/pe-av-small-16-frame
** WorldSenseAudioVideoZeroShotClassification** 0.123209
** WorldSenseVideoZeroShotClassification** 0.106972

WorldSenseVideoZeroShotClassification x random encoder
WorldSenseAudioVideoZeroShotClassification x random encoder

@deep9539 deep9539 mentioned this pull request Apr 28, 2026
72 tasks
@Samoed Samoed added new dataset Issues related to adding a new task or dataset video video extension labels Apr 28, 2026
@AdnanElAssadi56 AdnanElAssadi56 merged commit 22d1c00 into embeddings-benchmark:main Apr 29, 2026
13 checks passed
@Samoed
Copy link
Copy Markdown
Member

Samoed commented Apr 29, 2026

I missed a part that here missing a video of {name}. Discussion in #4537 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new dataset Issues related to adding a new task or dataset video video extension

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants