Skip to content

Add AVMeme-Exam zero shot classification task#4539

Merged
isaac-chung merged 6 commits intoembeddings-benchmark:mainfrom
deep9539:avmeme_exam
Apr 30, 2026
Merged

Add AVMeme-Exam zero shot classification task#4539
isaac-chung merged 6 commits intoembeddings-benchmark:mainfrom
deep9539:avmeme_exam

Conversation

@deep9539
Copy link
Copy Markdown
Contributor

  • I have outlined why this dataset is filling an existing gap in mteb
  • I have tested that the dataset runs with the mteb package.
  • I have run the following models on the task (adding the results to the pr). These can be run using the mteb run -m {model_name} -t {task_name} command.
    • mteb/baseline-random encoder
    • facebook/pe-av-small-16-frame or another small model (Takes too long, maybe a day)
  • I have checked that the performance is neither trivial (close to perfect scores) nor random.
  • I have considered the size of the dataset and reduced it if it is too big (e.g. 2048 examples for binary classification)
mteb/baseline-random encoder facebook/pe-av-small-16-frame
** AVMemeAudioVideoZeroShotClassification** 0.241111
** AVMemeVideoZeroShotClassification** 0.207778

AVMemeVideoZeroShotClassification x random encoder
AVMemeAudioVideoZeroShotClassification x random encoder

@deep9539 deep9539 mentioned this pull request Apr 28, 2026
72 tasks
Copy link
Copy Markdown
Collaborator

@x-tabdeveloping x-tabdeveloping left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good so far. Let's wait for what we do with the facebook encoder.

@Samoed Samoed added new dataset Issues related to adding a new task or dataset video video extension labels Apr 28, 2026
Comment thread mteb/tasks/zeroshot_classification/eng/avmeme_exam_classification.py Outdated
is_beta=True,
)
input_column_name = "video"
label_column_name: str = "category"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you using the correct label here? Is the dataset intended for "emotion"?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Thanks for catching this, I must be looking at wrong huggingface tab. You are right the dataset is not for emotion.

@AdnanElAssadi56
Copy link
Copy Markdown
Contributor

The category label for this dataset seems weird to me. I also saw it was chosen in the other merged task. maybe emotion is the better choice here; we'll have to look at original source and see the intention of the authors.

Comment thread mteb/tasks/zeroshot_classification/eng/avmeme_exam_classification.py Outdated
Comment thread mteb/tasks/zeroshot_classification/eng/avmeme_exam_classification.py Outdated
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Comment thread mteb/tasks/zeroshot_classification/eng/avmeme_exam_classification.py Outdated
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
@isaac-chung isaac-chung merged commit 061edcd into embeddings-benchmark:main Apr 30, 2026
13 checks passed
@deep9539
Copy link
Copy Markdown
Contributor Author

Thanks for changing the label @Samoed , I agree with you that emotion makes more sense than the previous label which was sound category.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new dataset Issues related to adding a new task or dataset video video extension

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants