Add Kinetics700 and Kinetics600 zeroshot classification task#4537
Add Kinetics700 and Kinetics600 zeroshot classification task#4537isaac-chung merged 6 commits intoembeddings-benchmark:mainfrom
Conversation
|
|
||
| def get_candidate_labels(self) -> list[str]: | ||
| return [ | ||
| name for name in self.dataset["test"].features[self.label_column_name].names |
There was a problem hiding this comment.
I used "video of {label}" for another dataset. Perhaps it makes a difference?
There was a problem hiding this comment.
Happy to add that. But, I think we can create an issue where we analyze the performance of the Zeroshot classification with and without "video of" prefix, wdyt?
Currently, this is fragmented and not standardized across dataset.
There was a problem hiding this comment.
We use this in all zero shot tasks and this would a lot of unnecessary compute
Currently, this is fragmented and not standardized across dataset.
What do you mean? I can't find examples where we don't use it, except for your tasks #4538
There was a problem hiding this comment.
Sorry for the confusion, done.
x-tabdeveloping
left a comment
x-tabdeveloping
left a comment
There was a problem hiding this comment.
It looks quite reasonable to me
Samoed
added
new dataset
deep9539
force-pushed
the
kinetics700_zeroshot
branch
from
a3ef504 to
c580d53
Compare
deep9539
force-pushed
the
kinetics700_zeroshot
branch
from
c580d53 to
3b01bfb
Compare
|
Can we merge this if it looks good? |
AdnanElAssadi56
left a comment
AdnanElAssadi56
left a comment
There was a problem hiding this comment.
can you run make lint?
Done |
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
5 participants
mtebmtebpackage.mteb run -m {model_name} -t {task_name}command.mteb/baseline-random encoder- Partially done, Remaining in progressfacebook/pe-av-small-16-frameor another small model (Takes too long, maybe a day)Kinetics600VAZeroShotClassification x baseline
Kinetics600VZeroShotClassification x baseline
Kinetics700VAZeroShotClassification x baseline
Kinetics700VAZeroShotClassification x baseline