[wwb] Update video inputs to avoid big video#3777
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the Who-What Benchmark video default dataset preparation to avoid sampling large videos from the downloaded tar archive, reducing resource usage when running video/VLM evaluations.
Changes:
- Filters tar archive members by a maximum size threshold before sampling videos.
- Switches from
tar.getnames()to iteratingtar.getmembers()to access member sizes.
Comment on lines
+238
to
+239
| # max resolution 1280x720, max size 6MB | ||
| max_video_size_bytes = 6 * 1024 * 1024 |
as-suvorov
approved these changes
May 5, 2026
Collaborator
|
@sbalandi @likholat It seems that tiny-random-llava-next lost it's accuracy on smaller resolutions |
|
|
||
|
|
||
| def run_test(model_id, model_type, optimum_threshold, genai_threshold, tmp_path): | ||
| def run_test(model_id, model_type, optimum_threshold, genai_threshold, tmp_path, extra_args=[]): |
| ) | ||
| def test_vlm_video(model_id, model_type, tmp_path): | ||
| run_test(model_id, model_type, 0.8, 0.8, tmp_path) | ||
| # add non default seed to avoid test with small resolution video, reason - CVS-186452 |
yatarkan
approved these changes
May 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The max size of the video in default data for
visual-video-text--model-typeis 38MB, resolution 2880x1080. Due to big memory loads with qwen2-vl model(CVS-180177) validation is craching on some platforms. We'll limit the video size. Max video size become 6MB. The resolutions of the video in the sample will be:360x480, 394x480, 640x480 , 640x360 , 480x270 , 480x318 , 1280x720Set non-default seed to avoid test with small resolution video, task for reason is created - CVS-186452
CVS-185026
Checklist: