Skip to content

feat: add Gemma 4 E2B-it model benchmark #6120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sallyseok merged 5 commits into from
May 21, 2026
Merged

feat: add Gemma 4 E2B-it model benchmark #6120

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
2c7cf6a
feat: add Gemma 4 E2B-it model benchmark
sallyseok May 20, 2026
44f3863
test: comment out all benchmark models except gemma-4-e2b-it
sallyseok May 20, 2026
a04736f
fix: set benchmark runner-scale-sets to empty list
sallyseok May 20, 2026
a28d098
feat: set gemma-4-e2b-it thresholds to ~50% of observed
sallyseok May 21, 2026
9d09eab
chore: restore all models, keep only gemma-4-e2b-it addition
sallyseok May 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/config/model-tests/vllm-model-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,17 @@ benchmark:
min_throughput: 680
min_rps: 5.3

- name: "gemma-4-e2b-it"
s3_model: "gemma-4-e2b-it.tar.gz"
fleet: "x86-g6xl-runner"
extra_args: "--tensor-parallel-size 1 --max-model-len 4096 --dtype bfloat16"
input_len: 512
output_len: 128
num_prompts: 64
batch_size: 4
min_throughput: 470
min_rps: 3.7

# Pending p5e.48xlarge fleet creation. Fleet name "x86-p5e-runner" is a placeholder.
# - name: "minimax-m2.7"
# s3_model: "minimax-m2.7.tar.gz"
Expand Down
Loading