Skip to content

Fix Qwen3-32B serving sequence caps and benchmark expectations#268

Open
HazyRaven wants to merge 1 commit into
AI-Hypercomputer:mainfrom
HazyRaven:fix-qwen3-32b-performance-recipe
Open

Fix Qwen3-32B serving sequence caps and benchmark expectations#268
HazyRaven wants to merge 1 commit into
AI-Hypercomputer:mainfrom
HazyRaven:fix-qwen3-32b-performance-recipe

Conversation

@HazyRaven

Copy link
Copy Markdown

Description:

This Pull Request addresses the throughput shortfall for the Qwen3-32B recipe on Ironwood by rectifying several configurations that bottlenecked performance.

Changes Included:

  1. Removed the Admission Queue Bottleneck: The server configuration in vllm-tpu.yaml had --max-num-seqs=135, while the benchmark offered a concurrency of 320. This artificially queued requests, causing severe degradation to the steady-state batch occupancy and sustained mean throughput. We raised --max-num-seqs to 320 to rectify this.

  2. Synchronized Docker Image to Latest Nightly: Bumped the container from nightly-20260330... to the modern, verified nightly nightly-20260710-5e0243d-f1a5add across the server and benchmark instances. The modern build exhibits up to 12.8% better throughput on long-sequence decoding workloads.

  3. Pinned Reproducible InferenceX Build: The benchmark workloads were repinned to 01af32e8 (2026-07-10) to use consistent workload samplers, preventing silent regression.

  4. Refreshed Table and Documented vllm bench Pitfalls: The throughput table has been refreshed with the exact measured InferenceX numbers (e.g. 1k/1k hitting 6743 tok/s), and a new note was appended briefly detailing how modern vllm bench serve users should map their distribution boundaries (e.g., using --random-range-ratio 0.1112) to get equivalent throughput to InferenceX.

@google-cla

google-cla Bot commented Jul 17, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant