[generation] Fix misleading synced_gpus warning in continuous batching#47158
Open
Partha-Shankar wants to merge 3 commits into
Open
[generation] Fix misleading synced_gpus warning in continuous batching#47158Partha-Shankar wants to merge 3 commits into
Partha-Shankar wants to merge 3 commits into
Conversation
Correct the warning negation when synced_gpus is passed in continuous batching generation mode. Since synced_gpus is ignored and not processed, the warning should read 'is ignored' instead of 'is not ignored'. Add a regression test in test_continuous_batching.py to verify.
cf880e8 to
1c18627
Compare
Collaborator
|
Thanks for fixing the nit, please remove the test and I will merge. |
Author
|
Done — removed the test, only the warning-message fix remains. Thanks for the review! |
Contributor
CI recapDashboard: View test results in Grafana |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR fixes a misleading user-facing warning message when calling
generate()withsynced_gpus=Truein continuous batching mode (e.g.cache_implementation="paged").Closes #47276
Summary
The warning message previously stated:
synced_gpus is not ignored for continuous batching. Got synced_gpus = TrueThis double negative incorrectly implies that
synced_gpusis active or supported. In fact,synced_gpusis discarded during continuous batching execution. The comment directly above the warning also explicitly reads# others are ignored.This PR corrects the warning to state:
synced_gpus is ignored for continuous batching. Got synced_gpus = TrueRoot Cause
A simple negation typo in the warning string format.
Solution
Change
"is not ignored"to"is ignored"in the warning message insrc/transformers/generation/utils.py.Why This Is a Standalone Fix
This PR addresses a user-facing negation typo in the warning message logged
when
synced_gpusis passed during continuous batching. Because continuousbatching ignores
synced_gpus, the warning mistakenly read "synced_gpus isnot ignored...", which can cause significant confusion for users debugging
distributed setups who believe the parameter is active. As this is a highly
visible, misleading warning specifically affecting the continuous batching
entry point, and a search of the generation codebase confirmed no other
stale warning negation bugs nearby, it is submitted as a targeted, standalone
fix rather than bundled into a broader cleanup.
Tests Run
pytest tests/generation/test_continuous_batching.py -v→ 49 passed, 71 skipped, 0 failed.ruff checkandruff formatlocally.AI-Assisted Tooling & Guidelines Compliance
assist in understanding the continuous batching parameter validation flow.
The fix was authored directly by the developer.
huggingface/transformers for PRs touching the same warning string or file
(src/transformers/generation/utils.py). No duplicates or similar
sweeping cleanup PRs were found.
$ ruff check src/transformers/generation/utils.py All checks passed!$ pytest tests/generation/test_continuous_batching.py -v ======================= 49 passed, 71 skipped in 12.14s =======================