Fix GRPOTrainer vLLM prompt token expansion bug with VLMs (#6294)

[Saurav-Gupta-9741]

What does this PR do?

Fixes #6294

Currently, when using GRPOTrainer with use_vllm=True for multimodal models (like SmolVLM), the image processor expands the <image> token into multiple image features (e.g. 729 tokens) before passing them to vLLM. vLLM then receives these already-expanded token IDs but applies its own prompt updates, causing the first image feature token to be re-expanded into another huge block of tokens. This corrupts the prompt and leads to garbage generation rollouts.

This PR implements the fix by allowing _tokenize_prompts to generate a pure, text-only set of vllm_prompt_ids (with add_special_tokens=False) before the image processor artificially expands them. These unexpanded tokens are now safely passed directly to vllm_generation.generate().

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline, Pull Request section?
• Was this discussed/approved via a GitHub issue? Please add a link to it if that's the case.
• Did you make sure to update the documentation with your changes?
• Did you write any new necessary tests?

AI writing disclosure

We welcome the use of AI tools to help with contributions. For transparency and to help us improve our review process, please indicate the level of AI involvement in this PR.

• No AI usage: the PR was written entirely by a human.
• AI-assisted: some parts were suggested or improved by AI, but the PR was written and reviewed by a human.
• AI-generated: the PR was mostly or fully generated by an AI tool.

Who can review?

Anyone in the community is free to review the PR once the tests have passed.

---

Note

Medium Risk
Changes only the vLLM generation input for conversational multimodal prompts; text-only and non-vLLM paths are unchanged, but GRPO rollouts depend on correct prompt/image alignment with vLLM.

Overview
Fixes corrupted vLLM rollouts when GRPOTrainer runs multimodal models (e.g. SmolVLM) with use_vllm=True: the VLM processor was expanding <image> into hundreds of feature tokens before vLLM, and vLLM expanded again on top of that.

_tokenize_prompts now also returns vllm_prompt_ids. When vLLM is on and the batch has images, those IDs come from the chat template as plain text (tokenize=False) plus tokenizer encoding with add_special_tokens=False—before the full processor run that inflates image tokens. Expanded prompt_ids and multimodal_fields are unchanged for the training-model forward path.

_generate_single_turn passes vllm_prompt_ids into vllm_generation.generate instead of the processor-expanded lists; _generate threads the new return value through the default rollout path.

^{Reviewed by Cursor Bugbot for commit e3c2772. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit dee50fe. Configure here.}

[Saurav-Gupta-9741]

Hi @kashif Sir and @lewtun Sir, I hope you're both doing well!

I have implemented a fix for the vLLM multimodal token expansion bug that was thoroughly detailed by @Strongich Sir in #6294. The PR is now ready for review.

Could one of you please approve the workflows so the GPU tests can run? I am very much looking forward to your feedback. If there are any changes or improvements you would like me to make to align better with the codebase, please let me know and I will update it right away!

Thank you for your time and the incredible work you do on this repository.