[Tokenizer] Upgrade tokenizer

[DrownFish19]

PR types

Function optimization

PR changes

APIs

Description

update function tokenize() for Qwen2 and Llama.

[paddle-bot]

Thanks for your contribution!

[github-actions]

This Pull Request is stale because it has been open for 60 days with no activity. 当前Pull Request 60天内无活动，被标记为stale。

[Copilot]

Pull Request Overview

This PR updates and optimizes the tokenization components for multiple models (RoBERTa, Llama, and Qwen2) and enhances testing and utility functions related to tokenizers. Key changes include:

• Enhancements to fast tokenizer implementations with improved special tokens handling and legacy behavior management (notably for Llama and Qwen2).
• Updates to dynamic tokenizer registration, local folder loading, and comprehensive tests for AutoTokenizer.
• Minor updates to generated protobuf files and utility modules for consistency and deprecation warnings.

Reviewed Changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
paddlenlp/transformers/roberta/tokenizer_fast.py	Adjusted special tokens processing and post-processing in the fast RoBERTa tokenizer.
tests/testing_utils.py & tests/transformers/auto/test_tokenizer.py	Added new tests, decorators, and adjustments for dynamic and local-tokenizer cases.
paddlenlp/transformers/llama/tokenizer.py	Revised legacy handling, special tokens building, and state serialization in the Llama tokenizer.
paddlenlp/transformers/qwen2/tokenizer.py	Updated the tokenize() method to call tokenize_without_strip via the superclass.
paddlenlp/transformers/sentencepiece_model_pb2*.py	Refreshed generated protobuf files and updated copyright notices.
paddlenlp/utils/import_utils.py & tokenizer_utils_base.py	Minor improvements in utility functions and deprecation warning messages.

Comments suppressed due to low confidence (2)

paddlenlp/transformers/llama/tokenizer.py:301

• The call to 'IdToPiece' uses an uppercase 'I', but the standard SentencePiece interface typically uses 'id_to_piece'. Please verify that this method name is correct to avoid token conversion issues.

token = self.sp_model.IdToPiece(index)

paddlenlp/transformers/tokenizer_utils_base.py:62

• Ensure that the formatted PROTOBUF_IMPORT_ERROR provides clear and actionable instructions to the user, including any required steps to restart the runtime after installing protobuf.

raise ImportError(PROTOBUF_IMPORT_ERROR.format(error_message))

[github-actions]

This Pull Request is stale because it has been open for 60 days with no activity. 当前Pull Request 60天内无活动，被标记为stale。

[paddle-bot]

Automatically closed by Paddle-bot.