Skip to content

[https://nvbugs/6157892][fix] Mistral format refactor#15123

Open
evezhier wants to merge 18 commits into
NVIDIA:mainfrom
evezhier:fix/mistral-format-refactor
Open

[https://nvbugs/6157892][fix] Mistral format refactor#15123
evezhier wants to merge 18 commits into
NVIDIA:mainfrom
evezhier:fix/mistral-format-refactor

Conversation

@evezhier

@evezhier evezhier commented Jun 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary by CodeRabbit

  • New Features

    • Added --custom_tokenizer CLI option to specify custom or built-in tokenizers

  • Bug Fixes

    • Fixed Mistral chat template support
    • Improved Mistral model configuration loading and robustness
    • Enhanced quantization configuration detection and handling
    • Improved RoPE parameter configuration fallback logic

  • Improvements

    • Refined model type resolution for multimodal serving scenarios

Description

Mistral format refactor for transformers v5.

  • HF checkpoint loading supported for Ministral3 models
  • v5 PretrainedConfig null rope parameters guarded
  • Both Mistral and HF processors supported. Each checkpoint format uses the corresponding processor.
  • Native tokenizer for text-only models is exposed via --custom_tokenizer, as in deepseekv3.2

Fixes also [https://nvbugs/6162588], [https://nvbugs/6162121]

Test Coverage

Existing Mistral tests should pass.

PR Checklist

Please review the following before submitting your PR:

  • PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

  • PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

  • Test cases are provided for new code paths (see test instructions)

  • If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

  • Any new dependencies have been scanned for license and vulnerabilities

  • CODEOWNERS updated if ownership changes

  • Documentation updated as needed

  • Update tava architecture diagram if there is a significant design change in PR.

  • The reviewers assigned automatically/manually are appropriate for the PR.

  • Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

@evezhier evezhier changed the title [https://nvbugs/6157892] fix/mistral format refactor [https://nvbugs/6157892][BREAKING] fix/mistral format refactor Jun 9, 2026
@evezhier evezhier changed the title [https://nvbugs/6157892][BREAKING] fix/mistral format refactor [https://nvbugs/6157892][BREAKING][fix] Mistral format refactor Jun 9, 2026
@evezhier evezhier changed the title [https://nvbugs/6157892][BREAKING][fix] Mistral format refactor [https://nvbugs/6157892][fix][BREAKING] Mistral format refactor Jun 9, 2026
@evezhier evezhier force-pushed the fix/mistral-format-refactor branch from 5f16ab7 to bedc948 Compare June 9, 2026 07:48
@evezhier evezhier changed the title [https://nvbugs/6157892][fix][BREAKING] Mistral format refactor [https://nvbugs/6157892][fix] [BREAKING] Mistral format refactor Jun 9, 2026
@evezhier

evezhier commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53004 [ run ] triggered by Bot. Commit: bedc948 Link to invocation

@evezhier evezhier force-pushed the fix/mistral-format-refactor branch from bedc948 to 5884020 Compare June 9, 2026 07:58
@evezhier

evezhier commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53008 [ run ] triggered by Bot. Commit: 5884020 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53004 [ run ] completed with state ABORTED. Commit: bedc948

Link to invocation

@evezhier evezhier force-pushed the fix/mistral-format-refactor branch from 5884020 to fe7e247 Compare June 9, 2026 08:12
@evezhier

evezhier commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53017 [ run ] triggered by Bot. Commit: fe7e247 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53008 [ run ] completed with state ABORTED. Commit: 5884020

Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53017 [ run ] completed with state SUCCESS. Commit: fe7e247
/LLM/main/L0_MergeRequest_PR pipeline #42241 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@evezhier evezhier force-pushed the fix/mistral-format-refactor branch from 6b33be9 to b6f16a8 Compare June 9, 2026 18:37
@evezhier

evezhier commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53124 [ run ] triggered by Bot. Commit: b6f16a8 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53124 [ run ] completed with state FAILURE. Commit: b6f16a8
/LLM/main/L0_MergeRequest_PR pipeline #42329 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@evezhier

evezhier commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53142 [ run ] triggered by Bot. Commit: 12009ef Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53142 [ run ] completed with state FAILURE. Commit: 12009ef
/LLM/main/L0_MergeRequest_PR pipeline #42346 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@evezhier

evezhier commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@tensorrt-cicd

tensorrt-cicd commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53321 [ run ] triggered by Bot. Commit: 46e8964 Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #53321 [ run ] completed with state FAILURE. Commit: 46e8964
/LLM/main/L0_MergeRequest_PR pipeline #42507 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@evezhier evezhier force-pushed the fix/mistral-format-refactor branch from 23ac419 to 6d03eba Compare June 11, 2026 08:55
@evezhier

evezhier commented Jun 11, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run --disable-fail-fast

@evezhier evezhier changed the title [https://nvbugs/6157892][fix] [BREAKING] Mistral format refactor [https://nvbugs/6157892][fix] Mistral format refactor Jun 11, 2026
@evezhier

evezhier commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #58472 [ run ] triggered by Bot. Commit: 295c11a Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jul 9, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #58472 [ run ] completed with state SUCCESS. Commit: 295c11a
/LLM/main/L0_MergeRequest_PR pipeline #47081 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@evezhier

evezhier commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #58643 [ run ] triggered by Bot. Commit: 295c11a Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #58643 [ run ] completed with state FAILURE. Commit: 295c11a
/LLM/main/L0_MergeRequest_PR pipeline #47230 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

@evezhier

evezhier commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #58660 [ run ] triggered by Bot. Commit: 295c11a Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #58660 [ run ] completed with state FAILURE. Commit: 295c11a
/LLM/main/L0_MergeRequest_PR pipeline #47246 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

evezhier added 18 commits July 10, 2026 14:29
@evezhier
add HF preprocessor
ddcb4df 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
allow HF processor for Mistral checkpoints
3532b94 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
small fixes
bc69624 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
text-only path fix
8aaf4dc 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
text-only Mistral models fix
0cb1c41 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
unify naming
2596e16 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
Mistral processor disambiguation in tests
9a3b937 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
remove circular import
34e0576 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
revert arbitrary processor selection
ddfc0af 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
model_type fix in test
49aeeac 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
remove stale base processor
d37aae3 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
review fixes
eafbfdc 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
unwaive
7e260c1 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
hardcode mistral_common for native checkpoints + comments
97af0db 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
review
54f6938 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
review
8aba3e7 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
rebase issue
51b2c2f 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier
argparse fix
12ae9dd 
Signed-off-by: Olya Kozlova <okozlova@nvidia.com>
@evezhier

evezhier commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator Author

/bot run

@tensorrt-cicd

tensorrt-cicd commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #58663 [ run ] triggered by Bot. Commit: 12ae9dd Link to invocation

@tensorrt-cicd

tensorrt-cicd commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

PR_Github #58663 [ run ] completed with state FAILURE. Commit: 12ae9dd
/LLM/main/L0_MergeRequest_PR pipeline #47249 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

  • Please check the failed tests and fix your PR
  • If you cannot view the failures, ask the CI triggerer to share details
  • Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments
@hchings hchings hchings approved these changes
@Funatiq Funatiq Funatiq approved these changes
@jieli-matrix jieli-matrix jieli-matrix approved these changes
@2ez4bz 2ez4bz 2ez4bz approved these changes
@moraxu moraxu Awaiting requested review from moraxu moraxu is a code owner automatically assigned from NVIDIA/trt-llm-multimodal-devs
@Wanli-Jiang Wanli-Jiang Awaiting requested review from Wanli-Jiang Wanli-Jiang is a code owner automatically assigned from NVIDIA/trt-llm-multimodal-devs
@syuoni syuoni Awaiting requested review from syuoni syuoni is a code owner automatically assigned from NVIDIA/trt-llm-llmapi-devs
@jdebache jdebache Awaiting requested review from jdebache jdebache is a code owner automatically assigned from NVIDIA/trt-llm-torch-models-mistral-devs
@QiJune QiJune Awaiting requested review from QiJune QiJune is a code owner automatically assigned from NVIDIA/trt-llm-torch-attention-devs
@venkywonka venkywonka Awaiting requested review from venkywonka venkywonka is a code owner automatically assigned from NVIDIA/trt-llm-doc-owners

At least 1 approving review is required to merge this pull request.

Assignees

@evezhier evezhier

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@evezhier @tensorrt-cicd @hchings @Funatiq @jieli-matrix @2ez4bz