feat: add NVIDIA NIM support for thinking models (Kimi-K2) - #11112
Conversation
- Auto-detect reasoning/thinking models that require R1 format - Add NVIDIA NIM specific parameters (chat_template_kwargs, reasoning_effort) - Add _isNvidiaNim method to detect NVIDIA NIM API endpoint - Support models with -thinking suffix (e.g., kimi-k2-thinking) This validates the user feedback that NVIDIA NIM works correctly with Kimi-K2 Thinking models when proper parameters are passed. Related to #10969
ghost
mentioned this pull request
Review completed. Found 2 issues to address:
Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| */ | ||
| private _isNvidiaNim(baseUrl?: string): boolean { | ||
| const urlHost = this._getUrlHost(baseUrl) | ||
| return urlHost.includes("api.nvidia.com") || urlHost.includes("nvidia.com") |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High
| */ | ||
| private _isNvidiaNim(baseUrl?: string): boolean { | ||
| const urlHost = this._getUrlHost(baseUrl) | ||
| return urlHost.includes("api.nvidia.com") || urlHost.includes("nvidia.com") |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High
| /** | ||
| * Check if the base URL is NVIDIA NIM API. | ||
| * NVIDIA NIM uses integrate.api.nvidia.com for their API endpoint. | ||
| * See: https://build.nvidia.com/docs/overview | ||
| */ | ||
| private _isNvidiaNim(baseUrl?: string): boolean { | ||
| const urlHost = this._getUrlHost(baseUrl) | ||
| return urlHost.includes("api.nvidia.com") || urlHost.includes("nvidia.com") | ||
| } |
There was a problem hiding this comment.
This PR adds new detection methods and parameters but lacks unit tests. The existing openai.spec.ts file has extensive test coverage for similar functionality (Azure, Grok, O3 models). Consider adding tests for:
_isNvidiaNim()detection method- NVIDIA NIM specific parameters (
chat_template_kwargs,reasoning_effort) isThinkingModeldetection logic- Extended
useR1Formatauto-detection
Fix it with Roo Code or mention @roomote and request a fix.
| // Add NVIDIA NIM specific parameters for thinking models | ||
| // See: https://build.nvidia.com/moonshotai/kimi-k2-thinking | ||
| ...(isNvidiaNim && isThinkingModel && { | ||
| chat_template_kwargs: { thinking: true }, | ||
| reasoning_effort: "high", | ||
| }), | ||
| } |
There was a problem hiding this comment.
This block is spread after ...(reasoning && reasoning) on line 179, so if a user has configured a different reasoning effort through their model settings, this hardcoded "high" value will silently override their preference. If NVIDIA NIM only supports "high" for thinking models, consider adding a comment explaining this constraint. Otherwise, consider using the user's configured value: reasoning_effort: reasoningEffort ?? "high".
Fix it with Roo Code or mention @roomote and request a fix.
2 participants
Summary
This PR attempts to address Issue #10969 by validating and implementing support for NVIDIA NIM with Kimi-K2 Thinking models.
Background
User @bozoweed provided detailed examples and documentation showing that NVIDIA NIM works correctly with Kimi-K2 Thinking models when the proper parameters are passed. This PR implements that support.
Changes
Auto-detection of reasoning/thinking models
-thinkingsuffix (e.g.,kimi-k2-thinking)deepseek-r1,deepseek/deepseek-r1)NVIDIA NIM specific parameters
When using NVIDIA NIM API (
integrate.api.nvidia.com) with thinking models, the following parameters are automatically added:These parameters are required for Kimi-K2 Thinking models on NVIDIA NIM as documented at:
New detection method
_isNvidiaNim()method to detect NVIDIA NIM API endpointsValidation
This implementation validates @bozoweed's findings that NVIDIA NIM works correctly with:
tool_choice: autoortool_choice: requiredreasoning_contentparsing from responsestool_call_idvaluesFiles Changed
src/api/providers/openai.ts- Added NVIDIA NIM detection and thinking model parametersFeedback and guidance are welcome!
Important
Adds NVIDIA NIM support for Kimi-K2 Thinking models by auto-detecting model types and adding specific parameters in
openai.ts.OpenAiHandler.chat_template_kwargs: { thinking: true }andreasoning_effort: "high"for thinking models._isNvidiaNim()method to detect NVIDIA NIM API endpoints inopenai.ts.This description was created by
for 2075e1b. You can customize this summary. It will automatically update as commits are pushed.