fix: update context size precedence to prioritize backend configuration over model configuration

[ilopezluna]

This pull request updates the logic to ensure that the backend configuration for context size (or equivalent parameter) always takes precedence over the model configuration. This change standardizes runtime configuration behavior and updates both the implementation and associated tests to reflect this new precedence order.

Backend context size precedence updates:

• Changed the logic in llamacpp, sglang, vllm, and mlx backend config files so that the backend config value for context size/max tokens is used if present, falling back to the model config only if the backend config is unset. This affects functions such as GetContextSize, GetContextLength, and GetMaxModelLen.

• Updated function documentation in the above files to clarify that backend config takes precedence over model config for context size parameters.

Test updates for new precedence:

• Modified and added tests in llamacpp_config_test.go, sglang_config_test.go, and vllm_config_test.go to verify that backend config takes precedence over model config. Added new test cases to ensure fallback to model config when backend config is not set.

[gemini-code-assist]

Code Review

This pull request updates context size precedence across backends to prioritize runtime configuration over packaging defaults. While the logic is correctly implemented in most areas, the MLX backend remains incomplete as GetMaxTokens still returns nil. Additionally, the vLLM implementation requires a stricter validation check to ensure the model's context size is positive. The test suites have been appropriately updated to reflect these changes, providing good verification for the new precedence rules.

[sourcery-ai]

Hey - I've left some high level feedback:

• The compose CLI tests for context-size duplicate logic from RunE (including the int32 range check), which may get out of sync over time; consider extracting that validation into a small helper function used by both the command and the tests.
• In the compose up code, BackendConfiguration is always created even when the context-size flag is not set; if you want to clearly distinguish 'no runtime override' cases, you might consider only instantiating or sending a non-empty BackendConfiguration when at least one override (e.g., ContextSize or Speculative) is present.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The compose CLI tests for context-size duplicate logic from RunE (including the int32 range check), which may get out of sync over time; consider extracting that validation into a small helper function used by both the command and the tests.
- In the compose up code, BackendConfiguration is always created even when the context-size flag is not set; if you want to clearly distinguish 'no runtime override' cases, you might consider only instantiating or sending a non-empty BackendConfiguration when at least one override (e.g., ContextSize or Speculative) is present.

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}