fix(platform): drop temperature for models that reject it - #1834
Conversation
LLM-as-judge evals on claude-sonnet-5 failed with HTTP 400 "`temperature` is deprecated for this model" while agent executions on the same model passed. LLM Gateway discovery publishes modelDetails.shouldSkipTemperature per model. uipath-langchain-python honors it, so agent runs omit the parameter. Both chat_completions paths here always sent it, so every judge call on such a model failed. In staging, all 137 normalized-route 400s over 24h carried x-uipath-agenthub-config: agentsevals; none came from agent execution. Resolve the flag from discovery (cached per base_url + agenthub config) and omit temperature when the model reports it. Unknown models and unreachable discovery keep the caller's parameters, so behaviour is unchanged wherever the flag is not set. Discovery is resolved through EndpointManager so it follows the same agenthub_/orchestrator_ routing as the completion it describes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018nEN5r7dtNaEkN9y5ygXCT
Ship the temperature fix and make it unskippable downstream: raise uipath's lower bound on uipath-platform to 0.2.14 so no install of uipath can resolve a uipath-platform that still sends temperature to models that reject it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018nEN5r7dtNaEkN9y5ygXCT
github-actions
Bot
added
test:uipath-langchain
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 29fc56c45b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if key not in _model_details: | ||
| _model_details[key] = await _fetch_model_details(service, agenthub_config) |
There was a problem hiding this comment.
When the first discovery request for a cache key is transiently unavailable or returns an invalid payload, _fetch_model_details returns {}, which is stored here as a successful process-lifetime result. Every later call for a model such as Sonnet 5 therefore keeps sending the rejected temperature parameter even after discovery recovers, so a brief outage can break all subsequent completions until process restart. Avoid caching failed/invalid reads, or expire and retry them.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
This PR updates the UiPath Python SDK’s LLM Gateway clients to honor a per-model discovery capability (modelDetails.shouldSkipTemperature) so requests omit temperature for models that reject it (e.g., claude-sonnet-5), preventing 400s in eval/judge flows while preserving fail-open behavior when discovery is unavailable.
Changes:
- Add a cached discovery-based capability resolver and use it to conditionally drop
temperaturein both normalized and OpenAI-compatiblechat_completionspaths. - Extend endpoint routing support with discovery endpoints (
agenthub_/...andorchestrator_/...) viaEndpointManager. - Add regression tests for discovery-driven temperature skipping and bump
uipath-platform/uipathversions + dependency floor to ensure the fix is pulled in.
Reviewed changes
Copilot reviewed 7 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| packages/uipath/uv.lock | Updates locked versions for uipath and uipath-platform. |
| packages/uipath/tests/cli/eval/mocks/test_input_mocker.py | Updates HTTPX mocks to account for the new discovery call in eval input generation. |
| packages/uipath/pyproject.toml | Bumps uipath version and raises minimum uipath-platform to include the fix. |
| packages/uipath-platform/uv.lock | Updates locked version for uipath-platform. |
| packages/uipath-platform/tests/services/test_llm_temperature_skip.py | Adds coverage for discovery-driven temperature omission (including caching and agenthub-config scoping). |
| packages/uipath-platform/src/uipath/platform/common/_endpoints_manager.py | Adds discovery endpoint constants and a selector method consistent with other LLM endpoints. |
| packages/uipath-platform/src/uipath/platform/chat/_model_capabilities.py | Implements cached discovery fetch + should_skip_temperature() helper (fail-open). |
| packages/uipath-platform/src/uipath/platform/chat/_llm_gateway_service.py | Uses discovery capability to omit temperature for models that reject it (both chat paths). |
| packages/uipath-platform/pyproject.toml | Bumps uipath-platform version to ship the fix. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Chat completions consults discovery to learn which parameters the model accepts. | ||
| httpx_mock.add_response( | ||
| url="https://example.com/llm/api/discovery", | ||
| status_code=200, | ||
| json=[{"modelName": "gpt-4o-mini-2024-07-18", "modelDetails": {}}], | ||
| ) |
|
|
||
| # Reasoning models (o1, o3, o4) don't support temperature; newer models | ||
| # reject it too, which only discovery knows about. | ||
| skip_temperature = is_reasoning_model or await should_skip_temperature( |
There was a problem hiding this comment.
should we remove is_reasining_model check here ?
There was a problem hiding this comment.
Good question, but I'd keep it — and it can't be deleted outright anyway.
Two reasons to keep it in the or:
-
should_skip_temperaturefails open by design. It returnsFalsefor an unknown model, unreachable discovery, or any unexpected exception. Ifis_reasoning_modelcomes out of the expression, a discovery outage means o1/o3/o4 start receivingtemperatureagain — reintroducing exactly the 400 this PR exists to fix, for models that were previously safe. It gets worse combined with the caching behaviour I flagged separately:_details_forcaches the failure, so one blip at process start would keep reasoning models 400ing for the whole process lifetime. -
It short-circuits. Python
ormeans a known reasoning model never makes the discovery call at all — no round-trip, no latency, no dependency.
There's also an unverified assumption buried in the removal: that gateway discovery actually reports shouldSkipTemperature: true for o1/o3/o4. Worth checking empirically, but even if it does, point 1 still argues for keeping the local guard as the deterministic floor.
And separately — the variable itself is load-bearing elsewhere: line 584, if not is_claude_model and not is_reasoning_model:, gates n, frequency_penalty, presence_penalty and top_p. So is_reasoning_model has to stay regardless; the only question was whether to drop it from the skip_temperature expression.
Net: the local check is a cheap deterministic floor, discovery is the extension for models we can't name ahead of time. Belt and braces is right here.
|
Auto-merge enabled (approved, checks green after a branch update from One follow-up that should land before this is cherry-picked to a release branch, flagged in review and not blocking the merge to
The module comment does name the behaviour ("an empty mapping caches 'couldn't read discovery'"), so it reads as deliberate — but the consequence for a hotfix aimed at exactly that 400 is worth a second look. Two small options: only cache successful results, or give cached entries a TTL. Merging anyway because it is a strict improvement on the status quo (today Sonnet 5 is 100% broken; after this it is broken only after a transient discovery failure). Raising it here so it is not lost before the release cherry-pick. |
|
Problem
LLM-as-judge evals on
claude-sonnet-5fail, while agent executions on the same model succeed:Root cause
LLM Gateway discovery publishes a per-model capability flag,
modelDetails.shouldSkipTemperature.uipath-langchain-python(chat_model_factory.py) resolveseffective_temperature = None if _should_skip_temperature(model_info) else temperature, so agent calls omit the parameter and pass.chat_completionspaths puttemperaturein the body unconditionally, so every judge call on such a model 400s.grep shouldSkipTemperatureacross the repo returned nothing before this PR.The existing guards were model-name heuristics (
"claude" in model,startswith(("o1","o3","o4"))), so each new model family breaks a path until someone adds another string match. Nothing regressed in this repo — Sonnet 5 simply arrived in discovery with the flag set, and the judge violated a contract it never read.Staging telemetry over 24h: all 137 normalized-route 400s carried
x-uipath-agenthub-config: agentsevals. None came from agent execution. In one trace the LangGraph agent completed againstanthropic.claude-sonnet-5(0 of 47 calls sent temperature) while the evaluator's call in the same trace 400'd.Fix
_model_capabilities.should_skip_temperature()resolves the flag from discovery, cached per(base_url, agenthub_config)— one GET per process, not per call. Bothchat_completionspaths omittemperaturewhen the model reports it.Fail-open by design: unknown models, unreachable discovery, and any unexpected error keep the caller's parameters, so behavior is unchanged everywhere the flag is not set. A discovery outage can never fail an LLM call.
Discovery resolves through
EndpointManager.get_discovery_endpoint()rather than a hardcoded path, because the gateway is reachable viaagenthub_/ororchestrator_/.Tests
tests/services/test_llm_temperature_skip.py— 7 cases: flag set / flag absent / model absent from discovery / discovery unreachable / cached across calls / scoped by agenthub config / OpenAI-compatible path.Three fail against pre-fix code on the exact assertion:
uipath-platformuipathtest_input_mocker.pygained a discovery mock: it asserts every HTTP request is expected, and the code now legitimately calls discovery. That is the only test in the repo coupled to the request set.Releases
uipath-platform0.2.13 → 0.2.14 (the fix),uipath2.13.18 → 2.13.19.uipath's floor moves touipath-platform>=0.2.14so no install ofuipathcan resolve auipath-platformthat still sends the parameter. Without it the constraint (>=0.2.4) would let a stale lockfile keep the broken version — which is how an earlier fix on this same path shipped but never reachedpython-eval-worker. Bothuv.lockfiles regenerated;check_version_uniquenessandcheck_dependency_version_bumpspass locally.Follow-up (not in this PR)
agents/backend/External.Clients/LlmGateway/LlmGatewayClient.cs:274sendsTemperature = settings?.Temperature ?? 0.0fon the same normalized route, andLlmModelMapperdropsShouldSkipTemperatureoff the discovery DTO even thoughDiscoveryModels.cs:36defines it. No 400s from that path yet — no low-code agent has run on a skip-temperature model — but it will break the first time one does.AgentHubService's_available_models_spechardcodes/agenthub_/llm/api/discovery, so it breaks in orchestrator-routed deployments. Left alone to keep this diff surgical.🤖 Generated with Claude Code
https://claude.ai/code/session_018nEN5r7dtNaEkN9y5ygXCT