[OPIK-6995] [FS] feat: add per-evaluation spend budget for LLM-as-judge evaluators#7312
Merged
Merged
Conversation
…ge evaluators Add an optional per-rule USD spend budget for agentic (trace + thread) LLM-as-judge online evaluators. The budget is a soft wrap-up trigger, not a hard ceiling: once cumulative spend reaches the limit, the agentic tool loop stops starting new turns and emits a best-effort verdict from the data gathered so far (spend may overshoot by the wrap-up call, by design). - BudgetGuard encapsulates the limit, running spend, and all cost math; spend is computed in-process via CostService (no provider round-trip) from each response's token usage. A non-positive limit degrades to no-limit. - LlmUsageExtractor (extracted from EvaluationEntityFactory) builds the cache-aware usage map for any provider Opik prices. - When the budget trips, the monitoring trace is tagged `budget_exceeded` and the wrap-up call uses a distinct instruction (best-effort from partial data rather than "investigation complete"). - Span evaluators are excluded: a single LLM call has no loop to wrap up. - Frontend: optional "Max cost per evaluation (USD)" field on trace/thread rules (hidden for span scope). Implements OPIK-6995. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
github-actions
Bot
added
java
![baz-reviewer[bot]](https://avatars.githubusercontent.com/in/933528?s=60&v=4){kind=small}
alexkuzmik
marked this pull request as ready for review
July 3, 2026 11:21
…atural stop Code-review follow-up on the spend-budget change: - ToolCallLoop: check the no-tool-calls terminal case BEFORE the round/budget gate. Previously a follow-up response that both finished naturally (no tool calls) and tipped spend over the budget hit the budget gate first and returned without appending the terminal AiMessage, so runWithWrapUp re-issued the structured request missing the model's final reasoning turn. Regression test added (appendsTerminalAiMessageWhenBudgetTripsOnANaturalStop). - LlmUsageExtractorTest: cover the null/zero cache-token guard branch (the value>0 check that keeps cost unchanged when caching is off). - FE LLMJudgeRuleDetails: guard event.target.valueAsNumber against NaN so an intermediate unparseable numeric input can't write NaN into maxCostUsd. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Contributor
…rs, drop min
Max cost per evaluation (USD) field:
- Block symbols type=number would otherwise accept (comma, sign, exponent) via
onKeyDown so only a positive decimal can be entered.
- Hide the up/down spinner arrows (webkit inner/outer + Firefox textfield).
- Remove the min={0.000001} attribute so the field no longer surfaces that value
by default; positivity is still enforced by the Zod .positive() rule.
- Use cn() for the conditional className (was string concatenation).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…evel, FE input
- ToolCallLoop: flag budget_exceeded on the recorder at the point the budget gate
actually fires (agentic loop), instead of inferring it post-hoc from
shouldWrapUp() in the scorers. Fixes two issues: a natural stop that merely
crossed spend is no longer mislabelled, and the tag survives a downstream error
(it's set before the wrap-up/scoring chain can throw). Removed the inferred
flagBudgetExceeded() calls from both scorers' map().
- ToolCallLoop: demote the internal "budget reached" log to debug (the scorer's
user-facing warn is the primary signal). Tests added: flag set on mid-loop
budget trip, not set on a natural under-budget stop.
- FE LLMJudgeRuleDetails: use FormLabel + FormDescription (forms.md) so the
label/input association and error styling are wired; reject non-plain-decimal
onChange input so pasted/IME exponent ("1e3") can't bypass the decimal-only rule.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
awkoy
previously approved these changes
Jul 3, 2026
…r transforms The two transforms that rebuild LlmAsJudgeCode (renameSchemaToAssertionKeys, applyTestSuitePrompt) used the positional constructor and dropped the new maxCostUsd field, so a test-suite LLM-judge evaluator configured with a spend budget was silently scored with BudgetGuard.UNLIMITED. Rebuild via toBuilder() so maxCostUsd — and any field added later — is carried through. Regression tests added (preserves maxCostUsd / leaves it null when unset). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-evaluation-spend-budget Reconcile the spend-budget feature with main's extraction of the tool-call loop into AgenticScoringService.runToolCallLoop: thread BudgetGuard through runToolCallLoop, pass the per-evaluation guard from the trace/thread scorers and BudgetGuard.UNLIMITED from the span scorer (budget-excluded). Frontend: keep the maxCostUsd field's cn import alongside main's renamed RESERVED_*_LLM_JUDGE_VARIABLES constants. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
⏱️ pre-commit per-hook timing
⏭️ 38 skipped (no matching files changed)
|
BorisTkachenko
left a comment
BorisTkachenko
left a comment
Contributor
There was a problem hiding this comment.
Opik reviewer (mined from your team's review history)
13 findings — 0 high · 2 medium · 11 low. Suppressed by team conventions: see suppressed.md.
React 👍/👎 on each comment — your feedback helps tune what it flags.
- Enforce @positive max_cost_usd at the API boundary: cascade validation into the polymorphic evaluator code record via @Valid (create + update paths), so a non-positive value is rejected with 422 instead of silently coerced to "no limit". Teach PODAM to manufacture strictly-positive values for @positive BigDecimal fields so the newly-enforced constraint doesn't flake create tests. - Unify the budget signals off one authoritative event: add BudgetGuard.wasBudgetEnforced(), set at the tool loop's budget gate alongside the budget_exceeded trace tag, and drive the wrap-up instruction + user-facing warn off it. A natural stop or inline single call that merely crosses spend is no longer mislabelled as a budget cut-off. - Warn once per guard when a limited guard resolves an unpriced model (cost stays zero, so the limit is silently ineffective). - MDC-wrap the budget-reached log line so it carries workspace_id / rule_id. - Reword the span scorer's UNLIMITED rationale (it does run the agentic loop; the real reason is the span DTO has no max_cost_usd field). - Test hardening: derive expected cost from CostService instead of hardcoding, drop dead mock stubbing in the unlimited paths, cover the charge() fail-open branch, assert a distinct total-token pass-through, split the bundled cache test, rename overBudgetGuard -> tightBudgetGuard. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Backend Tests - Integration Group 6181 tests 181 ✅ 3m 17s ⏱️ Results for commit bab3236. ♻️ This comment has been updated with latest results. |
Comment on lines
349
to
+353
| try (var logContext = wrapWithMdc(mdc)) { | ||
| if (costGuard.wasBudgetEnforced()) { | ||
| // Keyed on wasBudgetEnforced() (the loop's budget gate actually cut the run | ||
| // short), not shouldWrapUp() (mere spend >= limit): this warn claims the | ||
| // investigation was stopped, so it must not fire on the inline single-call path |
Contributor
There was a problem hiding this comment.
Duplicated budget warning logic
The if (costGuard.wasBudgetEnforced()) … warn block is duplicated in OnlineScoringTraceThreadLlmAsJudgeScorer.evaluate(), so any tweak to the budget-warning wrap-up or user-facing string means editing both scorers — should we extract a shared helper like logBudgetWarning(...)?
Want Baz to fix this for you? Activate Fixer
Three real bugs surfaced by the code-aware review:
1. Budget silently never enforced for Gemini/Vertex/OpenRouter judges. The
online-scoring provider is the LlmProvider value ("gemini", "vertex-ai"),
but CostService keys prices by the canonical provider ("google_ai",
"google_vertexai"/"anthropic_vertexai"), so the lookup missed and cost
resolved to zero. findModelPrice now canonicalizes the incoming provider and
tries candidate namespaces (Vertex hosts both Google and Anthropic models, so
both are tried and the model's own rows decide). Fixes it for every
cost-tracked provider and also fixes the monitoring trace's estimated cost;
backward compatible for callers already passing canonical names. OpenRouter
stays unpriced (no canonical namespace) and is surfaced by the unpriced warn.
2. Decimal entry impossible in the "Max cost per evaluation" field. A controlled
type=number bound to the committed number erased an in-progress trailing "."
("1." committed as 1 and re-rendered "1"). Extracted a MaxCostInput that keeps
the raw text as the display source of truth and commits the parsed number
separately, re-syncing only on external value changes.
3. Global @positive -> BigDecimalStrategy PODAM registration would throw for any
manufactured @positive int/long/Double field. Moved the positive-floor logic
into BigDecimalTypeManufacturer (scoped to BigDecimal fields), deriving the
floor from ValidationUtils.SCALE, and dropped the global attribute strategy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…al-eval create test Drop the CostService provider-canonicalization added while addressing an earlier review round. The online evaluator now hands its resolved LlmProvider value straight to CostService: OpenAI/Anthropic/Bedrock judges price on an exact provider match, and any provider the price table doesn't cover (Gemini/Vertex/ OpenRouter/custom-llm/...) resolves to no cost — surfaced by BudgetGuard's existing once-per-guard unpriced warn. No RUNTIME_PROVIDER_CANONICALS map, no Vertex dual-namespace guessing, no candidate-provider fallback loop; CostService is back to its main behavior, so all other cost callers are unchanged. Removes the two canonicalization tests tied to that logic. Fix ManualEvaluationResourceTest: the @Valid cascade on the evaluator code field now enforces @notempty arguments on the hand-built span UserDefinedMetricPython code (previously only set metric), so createEvaluator returned 422. Supply valid arguments, matching the canonical AutomationRuleEvaluatorsResourceTest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d judge is cost-tracked
Online evaluation resolves models to LlmProvider serialized values ("gemini",
"vertex-ai") that differ from the canonical price-table vocabulary ("google_ai",
"google_vertexai"), so cost tracking and the spend budget silently priced those
judges at zero. Add a small, single-valued RUNTIME_PROVIDER_MAPPING applied at
lookup in CostService — a deterministic name normalization, no candidate-tries
loop and no fallback. Vertex is unambiguous because only Gemini models are
offered on Vertex for online evaluation. openai/anthropic/bedrock already equal
their canonical names; self-hosted ollama/custom-llm have no public pricing to
map to and stay at zero. Canonical names pass through unchanged, so all other
cost callers are unaffected.
Covered by CostServiceTest (gemini -> google_ai, vertex_ai/gemini -> google_vertexai
both price > 0) and BudgetGuardTest (a Gemini judge now accrues spend and can trip
the budget).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
Python SDK E2E Tests Results (Python 3.10)286 tests 284 ✅ 5m 33s ⏱️ Results for commit 7db60c1. ♻️ This comment has been updated with latest results. |
BorisTkachenko
previously approved these changes
Jul 9, 2026
BorisTkachenko
left a comment
BorisTkachenko
left a comment
Contributor
There was a problem hiding this comment.
A few minor comments.
| * {@link #shouldWrapUp()} true on the first turn and silently wrap up every evaluation with no real | ||
| * work — worse than ignoring it. | ||
| */ | ||
| public static BudgetGuard create(BigDecimal maxCostUsd, String modelName, |
Contributor
There was a problem hiding this comment.
Suggested change
| public static BudgetGuard create(BigDecimal maxCostUsd, String modelName, | |
| public static BudgetGuard create(BigDecimal maxCostUsd, @NonNull String modelName, |
| } | ||
|
|
||
| /** Taps the LLM call to charge its cost, returning the response unchanged. Pass-through when unlimited. */ | ||
| public Mono<ChatResponse> track(Mono<ChatResponse> call) { |
Contributor
There was a problem hiding this comment.
Suggested change
| public Mono<ChatResponse> track(Mono<ChatResponse> call) { | |
| public Mono<ChatResponse> track(@NonNull Mono<ChatResponse> call) { |
| * per-provider switch below only <i>adds</i> the prompt-cache token counts, which each SDK exposes | ||
| * under a different subtype. | ||
| */ | ||
| public static Map<String, Integer> toUsageMap(ChatResponse response) { |
Contributor
There was a problem hiding this comment.
Suggested change
| public static Map<String, Integer> toUsageMap(ChatResponse response) { | |
| public static Map<String, Integer> toUsageMap(@NonNull ChatResponse response) { |
Address review: annotate the non-nullable parameters flagged on the public entry points — BudgetGuard.create(modelName), BudgetGuard.track(call), and LlmUsageExtractor.toUsageMap(response). maxCostUsd stays nullable (null = no limit). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…-spend-budget' into aliaksandrk/OPIK-6995-evaluation-spend-budget
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details
Adds an optional per-rule USD spend budget for agentic (trace + thread) LLM-as-judge online evaluators. Long/large inputs push the agentic
ToolCallLoop(up to 10 rounds, re-sending context each turn) into unbounded token spend with no ceiling — the only signal today is the provider bill after the fact (OPIK-6995).The budget is a soft wrap-up trigger, not a hard ceiling. Cumulative spend is tracked as the evaluation runs; the moment it reaches the limit, the agent stops starting new turns and does one final tools-stripped call to emit a best-effort verdict from the data already gathered. Spend may overshoot the limit slightly (by that wrap-up call) — by design. No pre-estimation, no hard kill.
Key pieces:
BudgetGuardencapsulates the limit, running spend, and all cost math, so neither the scorers norToolCallLooptouch pricing — they onlytrack()each LLM call and askshouldWrapUp()between turns. Cost is computed fully in-process viaCostService(no provider round-trip) from each response's token usage. Fail-open (a pricing/resolution failure degrades to no-limit); when a limited guard resolves an unpriced model (cost stays zero, so the limit can never trip) it warns once so the ineffective limit is operator-visible. Non-positive limits are now rejected at the API boundary (@Positiveonmax_cost_usdis@Valid-cascaded into the evaluatorcoderecord → 422); the guard keeps the non-positive→no-limit coercion as defense-in-depth for any legacy persisted row.LlmUsageExtractor(extracted fromEvaluationEntityFactory, now the single source of truth) builds the cache-aware usage map for any provider Opik prices — base prompt/completion/total plus provider-specific cache tokens (Anthropic / OpenAI / OpenAI-official / Gemini).budget_exceededtrace tag, the budget-specific wrap-up instruction, and the user-facing warn all key off one authoritative signal —BudgetGuard.wasBudgetEnforced(), set at the exact point the tool loop's budget gate abandons pending tool calls. So a natural stop (or an inline single call) that merely crosses the spend limit is not mislabeled as a budget cut-off: it keeps the normal "investigation complete" wrap-up, emits no warn, and is not tagged. When the budget genuinely cuts a run short, the hiddenonline_evaluationmonitoring trace is taggedbudget_exceeded(on both the success and error finalize paths) so users can filter/count those evaluations, and the wrap-up call uses the distinct instruction — "budget reached, give a best-effort verdict from partial data."max_cost_usdfield (unlike trace/thread), so there is no per-evaluation budget to enforce — the span path passesBudgetGuard.UNLIMITEDeven though it can run the same agentic loop.Stored in the evaluator
codeJSON column, so there is no DB migration; existing rows deserializenull= no limit (behavior unchanged).Change checklist
Issues
Testing
BudgetGuardTest(accumulation with cost derived fromCostService, limit trip, non-positive→unlimited, fail-open incl. a pricing-exception case, unknown-price never trips),LlmUsageExtractorTest(base with a distinct pass-through total + Anthropic/OpenAI/OpenAI-official/Gemini cache tokens + null, cache-omission branches split per case),ToolCallLoopTest(early wrap-up once budget reached; budget-specific vs. normal wrap-up instruction; natural-stop-over-budget keeps the completion instruction and is not flagged),OnlineEvaluationRecorderTest(budget_exceededtag on success and error paths;monitor()pass-through preserved),AutomationRuleEvaluatorMaxCostMappingTest(trace + thread + null API↔domain round-trip).AutomationRuleEvaluatorsResourceTestasserts a non-positivemax_cost_usdis rejected with 422 (the@Validcascade); PODAM now manufactures strictly-positive values for@PositiveBigDecimalfields so the newly-enforced constraint doesn't flake the create round-trips (all six subtypes still round-trip).has_tool_spans=false), best-effort score still stored, monitoring trace taggedbudget_exceeded, and the wrap-up span carried the budget-specific instruction. Spend/limit reflected in the rule logs and the monitoring trace'stotal_estimated_cost.Documentation
N/A — no documentation changes in this PR.