[Feature] CostGuard scorer skeleton#243
Merged
github-actions[bot] merged 2 commits intoJul 16, 2026
Merged
Conversation
davidbreitgand
force-pushed
the
costguard-scorer-skeleton
branch
from
July 15, 2026 19:06
bc46d2c to
7773044
Compare
Contributor
Author
|
/cc @shmuelk |
shmuelk
reviewed
Jul 16, 2026
shmuelk
reviewed
Jul 16, 2026
shmuelk
reviewed
Jul 16, 2026
shmuelk
reviewed
Jul 16, 2026
Signed-off-by: David Breitgand <davidbreitgand@users.noreply.github.com> Co-authored-by: Shmuel Kallner <kallner@il.ibm.com>
davidbreitgand
force-pushed
the
costguard-scorer-skeleton
branch
from
July 16, 2026 12:29
3f3c266 to
19739c5
Compare
Collaborator
|
/lgtm |
Signed-off-by: David Breitgand <davidbreitgand@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind feature
What this PR does / why we need it:
This PR is the first of a four-PR series that lands the
CostGuardscorer plugin (see README for rationale, algorithm, and design).Each PR compiles, runs, passes its own tests, and adds a bounded, well-scoped increment.
PR 1 (this PR): Plugin skeleton, config, factory, runner registration.
Lands the framework plumbing:
PluginType,Configwith full validation, factory, etc., and runner registration. Score returns a neutral0.5for every model, which is a valid "no opinion" state that keeps the scorer registerable and pipeline-safe.windowDurationparameter is parsed and validated but not consumed until PR 4 (marked with TODO in the code)Future PRs outline (for convenience of reference and scope of work:
PR2 and PR3 are two halves of the same algorithm that must land together (eventually) to be useful.
PR 2: Exploitation branch (rank + sigmoid scoring).
Introduces the deterministic ranking half of the algorithm: reads each model's
CostDigestfrom the datalayer, computes its intermediate rank, and applies a temperature-scaled sigmoid for obtaining the score in (0, 1). Models without cost history continue to receive the neutral0.5. PR2 is independently reviewable but not deployable in isolation since the scorer requires the exploration branch from PR 3 to bootstrap its own cost data collection.PR 3 — Exploration branch (epsilon-greedy).
Exploration branch (epsilon-greedy).
Adds the random exploration branch that drives sample collection on under-explored models. Together with PR 2, this completes the scoring algorithm. After PR 3 lands, the scorer is usable but not production-grade.
PR 4 — Epoch/window semantics (extractor owns the window).
Extends the
requestcostmetadataextractor to archive and reset its per-model t-digest eachwindowDuration, so published snapshots reflect only the current epoch. RemovesWindowDurationfrom the scorer's Config (the extractor becomes the single source of truth) and updates both plugin READMEs.Refs #214
Release note (write
NONEif no user-facing change):