requestcostmetadata: add cost-sample extractor publishing per-model t-digests#43
Open
davidbreitgand wants to merge 3 commits into
Open
requestcostmetadata: add cost-sample extractor publishing per-model t-digests#43davidbreitgand wants to merge 3 commits into
davidbreitgand wants to merge 3 commits into
Conversation
|
Your PR is large. Please consider breaking it into multiple PRs. The |
ronenkat
reviewed
Jun 14, 2026
davidbreitgand
commented
Jun 14, 2026
davidbreitgand
left a comment
davidbreitgand
left a comment
Contributor
Author
There was a problem hiding this comment.
@ronenkat Please take another look at the comments, If we agree, I'll proceed with the fixes.
Signed-off-by: David Breitgand <davidbreitgand@users.noreply.github.com>
Signed-off-by: David Breitgand <davidbreitgand@users.noreply.github.com>
Signed-off-by: David Breitgand <davidbreitgand@users.noreply.github.com>
davidbreitgand
force-pushed
the
requestcostmetadata-tdigest
branch
from
521861d to
36db767
Compare
|
Your PR is large. Please consider breaking it into multiple PRs. The |
Contributor
Author
|
@ronenkat please take another look |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
kind feature
What this PR does / why we need it:
Adds the
requestcostmetadataextractor plugin: on eachResponseEventTypeevent, it readsprompt_tokens/completion_tokensfrom the response'susageobject, looks up the model'spricing.TokenPrices, computes the per-request actual cost, and adds it into a per-model runningt-digest.At the end of each batch (i.e., the flush interval has elapsed), models that were updated during the flush interval get a digest snapshot published to their
AttributeMapunderpricing.CostDigestAttributeKey.No epoch handling — the digest accumulates without bound. Epoch boundary semantics will be added in a followup PR.
No warmup counter in
CostDigestorrequestcostmetadata. Will be added in a followup PR.Part of the
CostGuardimplementation track (proposal). Roadmap items #2 (partially).README.md added that documents the plugin + lists known limitations.
The code also have several TODO comments that spill out of the scope of this PR. They will be captured as issues and handled in the separate PRs.
Partially fixes Issue #35