What would you like to be added:
Add specialized metadatadata extractor plugin to the datalayer plugins to extract input/output tokens info from theusage metadata field on a response path (triggered by response event) -> convert to cost using model pricing attributes -> update t-digest of the model on the model's AttributeMap
Why is this needed:
This extractor is needed to implement CostGuard actual inference cost-aware model scorer according to PR #22 .
What would you like to be added:
Add specialized metadatadata extractor plugin to the datalayer plugins to extract input/output tokens info from the
usagemetadata field on a response path (triggered by response event) -> convert to cost usingmodelpricing attributes -> update t-digest of themodelon themodel'sAttributeMapWhy is this needed:
This extractor is needed to implement
CostGuardactual inference cost-aware model scorer according to PR #22 .