Commit 43021a7
feat: add comprehensive API and infrastructure cost tracking (#192)
Add a centralized, thread-safe CostTracker that records token usage
from every VLM/LLM API call and infrastructure time (GPU/VM hours).
The tracker is integrated at the vlm_call() level so all 15+ callers
automatically get cost tracking without any changes. Key integration
points have cost_label tags for per-component breakdown (planner,
grounder, vlm_judge, demo_verify, etc.).
- New openadapt_evals/cost_tracker.py with global singleton, pricing
tables, JSON persistence, and human-readable summary output
- vlm.py extracts response.usage tokens from both OpenAI and Anthropic
responses and reports to the tracker
- 18 unit tests covering pricing lookup, aggregation, thread safety,
persistence, and vlm.py integration
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>1 parent b89a15b commit 43021a7
12 files changed
Lines changed: 706 additions & 0 deletions
File tree
- openadapt_evals
- agents
- workflow/pipeline
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
315 | 315 | | |
316 | 316 | | |
317 | 317 | | |
| 318 | + | |
318 | 319 | | |
319 | 320 | | |
320 | 321 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
670 | 670 | | |
671 | 671 | | |
672 | 672 | | |
| 673 | + | |
673 | 674 | | |
674 | 675 | | |
675 | 676 | | |
| |||
733 | 734 | | |
734 | 735 | | |
735 | 736 | | |
| 737 | + | |
736 | 738 | | |
737 | 739 | | |
738 | 740 | | |
| |||
756 | 758 | | |
757 | 759 | | |
758 | 760 | | |
| 761 | + | |
759 | 762 | | |
760 | 763 | | |
761 | 764 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
| 59 | + | |
59 | 60 | | |
60 | 61 | | |
61 | 62 | | |
| |||
0 commit comments