ci(pricing): update LiteLLM snapshot automation#1186
Conversation
Update the locked LiteLLM flake input from e59e34bed3670a6894d43129c2af16af28057d03 to f27df8d516802ce4c1b32973992154fe83b851cf so Nix builds embed the latest upstream pricing data. Verified with the focused ccusage pricing test suite against the refreshed embedded pricing snapshot.
📝 WalkthroughWalkthroughThe workflow is updated to check for LiteLLM pricing changes more frequently. The schedule changes from weekly to every 12 hours, and the update job now compares pricing JSON before and after the flake update to avoid unnecessary PR creation when pricing data is unchanged. ChangesLiteLLM Pricing Update Workflow
Estimated code review effort🎯 2 (Simple) | ⏱️ ~8 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
@coderabbitai review |
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
ccusage-guide | 0add427 | Commit Preview URL Branch Preview URL |
May 29 2026, 11:15 AM |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
@ryoppippi I have started the AI code review. It will take a few minutes to complete. |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# Get PR details and diff
gh pr view 1186 --repo ryoppippi/ccusage --json title,body,files,additions,deletions,baseRefName,headRefName
echo "---DIFF---"
gh pr diff 1186 --repo ryoppippi/ccusageLength of output: 1481
Everything else ( [approve] |
ccusage
@ccusage/ccusage-darwin-arm64
@ccusage/ccusage-darwin-x64
@ccusage/ccusage-linux-arm64
@ccusage/ccusage-linux-x64
@ccusage/ccusage-win32-arm64
@ccusage/ccusage-win32-x64
commit: |
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runner startupExecution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one
Cached bunx execution performanceRuns the same large fixture through Fixtures: Claude
Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
Run the LiteLLM pricing update workflow every twelve hours instead of weekly so pricing changes are discovered sooner. Compare model_prices_and_context_window.json before and after the flake update, and restore flake.lock when the pricing JSON is unchanged so automation does not open lock-only PRs.
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runner startupExecution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one
Cached bunx execution performanceRuns the same large fixture through Fixtures: Claude
Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ryoppippi
changed the title
|
@coderabbitai review |
@ryoppippi I have started the AI code review. It will take a few minutes to complete. |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
gh pr view 1186 --repo ryoppippi/ccusage --json title,body,files,additions,deletions,baseRefName,headRefName,commits,state
echo "---DIFF---"
gh pr diff 1186 --repo ryoppippi/ccusageLength of output: 5554
|
| Field | Before | After |
|---|---|---|
rev |
e59e34bed3670a6894d43129c2af16af28057d03 |
f27df8d516802ce4c1b32973992154fe83b851cf |
lastModified |
1779231723 |
1780038305 |
narHash |
sha256-0MG6NXwV… |
sha256-os5zGH6k… |
All structural fields (owner, repo, type, flake: false) are unchanged. ✅
Overall: the implementation is correct and the intent is well-executed. No blocking concerns.
[approve]
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
.github/workflows/update-pricing.yaml (1)
18-23: ⚡ Quick winAdd retry/backoff to the pricing fetch.
Now that this runs every 12h, a transient 5xx or network blip from
raw.githubusercontent.comwill failcurl --failand abort the whole scheduled run. Add retries here and in the after-update fetch (Lines 30-32).♻️ Proposed resilience tweak
curl --fail --location --silent --show-error \ + --retry 3 --retry-delay 2 --retry-connrefused \ "https://raw.githubusercontent.com/BerriAI/litellm/${rev}/model_prices_and_context_window.json" \ --output /tmp/litellm-pricing-before.jsonApply the same
--retryflags to the fetch at Lines 30-32.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/workflows/update-pricing.yaml around lines 18 - 23, The current curl invocations used to "Capture current LiteLLM pricing JSON" (the fetch that writes /tmp/litellm-pricing-before.json) and the subsequent after-update fetch are brittle on transient network/5xx errors; update both curl commands to include retry/backoff flags (for example add --retry 5 --retry-delay 5 --retry-all-errors --retry-max-time 60 and a reasonable --connect-timeout like 10) while keeping the existing --fail/--location/--silent/--show-error and --output behavior so the workflow will retry transient failures instead of aborting immediately.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In @.github/workflows/update-pricing.yaml:
- Around line 18-23: The current curl invocations used to "Capture current
LiteLLM pricing JSON" (the fetch that writes /tmp/litellm-pricing-before.json)
and the subsequent after-update fetch are brittle on transient network/5xx
errors; update both curl commands to include retry/backoff flags (for example
add --retry 5 --retry-delay 5 --retry-all-errors --retry-max-time 60 and a
reasonable --connect-timeout like 10) while keeping the existing
--fail/--location/--silent/--show-error and --output behavior so the workflow
will retry transient failures instead of aborting immediately.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 22730103-39fe-4a7d-9850-8fd0e5dc0e25
⛔ Files ignored due to path filters (1)
flake.lockis excluded by!**/*.lock
📒 Files selected for processing (1)
.github/workflows/update-pricing.yaml
ccusage performance comparisonPR SHA: This compares the Rust PR release binary against the configured base package on the same CI runner. Package runner startupExecution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one
Cached bunx execution performanceRuns the same large fixture through Fixtures: Claude
Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
ccusage performance comparisonPR SHA: This compares the PR package against the configured base package on the same CI runner. Package runner startupExecution setup measures any pre-benchmark package materialization used by the execution benchmark. Bunx temp cache measures one
Cached bunx execution performanceRuns the same large fixture through Fixtures: Claude
Package runtime diagnosticsCompares the PR package wrapper, the installed native optional dependency binary, and the workspace release binary on the same large fixture. This identifies whether slow package results come from JavaScript wrapper overhead, the published native binary build, or the Rust core itself. Fixtures: Claude
Committed fixture performanceCommitted small fixtures for stable PR-to-PR feedback and explicit Claude/Codex command coverage. Fixtures: Claude
Large real-world-shaped fixture performanceGenerated fixtures shaped from aggregate local log statistics: thousands of JSONL files, many small sessions, and a long tail of larger sessions. No real prompts, paths, or outputs are stored in the fixtures. Fixtures: Claude
Artifact size
Lower medians and smaller artifacts are better. CI runner noise still applies; use same-run ratios as directional PR feedback, not release guarantees. |
1 participant
Updates the locked LiteLLM pricing input used by Nix builds.
Also changes the automated pricing update workflow to run every 12 hours and skip opening PRs when the LiteLLM pricing JSON is unchanged after the flake update.
Testing:
Summary by CodeRabbit