Skip to content

Commit 787189f

Browse files
rdimitrovclaude
andcommitted
Record per-skill turns/cost and surface in PR body + summary
Adds two capture steps that parse claude-code-action's execution log (`/home/runner/work/_temp/claude-execution-output.json`) right after each skill invocation, BEFORE the next one overwrites the shared file. Exposes `turns`, `cost_usd`, and `permission_denials` as step outputs for downstream use. Surfaces the data in two places: 1. PR body: new "Run cost" subsection inside the upstream-release- docs marker block. Per-session rows plus a Total row when both sessions reported. Applies to both pull_request and workflow_dispatch runs. 2. workflow_dispatch summary comment: adds Turns and Cost columns to the existing step table, plus a Total row summing both sessions. Useful for tracking per-release spend ($6 baseline) and catching regressions -- e.g. a release that suddenly takes 10x the turns is visible at a glance rather than requiring a drill-down into the Actions Step Summary. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 74b532c commit 787189f

1 file changed

Lines changed: 100 additions & 6 deletions

File tree

.github/workflows/upstream-release-docs.yml

Lines changed: 100 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -641,6 +641,35 @@ jobs:
641641
NO_CHANGES.md at repo root with a one-line explanation.
642642
Still do not hand-edit any file.
643643
644+
# Capture skill_gen's execution stats BEFORE skill_review runs
645+
# and overwrites the shared execution-output JSON at the
646+
# canonical claude-code-action location. Lets us report
647+
# per-invocation turns/cost in the PR body and the workflow_
648+
# dispatch summary comment. Missing-file defaults to 0 so a
649+
# failed run still emits plausible outputs.
650+
- name: Capture skill_gen stats
651+
id: skill_gen_stats
652+
if: always() && steps.skill_gen.conclusion == 'success'
653+
run: |
654+
LOG="/home/runner/work/_temp/claude-execution-output.json"
655+
if [ -f "$LOG" ]; then
656+
TURNS=$(jq -r '.num_turns // 0' "$LOG")
657+
COST=$(jq -r '.total_cost_usd // 0' "$LOG")
658+
DENIALS=$(jq -r '.permission_denials_count // 0' "$LOG")
659+
else
660+
TURNS=0
661+
COST=0
662+
DENIALS=0
663+
fi
664+
# Format cost with 4 decimal places for readability.
665+
COST_FMT=$(printf '%.4f' "$COST")
666+
{
667+
echo "turns=$TURNS"
668+
echo "cost_usd=$COST_FMT"
669+
echo "denials=$DENIALS"
670+
} >> "$GITHUB_OUTPUT"
671+
echo "skill_gen stats: turns=$TURNS cost=\$$COST_FMT denials=$DENIALS"
672+
644673
# Invocation 2: editorial re-review with FRESH CONTEXT. Running
645674
# docs-review in a separate session — with no exposure to the
646675
# generation session's internal reasoning — tends to catch style
@@ -695,6 +724,30 @@ jobs:
695724
if they exist -- they're signal files handed off to the
696725
next workflow step, not part of the docs.
697726
727+
# Mirror of skill_gen_stats for skill_review. Reads the same
728+
# canonical log path, which skill_review overwrote on exit.
729+
- name: Capture skill_review stats
730+
id: skill_review_stats
731+
if: always() && steps.skill_review.conclusion == 'success'
732+
run: |
733+
LOG="/home/runner/work/_temp/claude-execution-output.json"
734+
if [ -f "$LOG" ]; then
735+
TURNS=$(jq -r '.num_turns // 0' "$LOG")
736+
COST=$(jq -r '.total_cost_usd // 0' "$LOG")
737+
DENIALS=$(jq -r '.permission_denials_count // 0' "$LOG")
738+
else
739+
TURNS=0
740+
COST=0
741+
DENIALS=0
742+
fi
743+
COST_FMT=$(printf '%.4f' "$COST")
744+
{
745+
echo "turns=$TURNS"
746+
echo "cost_usd=$COST_FMT"
747+
echo "denials=$DENIALS"
748+
} >> "$GITHUB_OUTPUT"
749+
echo "skill_review stats: turns=$TURNS cost=\$$COST_FMT denials=$DENIALS"
750+
698751
# Count the commits the skill itself added between pre_skill
699752
# and now. Zero commits means skill_gen and skill_review both
700753
# concluded there was nothing to change -- e.g. because main
@@ -885,6 +938,10 @@ jobs:
885938
SKILL_COMMIT_COUNT: ${{ steps.skill_commits.outputs.count }}
886939
GEN_CONCLUSION: ${{ steps.skill_gen.conclusion }}
887940
REVIEW_CONCLUSION: ${{ steps.skill_review.conclusion }}
941+
GEN_TURNS: ${{ steps.skill_gen_stats.outputs.turns }}
942+
GEN_COST: ${{ steps.skill_gen_stats.outputs.cost_usd }}
943+
REVIEW_TURNS: ${{ steps.skill_review_stats.outputs.turns }}
944+
REVIEW_COST: ${{ steps.skill_review_stats.outputs.cost_usd }}
888945
run: |
889946
START='<!-- upstream-release-docs:start -->'
890947
END='<!-- upstream-release-docs:end -->'
@@ -966,6 +1023,29 @@ jobs:
9661023
echo "No non-bot contributors were found in the release range."
9671024
echo ""
9681025
fi
1026+
# Per-invocation cost/turns from each skill session's
1027+
# claude-execution-output.json. Useful for tracking per-run
1028+
# spend and noticing regressions (e.g. a release that
1029+
# suddenly takes 10x the turns).
1030+
if [ -n "$GEN_TURNS" ] || [ -n "$REVIEW_TURNS" ]; then
1031+
echo "### Run cost"
1032+
echo ""
1033+
echo "| Session | Turns | Cost (USD) |"
1034+
echo "| --- | ---: | ---: |"
1035+
if [ -n "$GEN_TURNS" ]; then
1036+
echo "| Generation (\`skill_gen\`) | $GEN_TURNS | \$$GEN_COST |"
1037+
fi
1038+
if [ -n "$REVIEW_TURNS" ]; then
1039+
echo "| Editorial review (\`skill_review\`) | $REVIEW_TURNS | \$$REVIEW_COST |"
1040+
fi
1041+
# Only render a Total row when both sessions reported.
1042+
if [ -n "$GEN_TURNS" ] && [ -n "$REVIEW_TURNS" ]; then
1043+
TOTAL_TURNS=$((GEN_TURNS + REVIEW_TURNS))
1044+
TOTAL_COST=$(awk -v a="$GEN_COST" -v b="$REVIEW_COST" 'BEGIN { printf "%.4f", a + b }')
1045+
echo "| **Total** | **$TOTAL_TURNS** | **\$$TOTAL_COST** |"
1046+
fi
1047+
echo ""
1048+
fi
9691049
echo "$END"
9701050
} > /tmp/section.md
9711051
@@ -1006,17 +1086,31 @@ jobs:
10061086
REVIEW_CONCLUSION: ${{ steps.skill_review.conclusion }}
10071087
AUTOFIX_CONCLUSION: ${{ steps.autofix.conclusion }}
10081088
SKILL_COMMIT_COUNT: ${{ steps.skill_commits.outputs.count }}
1089+
GEN_TURNS: ${{ steps.skill_gen_stats.outputs.turns }}
1090+
GEN_COST: ${{ steps.skill_gen_stats.outputs.cost_usd }}
1091+
REVIEW_TURNS: ${{ steps.skill_review_stats.outputs.turns }}
1092+
REVIEW_COST: ${{ steps.skill_review_stats.outputs.cost_usd }}
10091093
run: |
1094+
# Compute totals when both sessions reported. Awk handles
1095+
# the float add since bash can't.
1096+
if [ -n "$GEN_TURNS" ] && [ -n "$REVIEW_TURNS" ]; then
1097+
TOTAL_TURNS=$((GEN_TURNS + REVIEW_TURNS))
1098+
TOTAL_COST=$(awk -v a="$GEN_COST" -v b="$REVIEW_COST" 'BEGIN { printf "%.4f", a + b }')
1099+
else
1100+
TOTAL_TURNS=""
1101+
TOTAL_COST=""
1102+
fi
10101103
gh pr comment "$PR_NUMBER" --body "## Upstream-release-docs run summary
10111104
10121105
Project: \`$PROJECT_ID\` at tag \`$NEW_TAG\`
10131106
1014-
| Step | Conclusion |
1015-
| --- | --- |
1016-
| Generation (\`skill_gen\`) | \`${GEN_CONCLUSION:-(not run)}\` |
1017-
| Editorial review (\`skill_review\`) | \`${REVIEW_CONCLUSION:-(not run)}\` |
1018-
| Autofix (prettier/eslint) | \`${AUTOFIX_CONCLUSION:-(not run)}\` |
1019-
| Skill commits produced | \`${SKILL_COMMIT_COUNT:-?}\` |
1107+
| Step | Conclusion | Turns | Cost (USD) |
1108+
| --- | --- | ---: | ---: |
1109+
| Generation (\`skill_gen\`) | \`${GEN_CONCLUSION:-(not run)}\` | ${GEN_TURNS:-–} | ${GEN_COST:+\$$GEN_COST} |
1110+
| Editorial review (\`skill_review\`) | \`${REVIEW_CONCLUSION:-(not run)}\` | ${REVIEW_TURNS:-–} | ${REVIEW_COST:+\$$REVIEW_COST} |
1111+
| **Total** | | **${TOTAL_TURNS:-–}** | ${TOTAL_COST:+**\$$TOTAL_COST**} |
1112+
| Autofix (prettier/eslint) | \`${AUTOFIX_CONCLUSION:-(not run)}\` | – | – |
1113+
| Skill commits produced | \`${SKILL_COMMIT_COUNT:-?}\` | – | – |
10201114
10211115
Full report and Claude's step-by-step log: $RUN_URL" || true
10221116

0 commit comments

Comments
 (0)