Skip to content

Revamp PR body/comments + record per-skill turns and cost#782

Merged
rdimitrov merged 2 commits intomainfrom
redesign-pr-comms
Apr 22, 2026
Merged

Revamp PR body/comments + record per-skill turns and cost#782
rdimitrov merged 2 commits intomainfrom
redesign-pr-comms

Conversation

@rdimitrov
Copy link
Copy Markdown
Member

@rdimitrov rdimitrov commented Apr 22, 2026 •
edited
Loading

Two related changes to how the `upstream-release-docs` workflow communicates on PRs, bundled into one review.

1. Per-skill turns and cost (previously #781)

Parses `claude-code-action`'s execution log (`/home/runner/work/_temp/claude-execution-output.json`) right after each skill invocation, BEFORE the next one overwrites the shared file. Exposes `turns`, `cost_usd`, and `permission_denials` as step outputs and surfaces them in the PR body's new Run cost table. Baseline per-release run is around $6.

2. Reorganise PR body and comments for scannability

Rewrites everything the workflow posts on a PR. Driven by feedback after PRs #779 and #780 — the output read as three authors writing on the same page (log formatter, release-note generator, systems-internals doc), with no visual hierarchy so process lore competed with actionable info.

Structural principles

  1. PR body = single source of truth. All state (what changed, who reviews, cost, alerts) lives here.
  2. Comments = events only. Started, completed, failed — no state duplication.
  3. `
    Details` for process lore. The two-Opus-session narrative collapses by default.
  4. Three-tier alert grammar: `[!CAUTION]` (reviewer MUST act), `[!WARNING]` (degraded), `[!NOTE]` (informational). At most one alert at the top of the section.

New PR body skeleton

## Docs update for `toolhive` v0.23.1

[one alert at top, if applicable]

### At a glance

| | |
| --- | --- |
| **Upstream** | `stacklok/toolhive` [`v0.22.0` → `v0.23.1`](compare-url) |
| **Hand-written changes** | 3 commit(s) |
| **Reference assets** | refreshed (separate commit) |
| **Gaps** | 0 |
| **Release contributors** | 5 auto-assigned (see sidebar) |
| **Action required** | Spot-check skill-authored prose for accuracy |

### Gaps needing human context
(only when present, entries auto-demoted to H4 so they nest under At-a-glance)

### Additional release contributors
(only when there are overflow @-mentions GitHub refused to auto-assign)

### Run cost

| Session | Turns | Cost (USD) |
| --- | ---: | ---: |
| Generation | 89 | $5.7662 |
| Editorial review | 4 | $0.2689 |
| **Total** | **93** | **$6.0351** |

<details><summary>How this PR was built</summary>
Two Opus sessions + reference-path rules + Helper-prompt convention
</details>

Comments: simplified

Pre-skill (workflow_dispatch only) — was ~50 words, now 1 line:

Generating docs for `toolhive` v0.23.1… (run)

Post-skill (workflow_dispatch only) — was a 5-row table, now 1 line:

Done · 93 turns · $6.04 · run · see PR body for details

On failure, flags which step died:

Failed at generation (`failure`) · run · see PR body for details

Augmentation-failure comment — was a retry-hint paragraph, now a `[!CAUTION]` with the exact `gh` command:

Docs augmentation failed. Run log.
Retry: `gh workflow run upstream-release-docs.yml -f pr_number=779`

Dropped entirely

  • "Content additions by upstream-release-docs" heading (→ "Docs update for …")
  • Two-sentence process paragraph about "Two Claude Opus sessions produced this update…" (→ `
    Details`)
  • "Review guidance" paragraph (→ "Reference assets" cell in At-a-glance)
  • Bootstrap body narrative about "the workflow's bootstrap mode" (→ one line)
  • Meta-parenthetical in pre-skill comment about why the comment exists
  • "Skill commits produced" row in dispatch summary (→ "Hand-written changes" cell covers it; metric still computed internally for silent-run detection)
  • "Release contributors" section when all contributors got auto-assigned (sidebar shows them) — only rendered for overflow @-mentions now

Alerts and edge cases

  • Autogen drift (skill edited files under auto-generated reference paths) is now a `[!CAUTION]` at the top of the section with a bulleted revert list — it's a merge-blocker, should look like one. Was previously an inline `[!WARNING]` paragraph mid-body.
  • Compare failure (prev_tag missing upstream) moved from a top-of-section `[!WARNING]` into the At-a-glance "Release contributors" cell so it doesn't compete for attention with content alerts.
  • Silent run (skill ran OK but made no commits) and NO_CHANGES.md (skill said explicitly nothing to document) both surface as one-line `[!NOTE]` alerts.

Behaviour unchanged

Skill invocations, autofix step, reviewer-assignment logic, concurrency, bootstrap/retry modes. Purely additive on the stats side + presentation-layer changes to comments/body.

Validation

Once merged, the next `workflow_dispatch` or Renovate PR should produce a scannable PR body where a reviewer sees in 5 seconds: what changed, what needs them, and who else is on it. Drilling deeper (cost, process) requires one click.

rdimitrov and others added 2 commits April 22, 2026 12:33
@rdimitrov @claude
Record per-skill turns/cost and surface in PR body + summary
787189f 
Adds two capture steps that parse claude-code-action's execution
log (`/home/runner/work/_temp/claude-execution-output.json`) right
after each skill invocation, BEFORE the next one overwrites the
shared file. Exposes `turns`, `cost_usd`, and `permission_denials`
as step outputs for downstream use.

Surfaces the data in two places:

1. PR body: new "Run cost" subsection inside the upstream-release-
   docs marker block. Per-session rows plus a Total row when both
   sessions reported. Applies to both pull_request and
   workflow_dispatch runs.

2. workflow_dispatch summary comment: adds Turns and Cost columns
   to the existing step table, plus a Total row summing both
   sessions.

Useful for tracking per-release spend ($6 baseline) and catching
regressions -- e.g. a release that suddenly takes 10x the turns is
visible at a glance rather than requiring a drill-down into the
Actions Step Summary.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@rdimitrov @claude
Reorganise PR body and comments for scannability
d89c363 
Ground-up redesign of everything the workflow posts on a PR.
Previous output read as three authors writing on the same page
(log formatter + release notes + systems docs), with no visual
hierarchy and a lot of process lore competing with actionable
info. Feedback after PRs #779 and #780.

Key structural shift: the PR body is the single source of truth
for state. Comments are point-in-time events only. Process lore
goes behind <details>.

Changes:

**PR body** (rewritten end-to-end)
- New H2: "Docs update for <project> <tag>" replaces the
  process-descriptive "Content additions by upstream-release-docs".
- New "At a glance" table with 6 rows: Upstream (linked compare),
  Hand-written changes, Reference assets, Gaps, Release
  contributors, Action required. Answers the 5 questions a
  reviewer actually asks in < 5 seconds.
- Single top-of-section alert (at most one), ordered by severity:
  CAUTION for autogen-path drift, NOTE for NO_CHANGES.md, NOTE
  for silent run. Never stacked.
- Autogen drift promoted from inline [!WARNING] paragraph to
  [!CAUTION] at the top with a bulleted revert list -- it's a
  merge-blocker, should look like one.
- "Review guidance" paragraph deleted. The one operational claim
  ("reference files are auto-synced") is in the At-a-glance
  "Reference assets" row.
- "Release contributors" section dropped when everyone got
  auto-assigned (GitHub's sidebar already shows them). Only
  rendered as "Additional release contributors" when overflow
  @-mentions need a target.
- "Run cost" table kept visible, not buried in <details>, per
  request.
- Process narrative ("Two Claude Opus sessions...", auto-synced
  paths, Helper-prompt convention) collapsed into a single
  <details> block at the bottom.
- Compare-failure warning moved from a top-of-section [!WARNING]
  block into the At-a-glance "Release contributors" cell.

**Bootstrap PR body**
- Collapsed from two paragraphs to one line ("Manually dispatched
  by @x. Workflow run: ..."). Mechanism narration deleted.

**Pre-skill placeholder comment** (workflow_dispatch only)
- From a two-paragraph explanation + parenthetical meta-note
  ("this comment exists because track_progress isn't supported
  on workflow_dispatch") to a single line: "Generating docs for
  <project> <tag>... (run)".

**Post-skill summary comment** (workflow_dispatch only)
- Table replaced with a one-line status: "Done · 93 turns · $6.04
  · run · see PR body for details". On failure, identifies which
  step died. Duplicative detail (conclusions, commit counts) was
  already in the PR body.
- Dropped "Skill commits produced" row -- the at-a-glance
  "Hand-written changes" cell covers it.

**Augmentation failure comment**
- Wrapped in [!CAUTION] alert, compressed to two lines with the
  exact retry gh command inline (not a hand-wavy "retry via the
  workflow" pointer).

**Autogen-touch step (`detect-autogen`)**
- Now emits `touched` output (comma-separated paths) directly
  instead of pre-formatting a warning block. The Augment step
  composes the alert itself. Cleaner separation of concerns.

Unchanged: reviewer assignment logic, skill prompts, autofix
step, concurrency config.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@vercel
Copy link
Copy Markdown

vercel Bot commented Apr 22, 2026 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
docs-website Ready Ready Preview, Comment Apr 22, 2026 10:01am

Request Review

@rdimitrov rdimitrov changed the base branch from skill-token-cost-metrics to main April 22, 2026 10:02
@rdimitrov rdimitrov mentioned this pull request Apr 22, 2026
Closed
@rdimitrov rdimitrov changed the title Reorganise PR body and comments for scannability Revamp PR body/comments + record per-skill turns and cost Apr 22, 2026
@rdimitrov rdimitrov merged commit b3b2ab5 into main Apr 22, 2026
4 checks passed
@rdimitrov rdimitrov deleted the redesign-pr-comms branch April 22, 2026 10:15
@rdimitrov rdimitrov mentioned this pull request Apr 22, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JAORMX JAORMX JAORMX approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@rdimitrov @JAORMX