Skip to content

Commit 9ddcf06

Browse files
MarkusNeusingerMarkusNeusinger
authored
fix(workflows): update quality threshold to 90 and add CLI retry logic (#1089)
Fixes #967 ## Summary - Update quality threshold documentation from 85 to 90 - Add CLI retry logic for intermittent failures - Change issue lifecycle: spec-ready issues stay open until all implementations complete ## Changes ### Documentation (Issue Finding #4) Updates quality threshold from 85 to 90 in documentation to match actual workflow configuration in `impl-review.yml`. **Files:** - CLAUDE.md - README.md - docs/workflow.md - docs/concepts/claude-skill-plot-generation.md - prompts/quality-evaluator.md ### CLI Retry Logic (Issue Finding #3) Adds retry mechanism for Claude CLI steps to handle intermittent "Executable not found in $PATH" errors. Each Claude step now has `continue-on-error: true` and a retry step that runs if the first attempt fails. **Files:** - .github/workflows/spec-create.yml - .github/workflows/spec-update.yml - .github/workflows/impl-generate.yml - .github/workflows/impl-repair.yml - .github/workflows/util-claude.yml ### Issue Lifecycle (Bonus) - `spec-ready` issues now stay open until all 9 library implementations are merged - Changed `Closes #...` to `Related to #...` in spec PR body - Added auto-close logic in `impl-merge.yml` when all `impl:{library}:done` labels are present **Files:** - .github/workflows/spec-create.yml - .github/workflows/impl-merge.yml ## Test Plan - [ ] Verify documentation shows correct 90 threshold - [ ] Trigger a workflow and verify retry works on CLI failure - [ ] Create a test spec and verify issue stays open after merge - [ ] Verify issue closes when all 9 libraries have `impl:{library}:done`
1 parent 7287c52 commit 9ddcf06

File tree

11 files changed

+368
-20
lines changed

11 files changed

+368
-20
lines changed

‎.github/workflows/impl-generate.yml‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,71 @@ jobs:
196196
197197
- name: Run Claude Code to generate implementation
198198
id: claude
199+
continue-on-error: true
200+
timeout-minutes: 60
201+
uses: anthropics/claude-code-action@v1
202+
with:
203+
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
204+
claude_args: "--model opus"
205+
prompt: |
206+
## Task: Generate ${{ steps.inputs.outputs.library }} Implementation
207+
208+
You are generating the **${{ steps.inputs.outputs.library }}** implementation for **${{ steps.inputs.outputs.specification_id }}**.
209+
210+
### Step 1: Read required files
211+
1. `prompts/plot-generator.md` - Base generation rules
212+
2. `prompts/default-style-guide.md` - Visual style requirements
213+
3. `prompts/quality-criteria.md` - Quality requirements
214+
4. `prompts/library/${{ steps.inputs.outputs.library }}.md` - Library-specific rules
215+
5. `plots/${{ steps.inputs.outputs.specification_id }}/specification.md` - The specification
216+
217+
### Step 2: Generate implementation
218+
Create: `plots/${{ steps.inputs.outputs.specification_id }}/implementations/${{ steps.inputs.outputs.library }}.py`
219+
220+
The script MUST:
221+
- Save as `plot.png` in the current directory
222+
- For interactive libraries (plotly, bokeh, altair, highcharts, pygal, letsplot): also save `plot.html`
223+
224+
### Step 3: Test and fix (up to 3 attempts)
225+
Run the implementation:
226+
```bash
227+
source .venv/bin/activate
228+
cd plots/${{ steps.inputs.outputs.specification_id }}/implementations
229+
MPLBACKEND=Agg python ${{ steps.inputs.outputs.library }}.py
230+
```
231+
232+
If it fails, fix and try again (max 3 attempts).
233+
234+
### Step 4: Visual self-check
235+
Look at the generated `plot.png`:
236+
- Does it match the specification?
237+
- Are axes labeled correctly?
238+
- Is the visualization clear?
239+
240+
### Step 5: Format the code
241+
```bash
242+
source .venv/bin/activate
243+
ruff format plots/${{ steps.inputs.outputs.specification_id }}/implementations/${{ steps.inputs.outputs.library }}.py
244+
ruff check --fix plots/${{ steps.inputs.outputs.specification_id }}/implementations/${{ steps.inputs.outputs.library }}.py
245+
```
246+
247+
### Step 6: Commit
248+
```bash
249+
git config user.name "github-actions[bot]"
250+
git config user.email "github-actions[bot]@users.noreply.github.com"
251+
git add plots/${{ steps.inputs.outputs.specification_id }}/implementations/${{ steps.inputs.outputs.library }}.py
252+
git commit -m "feat(${{ steps.inputs.outputs.library }}): implement ${{ steps.inputs.outputs.specification_id }}"
253+
git push -u origin implementation/${{ steps.inputs.outputs.specification_id }}/${{ steps.inputs.outputs.library }}
254+
```
255+
256+
### Report result
257+
Print exactly one line:
258+
- `GENERATION_SUCCESS` - if everything worked
259+
- `GENERATION_FAILED: <reason>` - if it failed
260+
261+
- name: Retry Claude (on failure)
262+
if: steps.claude.outcome == 'failure'
263+
id: claude_retry
199264
timeout-minutes: 60
200265
uses: anthropics/claude-code-action@v1
201266
with:

‎.github/workflows/impl-merge.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,54 @@ jobs:
232232
233233
gh issue comment "$ISSUE" --body "$BODY"
234234
235+
- name: Close issue if all libraries done
236+
if: steps.check.outputs.should_run == 'true' && steps.issue.outputs.number != ''
237+
env:
238+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
239+
ISSUE: ${{ steps.issue.outputs.number }}
240+
SPEC_ID: ${{ steps.extract.outputs.specification_id }}
241+
run: |
242+
# All 9 supported libraries
243+
LIBRARIES="matplotlib seaborn plotly bokeh altair plotnine pygal highcharts letsplot"
244+
245+
# Get current labels on the issue
246+
LABELS=$(gh issue view "$ISSUE" --json labels -q '.labels[].name' 2>/dev/null || echo "")
247+
248+
# Count done implementations
249+
DONE_COUNT=0
250+
for lib in $LIBRARIES; do
251+
if echo "$LABELS" | grep -q "^impl:${lib}:done$"; then
252+
DONE_COUNT=$((DONE_COUNT + 1))
253+
fi
254+
done
255+
256+
echo "::notice::Libraries done: $DONE_COUNT/9"
257+
258+
# Close issue if all 9 libraries are done
259+
if [ "$DONE_COUNT" -eq 9 ]; then
260+
gh issue comment "$ISSUE" --body "## :tada: All Implementations Complete!
261+
262+
All 9 library implementations for \`${SPEC_ID}\` have been successfully merged.
263+
264+
| Library | Status |
265+
|---------|--------|
266+
| matplotlib | :white_check_mark: |
267+
| seaborn | :white_check_mark: |
268+
| plotly | :white_check_mark: |
269+
| bokeh | :white_check_mark: |
270+
| altair | :white_check_mark: |
271+
| plotnine | :white_check_mark: |
272+
| pygal | :white_check_mark: |
273+
| highcharts | :white_check_mark: |
274+
| letsplot | :white_check_mark: |
275+
276+
---
277+
:robot: *[impl-merge](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})*"
278+
279+
gh issue close "$ISSUE"
280+
echo "::notice::Closed issue #$ISSUE - all implementations complete"
281+
fi
282+
235283
- name: Trigger database sync
236284
if: steps.check.outputs.should_run == 'true'
237285
env:

‎.github/workflows/impl-repair.yml‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,68 @@ jobs:
110110
111111
- name: Run Claude Code to repair implementation
112112
id: claude
113+
continue-on-error: true
114+
timeout-minutes: 45
115+
uses: anthropics/claude-code-action@v1
116+
with:
117+
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
118+
claude_args: "--model opus"
119+
prompt: |
120+
## Task: Repair ${{ inputs.library }} Implementation for ${{ inputs.specification_id }}
121+
122+
This is **repair attempt ${{ inputs.attempt }}/3**. The previous implementation was rejected.
123+
124+
### Step 1: Read the AI review feedback
125+
Read `/tmp/ai_feedback.md` to understand what needs to be fixed.
126+
127+
### Step 2: Read reference files
128+
1. `prompts/library/${{ inputs.library }}.md` - Library-specific rules
129+
2. `plots/${{ inputs.specification_id }}/specification.md` - The specification
130+
3. `prompts/quality-criteria.md` - Quality requirements
131+
132+
### Step 3: Read current implementation
133+
`plots/${{ inputs.specification_id }}/implementations/${{ inputs.library }}.py`
134+
135+
### Step 4: Fix the issues
136+
Based on the AI feedback, fix:
137+
- Visual quality issues
138+
- Code quality issues
139+
- Spec compliance issues
140+
141+
### Step 5: Test the fix
142+
```bash
143+
source .venv/bin/activate
144+
cd plots/${{ inputs.specification_id }}/implementations
145+
MPLBACKEND=Agg python ${{ inputs.library }}.py
146+
```
147+
148+
### Step 6: Visual self-check
149+
View `plot.png` and verify fixes are correct.
150+
151+
### Step 7: Format the code
152+
```bash
153+
source .venv/bin/activate
154+
ruff format plots/${{ inputs.specification_id }}/implementations/${{ inputs.library }}.py
155+
ruff check --fix plots/${{ inputs.specification_id }}/implementations/${{ inputs.library }}.py
156+
```
157+
158+
### Step 8: Commit and push
159+
```bash
160+
git config user.name "github-actions[bot]"
161+
git config user.email "github-actions[bot]@users.noreply.github.com"
162+
git add plots/${{ inputs.specification_id }}/implementations/${{ inputs.library }}.py
163+
git commit -m "fix(${{ inputs.library }}): address review feedback for ${{ inputs.specification_id }}
164+
165+
Attempt ${{ inputs.attempt }}/3 - fixes based on AI review"
166+
git push origin ${{ env.branch }}
167+
```
168+
169+
### Report result
170+
Print: `REPAIR_SUCCESS` or `REPAIR_FAILED: <reason>`
171+
172+
- name: Retry Claude (on failure)
173+
if: steps.claude.outcome == 'failure'
174+
id: claude_retry
113175
timeout-minutes: 45
114176
uses: anthropics/claude-code-action@v1
115177
with:

‎.github/workflows/spec-create.yml‎

Lines changed: 97 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,102 @@ jobs:
6767
- name: Process with Claude
6868
if: steps.check.outputs.should_run == 'true'
6969
id: process
70+
continue-on-error: true
71+
timeout-minutes: 30
72+
uses: anthropics/claude-code-action@v1
73+
with:
74+
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
75+
claude_args: "--model opus"
76+
prompt: |
77+
## Task: Create New Specification
78+
79+
You are creating a new plot specification.
80+
81+
### Issue Details
82+
- **Title:** ${{ github.event.issue.title }}
83+
- **Number:** #${{ github.event.issue.number }}
84+
- **Author:** ${{ github.event.issue.user.login }}
85+
- **Body:**
86+
```
87+
${{ github.event.issue.body }}
88+
```
89+
90+
---
91+
92+
## Instructions
93+
94+
1. **Read the rules:** `prompts/spec-id-generator.md`
95+
96+
2. **Check for duplicates:**
97+
- List all existing specs: `ls plots/`
98+
- Read existing specification files if titles seem similar
99+
- If duplicate found: Post comment explaining which spec matches, then STOP
100+
101+
3. **Generate specification-id:**
102+
- Format: `{type}-{variant}` or `{type}-{variant}-{modifier}`
103+
- Examples: `scatter-basic`, `bar-grouped-horizontal`, `heatmap-correlation`
104+
- All lowercase, hyphens only
105+
106+
4. **Create specification branch:**
107+
```bash
108+
git checkout -b "specification/{specification-id}"
109+
```
110+
111+
5. **Post analysis comment:**
112+
Post a SHORT comment (max 3-4 sentences) to the issue using `gh issue comment`:
113+
- Is this a valid/useful plot type?
114+
- Does it already exist? (check `ls plots/`)
115+
- Any concerns?
116+
117+
6. **Create specification files:**
118+
- Read template: `prompts/templates/specification.md`
119+
- Read metadata template: `prompts/templates/specification.yaml`
120+
- Create directory: `plots/{specification-id}/`
121+
- Create: `plots/{specification-id}/specification.md` (follow template structure)
122+
- Create: `plots/{specification-id}/specification.yaml` with:
123+
- `specification_id`: the generated id
124+
- `title`: a proper title
125+
- `created`: Use `$(date -u +"%Y-%m-%dT%H:%M:%SZ")` for current timestamp
126+
- `issue`: ${{ github.event.issue.number }}
127+
- `suggested`: ${{ github.event.issue.user.login }}
128+
- `tags`: appropriate tags for this plot type
129+
- Create empty folder: `plots/{specification-id}/implementations/`
130+
- Create empty folder: `plots/{specification-id}/metadata/`
131+
132+
7. **Commit and push:**
133+
```bash
134+
git config user.name "github-actions[bot]"
135+
git config user.email "github-actions[bot]@users.noreply.github.com"
136+
git add plots/{specification-id}/
137+
git commit -m "spec: add {specification-id} specification
138+
139+
Created from issue #${{ github.event.issue.number }}"
140+
git push -u origin "specification/{specification-id}"
141+
```
142+
143+
8. **Update issue title:**
144+
```bash
145+
gh issue edit ${{ github.event.issue.number }} --title "[{specification-id}] {original title}"
146+
```
147+
148+
9. **Output for workflow:**
149+
After completing, print these lines exactly:
150+
```
151+
SPECIFICATION_ID={specification-id}
152+
BRANCH=specification/{specification-id}
153+
```
154+
155+
---
156+
157+
## Important Rules
158+
- Do NOT create a PR (the workflow does that)
159+
- Do NOT add labels
160+
- Do NOT close the issue
161+
- STOP after pushing the branch
162+
163+
- name: Retry Claude (on failure)
164+
if: steps.check.outputs.should_run == 'true' && steps.process.outcome == 'failure'
165+
id: process_retry
70166
timeout-minutes: 30
71167
uses: anthropics/claude-code-action@v1
72168
with:
@@ -194,7 +290,7 @@ jobs:
194290
--body "$(cat <<EOF
195291
## New Specification: \`${SPEC_ID}\`
196292
197-
Closes #${{ github.event.issue.number }}
293+
Related to #${{ github.event.issue.number }}
198294
199295
---
200296

‎.github/workflows/spec-update.yml‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,74 @@ jobs:
7070
7171
- name: Process update with Claude
7272
id: process
73+
continue-on-error: true
74+
timeout-minutes: 30
75+
uses: anthropics/claude-code-action@v1
76+
with:
77+
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
78+
claude_args: "--model opus"
79+
prompt: |
80+
## Task: Update Existing Specification
81+
82+
You are updating an existing plot specification.
83+
84+
### Issue Details
85+
- **Title:** ${{ github.event.issue.title }}
86+
- **Number:** #${{ github.event.issue.number }}
87+
- **Specification ID:** ${{ steps.extract.outputs.specification_id }}
88+
- **Body:**
89+
```
90+
${{ github.event.issue.body }}
91+
```
92+
93+
---
94+
95+
## Instructions
96+
97+
1. **Read current specification:**
98+
- `plots/${{ steps.extract.outputs.specification_id }}/specification.md`
99+
- `plots/${{ steps.extract.outputs.specification_id }}/specification.yaml`
100+
101+
2. **Post analysis comment:**
102+
Post a SHORT comment (max 3-4 sentences) to the issue using `gh issue comment`:
103+
- Is this a valid/useful change?
104+
- What will be modified?
105+
- Any concerns?
106+
107+
3. **Create update branch:**
108+
```bash
109+
git checkout -b "specification/${{ steps.extract.outputs.specification_id }}-update"
110+
```
111+
112+
4. **Apply updates:**
113+
- Modify `plots/${{ steps.extract.outputs.specification_id }}/specification.md` as needed
114+
- Update `plots/${{ steps.extract.outputs.specification_id }}/specification.yaml`:
115+
- Add entry to `history` array with:
116+
- `date`: current timestamp (ISO 8601 format)
117+
- `issue`: ${{ github.event.issue.number }}
118+
- `changes`: brief description of what changed
119+
120+
5. **Commit and push:**
121+
```bash
122+
git config user.name "github-actions[bot]"
123+
git config user.email "github-actions[bot]@users.noreply.github.com"
124+
git add plots/${{ steps.extract.outputs.specification_id }}/
125+
git commit -m "spec: update ${{ steps.extract.outputs.specification_id }}
126+
127+
Updated from issue #${{ github.event.issue.number }}"
128+
git push -u origin "specification/${{ steps.extract.outputs.specification_id }}-update"
129+
```
130+
131+
---
132+
133+
## Important Rules
134+
- Do NOT create a PR (the workflow does that)
135+
- Do NOT add labels
136+
- STOP after pushing the branch
137+
138+
- name: Retry Claude (on failure)
139+
if: steps.process.outcome == 'failure'
140+
id: process_retry
73141
timeout-minutes: 30
74142
uses: anthropics/claude-code-action@v1
75143
with:

0 commit comments

Comments
 (0)