Skip to content

Commit 395bea8

Browse files
committed
Update AI instructions for sequencing
1 parent 0c2b070 commit 395bea8

1 file changed

Lines changed: 75 additions & 6 deletions

File tree

dist/TEMPLATES/_AI_INSTRUCTIONS.md

Lines changed: 75 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ When a user requests **"Run template [name] end to end"**, execute the complete
3838

3939
1. **Run scan** → Generate JSON log (Phase 1c)
4040
2. **AI triage** → Analyze findings and update JSON (Phase 2)
41-
3. **Generate HTML** → Create final report with AI summary (Phase 2)
41+
3. **Generate HTML** → Create final report with AI summary (Phase 2) ⚠️ **MUST happen AFTER triage**
4242
4. **Create GitHub issue** → Automated or manual (Phase 3)
4343

4444
**No manual intervention required** - the AI agent handles all phases automatically.
@@ -49,12 +49,32 @@ When a user requests **"Run template [name] end to end"**, execute the complete
4949
- "Run gravityforms complete workflow"
5050
- "Scan, triage, and create GitHub issue for hypercart-helper"
5151

52-
**AI Agent Actions:**
52+
### ⚠️ CRITICAL: Orchestration Sequence
53+
54+
**The HTML report MUST be regenerated AFTER AI triage is complete.** This is the most important rule to prevent missing AI triage data in the final report.
55+
56+
**Correct Sequence:**
57+
```
58+
1. Run scan → JSON created
59+
2. AI triage → JSON updated with ai_triage section
60+
3. Regenerate HTML → HTML includes AI triage data
61+
4. Create GitHub issue → Issue includes AI classifications
62+
```
63+
64+
**Incorrect Sequence (DO NOT DO THIS):**
65+
```
66+
❌ 1. Run scan → JSON created
67+
❌ 2. Generate HTML → HTML created WITHOUT ai_triage data
68+
❌ 3. AI triage → JSON updated, but HTML is stale
69+
❌ 4. Create GitHub issue → Issue missing AI classifications
70+
```
71+
72+
**AI Agent Actions (Correct Order):**
5373
1. Execute scan: `dist/bin/run [template-name]` (wait for completion)
5474
2. Locate generated JSON: `dist/logs/[TIMESTAMP].json`
5575
3. Perform AI triage analysis (read JSON, analyze findings)
5676
4. Update JSON with `ai_triage` section
57-
5. Regenerate HTML: `python3 dist/bin/json-to-html.py [json] [html]`
77+
5. **Regenerate HTML: `python3 dist/bin/json-to-html.py [json] [html]`** ← Must happen AFTER step 4
5878
6. Create GitHub issue: `dist/bin/create-github-issue.sh --scan-id [TIMESTAMP]`
5979
7. Report completion with final HTML report and GitHub issue URL (if created)
6080

@@ -64,6 +84,13 @@ When a user requests **"Run template [name] end to end"**, execute the complete
6484
- If GitHub issue creation fails → issue body saved to `dist/issues/` for manual use
6585
- Provide progress updates as each phase completes
6686

87+
**Verification Checklist:**
88+
- [ ] JSON log created and contains findings
89+
- [ ] AI triage analysis completed
90+
- [ ] JSON updated with `ai_triage` section (verify with `jq '.ai_triage' dist/logs/[TIMESTAMP].json`)
91+
- [ ] HTML regenerated AFTER triage (verify with `grep -c 'ai_triage\|AI Triage\|False Positives' dist/reports/[TIMESTAMP].html`)
92+
- [ ] HTML report opened in browser to confirm AI summary is visible
93+
6794
---
6895

6996
## Phase 1a: Check for Existing Templates
@@ -666,6 +693,47 @@ python3 dist/bin/json-to-html.py "$latest_json" dist/reports/manual-report.html
666693
| `gh auth required` | Not authenticated with GitHub | Run: `gh auth login` |
667694
| `Scan hangs or times out` | Large codebase or slow disk | Use `--exclude-pattern` to skip vendor/node_modules |
668695

696+
### AI Triage Data Missing from HTML Report
697+
698+
**Symptom:** HTML report generated but doesn't show AI triage summary, false positives, or confidence levels.
699+
700+
**Root Cause:** HTML was regenerated BEFORE AI triage was performed, or HTML was not regenerated AFTER triage updated the JSON.
701+
702+
**Solution:**
703+
704+
1. **Verify AI triage data exists in JSON:**
705+
```bash
706+
jq '.ai_triage' dist/logs/[TIMESTAMP].json
707+
```
708+
- If output is `null` → AI triage hasn't been performed yet
709+
- If output shows triage data → JSON is correct, HTML needs regeneration
710+
711+
2. **Regenerate HTML after triage:**
712+
```bash
713+
python3 dist/bin/json-to-html.py dist/logs/[TIMESTAMP].json dist/reports/[TIMESTAMP].html
714+
```
715+
716+
3. **Verify AI triage is now in HTML:**
717+
```bash
718+
grep -c 'AI Triage\|False Positives\|Needs Review' dist/reports/[TIMESTAMP].html
719+
```
720+
- Should return a count > 0
721+
- If 0, check that JSON has `ai_triage` section
722+
723+
4. **Open HTML in browser:**
724+
```bash
725+
open dist/reports/[TIMESTAMP].html # macOS
726+
xdg-open dist/reports/[TIMESTAMP].html # Linux
727+
```
728+
729+
**Prevention:** Always follow the correct sequence:
730+
1. Run scan → JSON created
731+
2. Perform AI triage → JSON updated with `ai_triage` section
732+
3. Regenerate HTML → HTML includes AI summary
733+
4. Create GitHub issue → Issue includes AI classifications
734+
735+
**Remember:** The HTML converter reads the JSON file at the time it runs. If you regenerate HTML before updating the JSON with AI triage data, the HTML will not include the triage information.
736+
669737
### Getting Help
670738

671739
If you encounter issues not covered here:
@@ -700,12 +768,13 @@ If you encounter issues not covered here:
700768
- [ ] Verify JSON log created: `dist/logs/[TIMESTAMP].json`
701769
- [ ] Verify HTML report created: `dist/reports/[TIMESTAMP].html`
702770

703-
**Phase 2: AI Triage**
771+
**Phase 2: AI Triage** ⚠️ **CRITICAL: HTML regeneration must happen AFTER triage**
704772
- [ ] Read JSON log: `cat dist/logs/[TIMESTAMP].json`
705773
- [ ] Analyze findings for false positives (check context, safeguards)
706774
- [ ] Update JSON with `ai_triage` section (summary stats + recommendations)
707-
- [ ] Regenerate HTML: `python3 dist/bin/json-to-html.py [json] [html]`
708-
- [ ] Verify AI summary appears at top of HTML report
775+
- [ ] **VERIFY JSON was updated:** `jq '.ai_triage' dist/logs/[TIMESTAMP].json`
776+
- [ ] **THEN regenerate HTML:** `python3 dist/bin/json-to-html.py [json] [html]`
777+
- [ ] Verify AI summary appears at top of HTML report: `grep -c 'AI Triage\|False Positives' dist/reports/[TIMESTAMP].html`
709778

710779
**Phase 3: GitHub Issue**
711780
- [ ] Determine scan ID from JSON filename

0 commit comments

Comments
 (0)