feat(reflexion): implement issue-driven cloud runner (#78)

Implements the WS-7 issue-driven cloud runner for the Reflexion loop.
The runner allows developers to start and interact with the AI
engineering loop entirely through GitHub issues and comments.

- Adds `.github/workflows/reflexion-issue-runner.yml` to trigger on
  labeled issues, specific comments, and manual workflow dispatch.
- Introduces `scripts/reflexion-issue-runner.js` to process START
  and RESUME paths without ever pushing code directly.
- Handles artifact zip extraction using `adm-zip` and inputs using
  Zod schema validation (`js-yaml`).
- Enforces a 10-turn cap per issue loop.
- Uses `<!-- processed-comment-id:XXX -->` markers to ensure strict
  idempotency of webhook re-deliveries.
- Adds `docs/reflexion-issue-runner.md` for workflow setup.

Author: bronz3beard

.github/workflows/reflexion-issue-runner.yml

@@ -0,0 +1,76 @@
+name: Reflexion Issue Runner
+
+on:
+  issues:
+    types: [labeled]
+  issue_comment:
+    types: [created]
+  workflow_dispatch:
+    inputs:
+      issue_number:
+        description: 'Issue number to run against'
+        required: true
+        type: number
+
+concurrency:
+  group: reflexion-${{ github.event.issue.number || github.event.inputs.issue_number }}
+  cancel-in-progress: false
+
+permissions:
+  issues: write
+  actions: read
+  contents: read
+
+jobs:
+  run-reflexion:
+    runs-on: ubuntu-latest
+    if: >-
+      (github.event_name == 'issues' && github.event.action == 'labeled' && github.event.label.name == 'reflexion:run') ||
+      (github.event_name == 'issue_comment' && github.event.action == 'created' && startsWith(github.event.comment.body, '/reflexion')) ||
+      (github.event_name == 'workflow_dispatch')
+    steps:
+      - name: Check Secrets
+        id: check_secrets
+        run: |
+          if [ -z "${{ secrets.GEMINI_API_KEY }}" ] || [ -z "${{ secrets.ANTHROPIC_API_KEY }}" ]; then
+            echo "Missing required API keys in secrets."
+            ISSUE_NUM="${{ github.event.issue.number || github.event.inputs.issue_number }}"
+            if [ -n "$ISSUE_NUM" ]; then
+              curl -s -X POST \
+                -H "Authorization: Bearer ${{ secrets.GITHUB_TOKEN }}" \
+                -H "Accept: application/vnd.github.v3+json" \
+                https://api.github.com/repos/${{ github.repository }}/issues/$ISSUE_NUM/comments \
+                -d '{"body":"### 🚨 Reflexion Loop Error\nMissing required `GEMINI_API_KEY` or `ANTHROPIC_API_KEY` in repository secrets."}'
+            fi
+            false
+          fi
+
+      - name: Checkout Code
+        uses: actions/checkout@v4
+
+      - name: Setup Node
+        uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+
+      - name: Install Dependencies
+        run: npm ci
+
+      - name: Run Reflexion Loop
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          REFLEXION_MAX_COST_USD: ${{ vars.REFLEXION_MAX_COST_USD || '3' }}
+          REFLEXION_MAX_REVISIONS: ${{ vars.REFLEXION_MAX_REVISIONS || '3' }}
+        run: |
+          node scripts/reflexion-issue-runner.js
+
+      - name: Upload State Artifact
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: reflexion-state-${{ github.event.issue.number || github.event.inputs.issue_number }}
+          path: .reflexion-out/*
+          retention-days: 7
+          if-no-files-found: ignore

docs/reflexion-issue-runner.md

@@ -0,0 +1,39 @@
+# Reflexion Issue Runner
+
+The Reflexion Issue Runner enables cloud-based scheduling of the AI engineering loop. It allows engineers to kick off and guide the iterative process of planning, critiquing, and refining purely through GitHub Issues—even from a mobile device.
+
+## The Workflow ("Label at Night, Answer Over Coffee")
+
+1. **Start the Loop:** Create an issue describing your goal. Apply the `reflexion:run` label.
+2. **Review Feedback:** The automated loop runs (generator & critic) and uploads an artifact with its findings, then leaves a comment on the issue. This comment includes the adjudicator verdict and interview questions.
+3. **Resume the Loop:** Over coffee, reply to the issue starting with `/reflexion answers`, and include your filled-in YAML block based on the provided answers template.
+4. **Final Approval:** Repeat until satisfied, then approve it to generate a final `ide-prompt.md`.
+
+## Setup & Configuration
+
+### Required Secrets
+
+Ensure the repository has the following Action secrets configured:
+
+- `GEMINI_API_KEY`: Used for generation (Google AI SDK).
+- `ANTHROPIC_API_KEY`: Used for the Critic & Adjudicator (Anthropic AI SDK).
+
+### Environment Variables
+
+Configure these repository environment variables (defaults are applied securely if unset):
+
+- `REFLEXION_MAX_COST_USD` (Default: `3`)
+- `REFLEXION_MAX_REVISIONS` (Default: `3`)
+
+## Security Model
+
+- **No Push:** The runner is structurally prohibited from pushing code. The pipeline communicates via read-only state files and outputs solely as artifacts/comments.
+- **Isolated Artifacts:** Loop state and execution outputs are safely zipped into a workflow artifact named `reflexion-state-<issue#>`. The state uses standard GitHub Actions artifact retention window restrictions.
+- **Controlled I/O Boundaries:** LLMs only receive the parsed issue brief and YAML answers block as an isolated payload.
+- **Untrusted Input Processing:** Comments are defensive-parsed using deterministic regex boundary extraction avoiding direct LLM routing.
+
+## Troubleshooting
+
+- **Missing Keys:** If a run immediately fails with a diagnostic comment stating missing API keys, check GitHub Secrets.
+- **Cap Reached:** The system will log a comment terminating if max cost threshold or `10` comment loop cycles per issue turn is reached.
+- **Stuck State:** Due to concurrency boundaries per issue, parallel comments on the same issue will wait.

package.json

@@ -90,6 +90,7 @@
     "easymde": "^2.20.0",
     "gray-matter": "^4.0.3",
     "html2canvas": "^1.4.1",
+    "js-yaml": "^5.2.1",
     "langfuse-core": "^3.38.20",
     "langfuse-node": "^3.38.20",
     "lucide-react": "^0.477.0",

pnpm-lock.yaml

@@ -0,0 +1,72 @@
+const { extractRunIdMarker, extractProcessedCommentIdMarker, formatRunnerComment, extractYamlBlock, validateAnswers } = require('../reflexion-issue-runner-utils.js');
+
+describe('reflexion-issue-runner-utils', () => {
+  describe('extractRunIdMarker', () => {
+    it('should extract a valid marker', () => {
+      const body = "Some text\n<!-- reflexion-run:987654321 -->\nMore text";
+      expect(extractRunIdMarker(body)).toBe("987654321");
+    });
+  });
+
+  describe('extractProcessedCommentIdMarker', () => {
+    it('should extract a valid marker', () => {
+      const body = "Some text\n<!-- processed-comment-id:444555 -->\nMore text";
+      expect(extractProcessedCommentIdMarker(body)).toBe("444555");
+    });
+  });
+
+  describe('formatRunnerComment', () => {
+    it('should format a diagnostic error message', () => {
+      const result = formatRunnerComment({
+        runId: "123",
+        diagnostic: "Something went wrong"
+      });
+      expect(result).toContain('<!-- reflexion-run:123 -->');
+      expect(result).toContain('### 🚨 Reflexion Loop Error');
+    });
+
+    it('should include processed-comment-id marker', () => {
+      const result = formatRunnerComment({
+        runId: "123",
+        triggeringCommentId: "888",
+        diagnostic: "Something went wrong"
+      });
+      expect(result).toContain('<!-- processed-comment-id:888 -->');
+    });
+  });
+
+  describe('extractYamlBlock', () => {
+    it('should extract a standard yaml block', () => {
+      const body = "Here is the yaml:\n```yaml\nfoo: bar\n```";
+      expect(extractYamlBlock(body).trim()).toBe("foo: bar");
+    });
+  });
+
+  describe('validateAnswers', () => {
+    it('validates correct answers structure', () => {
+      const valid = {
+        runId: "987",
+        decisions: [{ id: "q1", answer: "because" }]
+      };
+      const res = validateAnswers(valid);
+      expect(res.success).toBe(true);
+    });
+
+    it('validates directive only', () => {
+      const valid = {
+        runId: "987",
+        directive: "approve"
+      };
+      const res = validateAnswers(valid);
+      expect(res.success).toBe(true);
+    });
+
+    it('fails on missing runId', () => {
+      const invalid = {
+        decisions: [{ id: "q1", answer: "because" }]
+      };
+      const res = validateAnswers(invalid);
+      expect(res.success).toBe(false);
+    });
+  });
+});

scripts/__tests__/reflexion-issue-runner-utils.test.js

@@ -0,0 +1,104 @@
+const { z } = require('zod');
+
+// We use the AnswersSchema definition structure from src/lib/ai/reflexion/schema.ts
+const AnswersSchema = z.object({
+  runId: z.string(),
+  decisions: z.array(z.object({ id: z.string(), answer: z.string() })).default([]),
+  directive: z.enum(['approve', 'stop']).optional(),
+});
+
+function extractRunIdMarker(body) {
+  if (!body) return null;
+  const match = body.match(/<!--\s*reflexion-run:(\d+)\s*-->/);
+  return match ? match[1] : null;
+}
+
+function extractProcessedCommentIdMarker(body) {
+  if (!body) return null;
+  const match = body.match(/<!--\s*processed-comment-id:(\d+)\s*-->/);
+  return match ? match[1] : null;
+}
+
+function formatRunnerComment(opts) {
+  const parts = [];
+
+  parts.push(`<!-- reflexion-run:${opts.runId} -->`);
+  if (opts.triggeringCommentId) {
+    parts.push(`<!-- processed-comment-id:${opts.triggeringCommentId} -->`);
+  }
+
+  if (opts.diagnostic) {
+    parts.push('### 🚨 Reflexion Loop Error');
+    parts.push(opts.diagnostic);
+    return parts.join('\n\n');
+  }
+
+  if (opts.idePrompt) {
+    parts.push('### ✅ Reflexion Loop Approved');
+    parts.push('The plan has been approved. The finalized prompt is below:');
+    parts.push('<details><summary><b>ide-prompt.md</b></summary>');
+    parts.push('');
+    parts.push('```markdown');
+    parts.push(opts.idePrompt);
+    parts.push('```');
+    parts.push('');
+    parts.push('</details>');
+
+    if (opts.usageCost) {
+      parts.push(`\n*${opts.usageCost}*`);
+    }
+    return parts.join('\n\n');
+  }
+
+  parts.push('### 🤖 Reflexion Loop Status');
+
+  if (opts.scoreTable) {
+    parts.push(opts.scoreTable);
+  }
+
+  if (opts.adjudicatorVerdict) {
+    parts.push(`**Verdict:** ${opts.adjudicatorVerdict}`);
+  }
+
+  if (opts.usageCost) {
+    parts.push(`*${opts.usageCost}*`);
+  }
+
+  if (opts.interviewQuestions) {
+    parts.push('#### Questions');
+    parts.push(opts.interviewQuestions);
+  }
+
+  if (opts.answersTemplate) {
+    parts.push('#### Your Turn: Reply to this issue');
+    parts.push('Copy and edit the YAML block below, and reply with it starting your comment with `/reflexion answers`:');
+    parts.push('');
+    parts.push(opts.answersTemplate);
+  }
+
+  return parts.join('\n\n');
+}
+
+function extractYamlBlock(text) {
+  if (!text) return null;
+  const match = text.match(/```yaml(?: answers:)?\s*([\s\S]*?)```/);
+  if (match) return match[1];
+
+  const match2 = text.match(/```answers:\s*([\s\S]*?)```/);
+  if (match2) return match2[1];
+
+  return null;
+}
+
+function validateAnswers(yamlObj) {
+  // We expect an object parseable by AnswersSchema
+  return AnswersSchema.safeParse(yamlObj);
+}
+
+module.exports = {
+  extractRunIdMarker,
+  extractProcessedCommentIdMarker,
+  formatRunnerComment,
+  extractYamlBlock,
+  validateAnswers
+};