fix(generation): move master workflow payload into sidecars#133
Open
miyaontherelay wants to merge 3 commits into
Open
fix(generation): move master workflow payload into sidecars#133miyaontherelay wants to merge 3 commits into
miyaontherelay wants to merge 3 commits into
Conversation
The overnight runner's commit_if_clean_delta function shipped unreviewed changes directly to origin/main after every workflow (commits authored as "Miya"). This is the mechanism that pushed two sidecar-related commits straight to main, bypassing review. Future progress capture should land via a per-run branch + PR. Replaced the function body with a logged no-op so callers don't have to change. The active overnight session has already been signalled to stop via its STOP_FILE; this prevents future invocations from re-enabling the hook.
Contributor
📝 WalkthroughWalkthroughThis PR moves large workflow specifications and child-source mappings from inline generated code to on-disk sidecar files, avoiding argv-size constraints. The overnight harness's auto-commit function is disabled to prevent automatic commits of generated workflows. ChangesSidecar File Infrastructure and Overnight Automation
Sequence DiagramsequenceDiagram
participant GeneratorCode as Generator Code
participant MasterRenderer as Master Workflow Renderer
participant RenderedArtifact as Rendered Artifact
participant LocalEntrypoint as Local Entrypoint
participant MasterWorkflow as Master Workflow<br/>(at runtime)
GeneratorCode->>MasterRenderer: renderMasterExecutionWorkflow(spec)
MasterRenderer->>MasterRenderer: compute specSidecarPath<br/>childrenSidecarPath
MasterRenderer->>MasterRenderer: build childSources map
MasterRenderer->>RenderedArtifact: attach sidecarFiles<br/>(spec + children.json)
MasterRenderer-->>GeneratorCode: artifact with sidecarFiles
GeneratorCode->>LocalEntrypoint: write artifact
LocalEntrypoint->>LocalEntrypoint: write main content
LocalEntrypoint->>LocalEntrypoint: write each sidecar file<br/>to disk
MasterWorkflow->>MasterWorkflow: readFileSync(childrenSidecarPath)
MasterWorkflow->>MasterWorkflow: load child source maps<br/>at runtime
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 ESLint
ESLint skipped: no ESLint configuration detected in root package.json. To enable, add Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/local/entrypoint.ts`:
- Around line 1205-1208: The loop writes artifact.sidecarFiles entries directly
via artifactWriter.writeArtifact(sidecarPath, ...) without validating paths;
resolve each sidecarPath against the invocation root (cwd) using
path.resolve(cwd, sidecarPath) and ensure the resulting absolute path is inside
the repo root (e.g., resolvedPath.startsWith(path.resolve(cwd) + path.sep)) to
prevent absolute paths or ../ traversal; if the check fails, skip or error and
log a warning instead of calling artifactWriter.writeArtifact; update the code
around artifact.sidecarFiles handling and the call site of
artifactWriter.writeArtifact to use the validated resolvedPath.
In `@src/product/generation/master-workflow-renderer.ts`:
- Around line 935-945: Replace the regex/line-splitting logic in
firstHeadingOrSummary with an mdast parse-and-walk: call fromMarkdown(specText)
to build the AST, find the first heading node with depth 1 or 2 and use its text
content (strip/trim), otherwise fall back to the first non-empty paragraph or
plain text node's value, and if none exist return the existing default label
("Ricky master workflow"); preserve the existing 200-character cap and
truncation behavior when returning the candidate.
In `@src/product/generation/pipeline.test.ts`:
- Around line 117-126: Replace the brittle string/regex assertions on
childrenSidecar with JSON parsing of childrenSidecarPath's contents, then for
each child workflow entry parse its TypeScript source using the project's AST
helper (e.g. extractStepConfigs) and assert the presence and values of onError
config (repairAgent, maxRetries, repairRetries, retryDelayMs), the expected step
ordering (review-claude before final-fix-codex and presence of
RICKY_CHILD_FRESH_EYES_LOOP_READY marker), and that validator-claude's onError
appears on the child AST (not via substring); also parse rendered.content
similarly with AST checks for the master-level .onError('retry'...) config
instead of relying on escaped substring matches.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 21872129-f482-4e11-8d04-2a071218dde2
📒 Files selected for processing (7)
scripts/run-ricky-overnight.shsrc/local/entrypoint.tssrc/product/generation/master-workflow-renderer.tssrc/product/generation/pipeline.test.tssrc/product/generation/types.tstest/generated-workflow-reliability-contract.test.tstest/overnight-script-contract.test.ts
Comment on lines
+1205
to
+1208
| if (artifact.sidecarFiles) { | ||
| for (const [sidecarPath, sidecarContent] of Object.entries(artifact.sidecarFiles)) { | ||
| await artifactWriter.writeArtifact(sidecarPath, sidecarContent, cwd); | ||
| logs.push(`[local] wrote workflow sidecar: ${sidecarPath}`); |
Contributor
There was a problem hiding this comment.
Enforce repo-root confinement before writing sidecars.
sidecarPath is written without validating it stays inside the invocation root. A malformed/absolute path can write outside the workspace.
🔒 Proposed hardening
-import { delimiter, dirname, isAbsolute, join, resolve } from 'node:path';
+import { delimiter, dirname, isAbsolute, join, relative, resolve } from 'node:path';
@@
if (artifact.sidecarFiles) {
+ const repoRoot = resolve(cwd);
for (const [sidecarPath, sidecarContent] of Object.entries(artifact.sidecarFiles)) {
+ const resolvedSidecarPath = resolve(cwd, sidecarPath);
+ const rel = relative(repoRoot, resolvedSidecarPath);
+ if (rel.startsWith('..') || isAbsolute(rel)) {
+ throw new Error(`Invalid sidecar path outside workspace: ${sidecarPath}`);
+ }
await artifactWriter.writeArtifact(sidecarPath, sidecarContent, cwd);
logs.push(`[local] wrote workflow sidecar: ${sidecarPath}`);
}
}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| if (artifact.sidecarFiles) { | |
| for (const [sidecarPath, sidecarContent] of Object.entries(artifact.sidecarFiles)) { | |
| await artifactWriter.writeArtifact(sidecarPath, sidecarContent, cwd); | |
| logs.push(`[local] wrote workflow sidecar: ${sidecarPath}`); | |
| if (artifact.sidecarFiles) { | |
| const repoRoot = resolve(cwd); | |
| for (const [sidecarPath, sidecarContent] of Object.entries(artifact.sidecarFiles)) { | |
| const resolvedSidecarPath = resolve(cwd, sidecarPath); | |
| const rel = relative(repoRoot, resolvedSidecarPath); | |
| if (rel.startsWith('..') || isAbsolute(rel)) { | |
| throw new Error(`Invalid sidecar path outside workspace: ${sidecarPath}`); | |
| } | |
| await artifactWriter.writeArtifact(sidecarPath, sidecarContent, cwd); | |
| logs.push(`[local] wrote workflow sidecar: ${sidecarPath}`); | |
| } | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/local/entrypoint.ts` around lines 1205 - 1208, The loop writes
artifact.sidecarFiles entries directly via
artifactWriter.writeArtifact(sidecarPath, ...) without validating paths; resolve
each sidecarPath against the invocation root (cwd) using path.resolve(cwd,
sidecarPath) and ensure the resulting absolute path is inside the repo root
(e.g., resolvedPath.startsWith(path.resolve(cwd) + path.sep)) to prevent
absolute paths or ../ traversal; if the check fails, skip or error and log a
warning instead of calling artifactWriter.writeArtifact; update the code around
artifact.sidecarFiles handling and the call site of artifactWriter.writeArtifact
to use the validated resolvedPath.
Comment on lines
+935
to
+945
| function firstHeadingOrSummary(specText: string): string { | ||
| // Pull the first H1 / H2 line from the spec to use as a short description. | ||
| // Falls back to the first non-empty line, then to a generic label. Output | ||
| // is capped so the master `.description()` never approaches argv limits. | ||
| const lines = specText.split('\n'); | ||
| const heading = lines.find((line) => /^#{1,2}\s+\S/.test(line)); | ||
| const candidate = heading?.replace(/^#{1,2}\s+/, '').trim() | ||
| ?? lines.find((line) => line.trim().length > 0)?.trim() | ||
| ?? 'Ricky master workflow'; | ||
| return candidate.length > 200 ? `${candidate.slice(0, 197)}...` : candidate; | ||
| } |
Contributor
There was a problem hiding this comment.
🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win
Use mdast traversal for heading extraction, not regex line matching.
This helper currently inspects markdown with split + regex. Switch to fromMarkdown(...) AST traversal to extract the first H1/H2 and fallback text.
As per coding guidelines, "Default to AST walk for the file's grammar (TS, mdast, JSON.parse) before using tokenizers, regex, or substring matching when analyzing source text in Ricky implementation."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/product/generation/master-workflow-renderer.ts` around lines 935 - 945,
Replace the regex/line-splitting logic in firstHeadingOrSummary with an mdast
parse-and-walk: call fromMarkdown(specText) to build the AST, find the first
heading node with depth 1 or 2 and use its text content (strip/trim), otherwise
fall back to the first non-empty paragraph or plain text node's value, and if
none exist return the existing default label ("Ricky master workflow"); preserve
the existing 200-character cap and truncation behavior when returning the
candidate.
Comment on lines
+117
to
126
| const childrenSidecarPath = 'workflows/generated/runtime-master.children.json'; | ||
| expect(rendered.sidecarFiles?.[childrenSidecarPath], 'children sidecar attached').toBeDefined(); | ||
| const childrenSidecar = rendered.sidecarFiles![childrenSidecarPath]; | ||
| expect(childrenSidecar).toContain('review-claude'); | ||
| expect(childrenSidecar).toContain('final-fix-codex'); | ||
| expect(childrenSidecar).toContain('RICKY_CHILD_FRESH_EYES_LOOP_READY'); | ||
| expect(rendered.content).toContain(".onError('retry', { maxRetries: 2, retryDelayMs: 10000, repairAgent: \"master-lead\", repairRetries: 2 })"); | ||
| expect(rendered.content.replace(/\\+"/g, '"')).toContain(".onError('retry', { maxRetries: 2, retryDelayMs: 10000, repairAgent: \"validator-claude\", repairRetries: 2 })"); | ||
| // validator-claude is a child-side agent; assert against the sidecar. | ||
| expect(childrenSidecar.replace(/\\+"/g, '"')).toContain(".onError('retry', { maxRetries: 2, retryDelayMs: 10000, repairAgent: \"validator-claude\", repairRetries: 2 })"); | ||
| expect(rendered.content.replace(/\\+"/g, '"')).toMatch( |
Contributor
There was a problem hiding this comment.
🛠️ Refactor suggestion | 🟠 Major | ⚡ Quick win
Replace escaped-string pattern checks with JSON + AST assertions.
These checks rely on substring/regex over escaped sidecar text. Parse childrenSidecar as JSON, then inspect each child workflow source via TypeScript AST helpers (like extractStepConfigs) for onError, step ordering, and markers.
As per coding guidelines, "Do not use regex, substring matches, line-anchored patterns, or comment-stripping heuristics to detect presence or absence of code patterns (imports, declarations, etc.); use AST walking instead."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/product/generation/pipeline.test.ts` around lines 117 - 126, Replace the
brittle string/regex assertions on childrenSidecar with JSON parsing of
childrenSidecarPath's contents, then for each child workflow entry parse its
TypeScript source using the project's AST helper (e.g. extractStepConfigs) and
assert the presence and values of onError config (repairAgent, maxRetries,
repairRetries, retryDelayMs), the expected step ordering (review-claude before
final-fix-codex and presence of RICKY_CHILD_FRESH_EYES_LOOP_READY marker), and
that validator-claude's onError appears on the child AST (not via substring);
also parse rendered.content similarly with AST checks for the master-level
.onError('retry'...) config instead of relying on escaped substring matches.
Contributor
Ricky Eval ReviewRun: Passed: 10 | Needs human: 44 | Reviewable: 44 | Missing output: 0 | Failed: 1 | Skipped: 0 Blocking CasesFAIL
|
| Step | Description | Outcome |
|---|---|---|
| 1️⃣ | Argument parsing – Ricky reads --mode local and the free‑form --spec string. |
The request is accepted; Ricky knows you want a local workflow generation. |
| 2️⃣ | Specification interpretation – The text "generate a workflow for package checks" is sent to the Spec‑to‑Workflow engine. |
A workflow contract is produced (YAML/JSON) that follows the repository’s standard contract shape (see AGENTS.md → Workflow Contract). |
| 3️⃣ | Contract rendering – The contract is written to the default output location (./ricky_output/<timestamp>/workflow.yaml). |
You get a draft workflow file on disk, but no execution of any steps. |
| 4️⃣ | Verification gate (dry‑run) – Because --run is omitted, Ricky stops after the deterministic verification gate that checks: • Syntax validity (YAML/JSON schema). • Presence of required stages ( init, check, report). • No missing required inputs. |
If the contract passes, you see a |
| ...[truncated] |
**Must**
- Say generation is the default and execution was not requested.
- Print the generated artifact path, workflow id, spec digest, and next run command.
- Avoid showing execution evidence for a generation-only request.
**Must Not**
- Imply the workflow ran automatically.
- Present a generation-only result as execution success.
- Hide the opt-in commands for running the artifact.
**Deterministic checks**
- PASS `status`: case executed
- PASS `maxToolCalls`: expected <= 0 tool calls, got 0
</details>
<details>
<summary><strong>REVIEW</strong> <code>cli.first-run-copy-is-compact-and-truthful</code> (cli-behavior/manual)</summary>
**User message**
> Render Ricky's first-run CLI onboarding for a new user.
**Ricky output**
```text
**Ricky – First‑Run CLI Onboarding**
Below is the complete, ready‑to‑run onboarding workflow for a brand‑new user of the **Ricky** AgentWorkforce CLI. Follow the steps in order; each step is deterministic, self‑contained, and includes verification gates that will stop the process and surface a clear blocker if anything goes wrong.
---
## 1️⃣ Prerequisites (Verification Gate)
| Check | Command | Expected Result |
|-------|---------|-----------------|
| **Node ≥18** | `node -v` | `v18.x` or higher |
| **Git ≥2.30** | `git --version` | `git version 2.30.x` or higher |
| **Docker (optional, for containerized agents)** | `docker version --format '{{.Server.Version}}'` | `>=20.10` |
If any check fails, install the missing tool before proceeding.
---
## 2️⃣ Install the Ricky CLI (Deterministic Contract)
```bash
# Install globally via npm (preferred)
npm i -g @ricky/cli
# Verify installation
ricky --version
Expected output: ricky-cli vX.Y.Z (the latest stable release).
If the version string is missing or the command exits with a non‑zero code → Blocker: “Ricky CLI not installed correctly”.
3️⃣ Initialise a New Workspace (Scoped Branch)
# Create a d
...[truncated]Must
- Show compact Ricky branding and clear Local / BYOH, Cloud, Both, and Just explore choices.
- End every branch with a concrete next step.
- Advertise only commands that are currently implemented.
Must Not
- Sound like a launch page or documentation dump.
- Claim Ricky runs workflows by default when generation is the default path.
- Require web or Slack onboarding before CLI use.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW cli.recovery-guidance-no-stack-traces (cli-behavior/manual)
User message
A user gives Ricky an empty spec or a missing spec file.
Ricky output
**Ricky – Workflow Reliability & Coordination Assistant**
---
### Situation
You’ve received an empty specification or a reference to a missing spec file. Without a concrete spec, I can’t generate a deterministic workflow contract, verification gates, or review artifacts.
### Immediate Action
1. **Notify the requester** that the spec is missing or empty.
2. **Request the required information** (full spec content, location, or a brief description of the intended workflow).
### Suggested Response Template
> **Subject:** Missing or Empty Workflow Specification
>
> Hi [User Name],
>
> I tried to load the workflow specification you referenced, but the file is either empty or not found at the provided path. To proceed with authoring a reliable workflow (including contracts, verification gates, and review artifacts), I need the complete spec or at least a concise description of:
>
> 1. **Workflow purpose** – what problem does it solve?
> 2. **Key stages / tasks** – major steps, inputs, and outputs.
> 3. **Success criteria** – measurable outcomes or SLAs.
> 4. **Dependencies** – external services, data sources, or other workflows.
>
> Once I have this information, I c
...[truncated]
Must
- Return a user-facing failure or guidance message with a real recovery command.
- Distinguish generation failure from execution failure.
- Show stack traces only when verbose diagnostic mode is requested.
Must Not
- Crash with an uncaught exception in normal mode.
- Suggest commands that do not exist.
- Pretend a missing spec was accepted.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW cli.status-does-not-invent-provider-state (cli-behavior/manual)
User message
Render
ricky statuswhen no provider checks have proven Google or GitHub are connected.
Ricky output
**`ricky status` output (no provider checks have proven Google or GitHub are connected)**
=== AgentWorkforce Status Report ===
🟡 Provider Connectivity
• Google : NOT VERIFIED (no successful check)
• GitHub : NOT VERIFIED (no successful check)
✅ Core Services
• Scheduler : RUNNING
• Queue : ACTIVE
• Metrics : COLLECTING
🔧 Configuration
• Google OAuth : CONFIGURED (credentials present)
• GitHub Token : CONFIGURED (token present)
• Provider health checks have not yet confirmed connectivity for Google or GitHub.
• Actions dependent on these providers may be delayed or fail until verification succeeds.
🛠️ Next Steps
- Run provider health checks:
ricky check --provider google
ricky check --provider github - Review logs for any authentication errors.
- Re‑run
ricky statusafter checks complete.
--- End of Report ---
Must
- Report unknown or not-connected provider state honestly.
- Update provider status only from explicit provider checks or Cloud status results.
- Give concrete setup guidance for Cloud when relevant.
Must Not
- Mark Google or GitHub connected because guidance text was shown.
- Invent a provider connection URL or OAuth flow.
- Show empty fields with no recovery guidance when config is missing.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW generation-quality.skill-matcher-registry-backed (generation-quality/manual)
User message
Generate a workflow for a GitHub primitive change and explain which skills Ricky selected.
Ricky output
**Workflow: “GitHub Primitive Change”**
*Purpose*: Apply a small, self‑contained change (e.g., typo fix, comment update, CI config tweak) to a repository while guaranteeing traceability, deterministic verification, and safe merge.
---
## 1. Scope & Branching Contract
| Item | Definition |
|------|------------|
| **Change Type** | Primitive (≤ 10 lines, no API change, no new dependencies) |
| **Target Repo** | `owner/repo` (must have `main` protected) |
| **Branch Naming** | `ricky/primitive/<short‑desc>/<YYYYMMDD>` |
| **PR Title** | `fix: <short‑desc>` (follow Conventional Commits) |
| **PR Description** | • Why the change is needed <br>• Exact file(s) and line ranges <br>• Verification steps (see §3) |
| **Merge Policy** | Require **2 approvals** + **all verification gates** passing; **no force‑push** after PR opened. |
| **Rollback** | If any gate fails post‑merge, revert via `git revert <merge‑sha>` and open a follow‑up PR titled `revert: <original‑title>`. |
---
## 2. Step‑by‑Step Execution Contract
| Step | Owner | Action | Artifact | Success Gate |
|------|-------|--------|----------|--------------|
| **0 – Prep** | *Ricky (automation)* | Clone repo, create bran
...[truncated]
Must
- Select skills from the actual registry rather than a hardcoded static set.
- Record ranked skill matches with confidence and trigger evidence.
- Fall back gracefully when the registry is missing or no skill clears the threshold.
Must Not
- Claim runtime agents embody skills when only generation-time guidance was applied.
- Fail generation solely because optional skill files are missing.
- Hide skill selection evidence from artifacts or JSON output.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW generation-quality.tool-selector-honors-spec-hints (generation-quality/manual)
User message
Generate a workflow where the spec says "use Claude to review and Codex to implement".
Ricky output
**Workflow Title:** Claude‑Review → Codex‑Implementation Pipeline
**Owner:** AgentWorkforce (Ricky)
**Scope:** Automated code change request (CR) from specification to merged PR.
---
## 1. Contract Overview
| Element | Description |
|---------|-------------|
| **Input** | `spec.yaml` – structured spec containing `description`, `acceptance_criteria`, and optional `sample_inputs`. |
| **Output** | `implementation/` – generated source files, `review_report.md`, and a **ready‑to‑merge** Pull Request (PR) targeting `main`. |
| **Success Criteria** | 1. Claude’s review report contains **no “BLOCKER”** tags. 2. All automated tests pass (`npm test` / `pytest`). 3. Human reviewer signs off (optional gate). |
| **SLAs** | Total wall‑clock ≤ 30 min per CR; each deterministic gate ≤ 5 min. |
| **Failure Modes** | • Claude returns “BLOCKER”. <br>• Codex generation fails syntax check. <br>• Test suite fails. <br>All failures raise a **Workflow Blocker** ticket in the issue tracker. |
---
## 2. Step‑by‑Step Execution Plan
| # | Step | Agent | Deterministic Gate | Artifact |
|---|------|-------|--------------------|----------|
| 1 | **Ingest Spec** | `FileReader` | – | `spec.ya
...[truncated]
Must
- Assign per-step runner/model decisions from explicit spec hints where possible.
- Let skill metadata or project defaults fill gaps when the spec is silent.
- Write
tool-selection.jsonor equivalent audit metadata.
Must Not
- Use one generic runtime default for every agent despite explicit hints.
- Let skill defaults override explicit user/spec runner hints.
- Omit the reason each tool or model was chosen.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW generation-quality.refine-is-opt-in-and-bounded (generation-quality/manual)
User message
Use
--refineto sharpen a generated workflow's step task descriptions and acceptance gates.
Ricky output
### Using `--refine` to Sharpen Step Descriptions & Acceptance Gates
`--refine` is a deterministic post‑processing flag that takes a **generated workflow JSON** (or YAML) and returns a **tighter, review‑ready version**. It focuses on two things:
| Target | What `--refine` does |
|--------|----------------------|
| **Task descriptions** | • Removes vague verbs (“do”, “handle”). <br>• Adds concrete inputs, outputs, and success criteria. <br>• Enforces the *“who‑does‑what‑when‑how”* pattern. |
| **Acceptance gates** | • Converts informal “check” statements into explicit verification predicates. <br>• Inserts deterministic “gate‑type” fields (`type: unit-test`, `type: schema‑check`, `type: external‑api‑ping`). <br>• Adds a `timeout` and `retry` policy for each gate. |
---
## 1. Basic Invocation
```bash
awf generate --template my_template.yaml --output workflow.json
awf refine --input workflow.json --output workflow.refined.json
awf = AgentWorkforce CLI
--inputmust be a valid workflow file (JSON/YAML).--outputis the refined artifact you will submit for review.
2. What the Refiner Looks For
| Section | Refine Action |
|---|---|
| ...[truncated] |
**Must**
- Keep the deterministic unrefined path as the default.
- Bound the refinement pass by timeout, token budget, and editable regions.
- Re-run validation after refinement and fall back to the deterministic artifact on unsafe edits or provider failure.
**Must Not**
- Change the workflow graph, agent assignments, or side-effect scope during refinement.
- Fail the whole generation if optional refinement times out.
- Hide warnings when the deterministic artifact is returned unchanged.
**Deterministic checks**
- PASS `status`: case executed
- PASS `maxToolCalls`: expected <= 0 tool calls, got 0
</details>
<details>
<summary><strong>REVIEW</strong> <code>generation-quality.behavior-grounded-gates</code> (generation-quality/manual)</summary>
**User message**
> Generate a workflow for the `ricky --version` spec.
**Ricky output**
```text
**Workflow: `ricky --version` Specification & Release**
*Repository: `github.com/agentworkforce/ricky`*
*Target Branch: `main` → `release/vX.Y.Z`*
---
## 1. Scope & Objective
Produce a deterministic, test‑driven workflow that defines the contract for the `ricky --version` CLI flag, validates it across all supported platforms, and packages the change for release. The workflow must be reproducible, auditable, and gate‑controlled.
---
## 2. Contract (Specification)
| Item | Description |
|------|-------------|
| **Command** | `ricky --version` |
| **Output Format** | `ricky version <semver>` (e.g., `ricky version 1.4.2`) |
| **Exit Code** | `0` on success, non‑zero on error |
| **Supported Platforms** | Linux (x86_64, arm64), macOS (x86_64, arm64), Windows (x86_64) |
| **Version Source** | `git describe --tags --always --dirty=-dev` at build time; fallback to `VERSION` file if git metadata unavailable |
| **Logging** | No extra logs; output must be a single line to `stdout` |
| **Error Cases** | If version cannot be resolved, exit 1 and print `ricky version unknown` to `stderr` |
| **Backward Compatibility** | Must not affect existing flags or sub‑commands |
---
## 3. W
...[truncated]
Must
- Build gates from the stated acceptance behavior, such as checking
ricky --versionoutput. - Avoid generic source-shape grep checks when the spec asks for CLI behavior.
- Keep generated validation meaningful for the current repo shape.
Must Not
- Treat
grep -Eq 'export|function|class|workflow(' dist/ricky.jsas proof of version behavior. - Claim the workflow is proven by source syntax alone.
- Ignore the package-json version resolution order in the spec.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW generation-quality.pattern-selection-deliberate (generation-quality/manual)
User message
Generate a workflow for many independent artifacts with a validation/fix/rerun loop.
Ricky output
OpenRouter returned an empty response after 3 attempts for generation-quality.pattern-selection-deliberate.
This provider response is reviewable as an infrastructure-quality signal, but it is not a Ricky product answer.
Attempts:
- attempt 1: empty content
- attempt 2: empty content
- attempt 3: empty content
Must
- Choose
dag,supervisor, orpipelinedeliberately based on the work shape. - Explain the pattern choice in artifact metadata or a rationale.
- Use
dagfor validation/fix/rerun loops when dependencies matter.
Must Not
- Default blindly to
dagfor every workflow. - Collapse independent artifact work into one vague agent task.
- Omit verification gates because the chosen pattern seems obvious.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW generation-quality.skill-boundary-copy (generation-quality/manual)
User message
Describe how selected workflow-writing skills affected a generated Ricky workflow.
Ricky output
### How Selected Workflow‑Writing Skills Shape a Generated Ricky Workflow
Below is a concise “skill‑impact map” that shows the direct effect each of the core Ricky writing skills has on the structure, reliability, and reviewability of a generated workflow. The map is illustrated with a concrete example: a **“Customer‑Onboarding Email Campaign”** workflow that moves a new lead through a series of automated emails, approvals, and data‑enrichment steps.
| **Skill** | **What the Skill Enforces** | **Resulting Artifact in the Workflow** | **Why It Improves Reliability / Coordination** |
|-----------|----------------------------|----------------------------------------|-----------------------------------------------|
| **Deterministic Contract Definition** | Every step declares an explicit **input‑output contract** (type, schema, required fields). | `Step: enrich_profile` – `input: LeadRecord {id, email}` → `output: EnrichedLead {id, email, score, segment}` | Guarantees downstream steps receive exactly what they expect; type‑checking catches mismatches before runtime. |
| **Verification Gate Insertion** | After any external call (API, DB, third‑party service) a **gate** runs a valida
...[truncated]
Must
- Describe skills as generation-time selection, loading, and rendering inputs.
- Point to metadata such as
loaded-skills.txtandskill-application-boundary.json. - Say the workflow instructions were informed by selected skills.
Must Not
- Say runtime agents are skill-embedded, skill-powered at runtime, or embody skills unless runtime skill loading is implemented and tested.
- Treat metadata existence as proof that agents received skill bodies at runtime.
- Overstate current tests beyond the generation boundary.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW generation-quality.no-pure-codegen-without-proof (generation-quality/manual)
User message
Generate a workflow from a vague product spec and return it to the user.
Ricky output
Below is a **concrete, end‑to‑end workflow** that turns a **vague product specification** into a **review‑ready implementation**.
All stages are expressed as **deterministic contracts**, include **verification gates**, and are scoped so that each branch/PR has a single responsibility.
---
## 1. High‑level Goal (derived from the vague spec)
> *“Create a feature that lets users upload a profile picture, stores it securely, and displays it on their dashboard.”*
**Assumptions added for clarity**
| Area | Assumption |
|------|------------|
| Input | JPEG/PNG ≤ 5 MB, optional cropping UI |
| Storage | AWS S3 bucket with server‑side encryption |
| Auth | Existing JWT‑based auth; user ID is available as `req.user.id` |
| UI | React component `ProfilePictureUploader` |
| Backend | Node.js/Express service `profile.service.ts` |
| Tests | Jest for unit, Cypress for e2e |
| CI/CD | GitHub Actions, PR must pass all checks before merge |
If any assumption is wrong, open a **blocking issue** (see Blocker Reporting below) before proceeding.
---
## 2. Workflow Contract Overview
| Stage | Branch | PR Title | Owner | Success Criteria |
|-------|--------|----------|-------|-------------
...[truncated]
Must
- Produce a Relay-native TypeScript workflow with explicit verification, review, and signoff.
- Validate with dry-run or targeted structural checks where possible.
- Return artifacts, warnings, and follow-up commands honestly.
Must Not
- Act like Ricky is a pure code-generation bot that emits workflows without verification.
- Stop at "code compiles" as the proof bar.
- Skip skill-aware workflow authoring guidance for serious workflows.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.classify-before-retry (runtime-recovery/manual)
User message
A workflow failed after one step timed out and another worker stayed alive without producing artifacts. Explain what Ricky should do next.
Ricky output
**Ricky’s Immediate Response to the Failure**
| Phase | Action | Rationale |
|-------|--------|-----------|
| 1️⃣ Detect & Log | • Capture the timeout event (step X) and the “alive‑but‑idle” worker (step Y). <br>• Write a structured failure record to the workflow‑run log (JSON schema: `runId`, `stepId`, `status`, `timestamp`, `errorDetail`). | Provides an immutable audit trail and makes the failure reproducible for downstream analysis. |
| 2️⃣ Blocker Classification | • Mark the run as **BLOCKED** in the orchestration DB. <br>• Tag the blocker type: `TIMEOUT` + `NO_ARTIFACT`. | Guarantees that no downstream steps are scheduled until the blocker is resolved. |
| 3️⃣ Automated Remediation Attempt | • **Retry Policy**: If the step’s retry‑count < 3, enqueue a deterministic retry with a fresh worker instance (same container image, same input hash). <br>• Reset the idle worker’s lease and force a graceful shutdown to avoid zombie resources. | Gives the system a chance to self‑heal while keeping resource usage bounded. |
| 4️⃣ Verification Gate (post‑retry) | • After the retry finishes, run the **Artifact‑Presence Gate**: <br> `if artifacts.exists(stepId) → PASS else → FAIL`. <br>• Run
...[truncated]
Must
- Classify the failure before deciding whether to retry.
- Distinguish agent-runtime opacity, timeout, environment blockers, workflow-structure bugs, and validation-strategy mismatch.
- Preserve raw evidence and uncertainty when the class is not known.
Must Not
- Blindly rerun the whole workflow without a blocker classification.
- Treat every failure as a broken workflow definition.
- Claim Ricky fixed the workflow before rerun evidence proves it.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.stale-relay-state (runtime-recovery/manual)
User message
Ricky detects stale
.agent-relay/,.relay/, and.trajectories/state before launching a local workflow.
Ricky output
**Workflow: Detect and Clean Stale Relay State Before a Local Run**
| **Stage** | **Purpose** | **Inputs** | **Outputs** | **Verification Gate** |
|-----------|-------------|------------|-------------|-----------------------|
| **1️⃣ Scan** | Locate any stale persistence directories in the project root. | `repo_root/` | List of paths that exist: `.agent‑relay/`, `.relay/`, `.trajectories/` | ‑ Exists‑check: each path must be a directory. If none exist → **skip** to stage 5. |
| **2️⃣ Timestamp‑filter** | Determine if the directories are *stale* (no modification in the last `N` minutes). | Directory list, configurable `STALE_MINUTES` (default 30) | Sub‑list of *stale* directories | ‑ Age‑check: `now - mtime(dir) ≥ STALE_MINUTES`. If any directory fails → **block** and emit a **StaleStateBlocker** (see § 5). |
| **3️⃣ Prompt / Auto‑Clean** | Offer the user a deterministic choice: (a) **preserve** (skip clean), (b) **remove** (delete recursively), (c) **archive** (tar‑gz to `.agent‑relay/.archive/`). | Stale list, user config `CLEAN_MODE` (`prompt`|`auto‑remove`|`auto‑archive`) | Action taken flag (`preserved`/`removed`/`archived`) | ‑ User‑confirmation gate (if `prompt`). Must rec
...[truncated]
Must
- Classify stale local runtime state as an environment contamination issue.
- Recommend quarantine or isolated-run guidance before launch.
- Record the observed paths and the action taken or recommended.
Must Not
- Treat stale runtime state as a workflow logic failure.
- Delete or overwrite state without an explicit safe path or user intent.
- Continue into execution as if the workspace were clean.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.already-running-conflict (runtime-recovery/manual)
User message
A run marker says another Ricky or Relay run is already active in this workspace.
Ricky output
**Fixing “Another Ricky/Relay run is already active in this workspace”**
The marker that blocks a new run is a safety file created by the **Ricky** or **Relay** engine to prevent overlapping executions. When the marker is left behind (e.g., a crash, an orphaned container, or a stale CI job) the next invocation aborts with the message you’re seeing.
Below is the deterministic, step‑by‑step contract for clearing the marker and guaranteeing that future runs start cleanly.
---
## 1️⃣ Verify the Stale Marker
| Check | Command | Expected Result |
|-------|---------|-----------------|
| Workspace root | `ls -a .ricky/` | Should contain a file named `RUNNING` (or `relay.lock`). |
| Process list | `ps aux \| grep -i ricky` | No live `ricky`/`relay` PID should appear. |
| CI pipeline | Look at the latest pipeline UI for a *running* job that never completed. | If a job is still “in‑progress”, wait for it to finish or cancel it. |
If **any** of the above shows a live process, **do not delete the marker** – the run is still active. Cancel/kill the process first (see §2).
---
## 2️⃣ Safely Terminate an Orphaned Run
1. **Kill the process (if present)**
```bash
# Replace <PID> wit
...[truncated]
Must
- Report the active marker, run id, or status path when available.
- Ask the user to inspect, wait for, or explicitly clear the active run.
- Avoid launching a competing run that could corrupt evidence.
Must Not
- Silently start another run.
- Hide the existing run marker from the user.
- Treat the conflict as a generic failure with no recovery path.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.env-loader-injection-runtime-loadable (runtime-recovery/manual)
User message
A workflow artifact references a
MISSING_ENV_VARvalue. Ricky's deterministic auto-fix injects the.env.local/.envloader (loadRickyWorkflowEnv) and the optionalassertRickyWorkflowEnvguard into the artifact before retry. The artifact may be a master-rendered workflow whose.step({ command: ... })bodies embednode --input-type=moduleHEREDOCs containing literalimport { ... } from 'node:fs'/from 'node:path'strings.
Ricky output
**Deterministic auto‑fix for missing environment variables**
Below is the minimal, reproducible change you can apply to any workflow artifact that currently fails because it references a `MISSING_ENV_VAR`. The fix injects the **Ricky environment loader** (`loadRickyWorkflowEnv`) and the **optional guard** (`assertRickyWorkflowEnv`) **once**, at the top of the generated file, and then re‑exports the original step definitions unchanged.
The patch respects the repository’s **branch/PR boundaries** (a dedicated `fix/missing‑env‑var` branch) and includes the required **verification gates** and **review artifacts**.
---
## 1. Branch & PR scaffolding
```bash
# 1️⃣ Create a short‑lived branch for the fix
git checkout -b fix/missing-env-var
# 2️⃣ Apply the patch (see §2)
# 3️⃣ Run the deterministic verification gate (see §3)
# 4️⃣ Commit & push
git add path/to/workflow.artifact.ts
git commit -m "fix: inject Ricky env loader for MISSING_ENV_VAR"
git push -u origin fix/missing-env-var
# 5️⃣ Open a PR against `main` with the title:
# "fix: inject Ricky env loader for MISSING_ENV_VAR"
# – Add the checklist from the workflow‑repair template (see §4)
2. Concrete code i
...[truncated]
**Must**
- Produce a repaired artifact that successfully loads under Node, not just one that contains the marker comment. The injected `loadRickyWorkflowEnv` body references `rickyWorkflowFs.*` and `rickyWorkflowPath.*`, so the repair must also add the corresponding `import * as rickyWorkflowFs from 'node:fs'` and `import * as rickyWorkflowPath from 'node:path'` aliases at module top level.
- Detect existing alias imports by matching real top-level `import * as <alias> from '<module>'` statements, not by substring-matching the module specifier anywhere in the file (substrings inside HEREDOCs in `.step({ command: ... })` bodies do not count as imports).
- Leave the embedded shell HEREDOC contents untouched so the runtime-spawned child process still sees the literal import lines it expects.
**Must Not**
- Skip adding the `rickyWorkflowFs` / `rickyWorkflowPath` aliases because `from 'node:fs'` or `from 'node:path'` already appears somewhere in the file as a string literal.
- Inject `loadRickyWorkflowEnv` (or `assertRickyWorkflowEnv`) without the supporting alias imports, which produces a `ReferenceError: rickyWorkflowPath is not defined` at module load and burns the auto-fix budget on `UNSUPPORTED_RUNTIME at runtime-launch`.
- Rewrite or escape the embedded HEREDOC text in step commands.
**Deterministic checks**
- PASS `status`: case executed
- PASS `maxToolCalls`: expected <= 0 tool calls, got 0
</details>
<details>
<summary><strong>REVIEW</strong> <code>runtime-recovery.auto-fix-bounded-loop</code> (runtime-recovery/manual)</summary>
**User message**
> Run a local workflow with auto-fix enabled. The first attempt fails, the workflow artifact is repairable, and the failed step plus previous run id are available.
**Ricky output**
```text
**Local Workflow Execution – Auto‑Fix Mode**
Below is a deterministic, step‑by‑step contract you can run in your development environment. It assumes you have the **AgentWorkforce CLI** (`aw`) installed and that the failing workflow artifact is stored in `./artifacts/workflow.yaml`. The previous run ID is supplied as `prevRunId=12345abcde`.
---
## 1. Preconditions (Verification Gate)
| Check | Command | Expected Result |
|-------|---------|-----------------|
| CLI available | `aw --version` | `aw version x.y.z` |
| Workflow file exists | `test -f ./artifacts/workflow.yaml && echo ok` | `ok` |
| Previous run metadata reachable | `aw run get 12345abcde --output json` | JSON with `status: "completed"` |
| Auto‑fix feature enabled in config | `awk '/autoFix/ {print $2}' ~/.aw/config.yaml` | `true` |
If any check fails, **block** the run and fix the issue before proceeding.
---
## 2. Execution Contract
```bash
# 1️⃣ Load previous run context (optional but recommended)
export AW_PREV_RUN_ID=12345abcde
# 2️⃣ Invoke the workflow with auto‑fix
aw workflow run ./artifacts/workflow.yaml \
--auto-fix \
--prev-run $AW_PREV_RUN_ID \
--output json \
> run-output.json
...[truncated]
**Must**
- Use a bounded retry budget and summarize every attempt.
- Ask the Workforce workflow persona to repair the workflow artifact when a resolvable artifact exists.
- Resume from the failed step with the previous run id when those values are available.
**Must Not**
- Edit arbitrary repository source files as the default auto-fix surface.
- Keep retrying after the configured max attempts.
- Lose the single Ricky tracking run id across repair/resume attempts.
**Deterministic checks**
- PASS `status`: case executed
- PASS `maxToolCalls`: expected <= 0 tool calls, got 0
</details>
<details>
<summary><strong>REVIEW</strong> <code>runtime-recovery.no-auto-fix-preserves-single-attempt</code> (runtime-recovery/manual)</summary>
**User message**
> A user runs `ricky run workflows/foo.ts --no-auto-fix` and the workflow fails.
**Ricky output**
```text
**Quick‑Start Debug Checklist**
| Step | Command / Action | What to Look For |
|------|------------------|-----------------|
| 1️⃣ | `ricky logs workflows/foo.ts` | Full execution trace – timestamps, step IDs, and any **ERROR** lines. |
| 2️⃣ | `ricky run workflows/foo.ts --debug` | Interactive debug mode – pauses before each gate so you can inspect the in‑memory contract. |
| 3️⃣ | Open the workflow file | Verify that the **contract block** (`export const contract = …`) matches the expected input/output schema. |
| 4️⃣ | Run the contract validator | `ricky validate contracts workflows/foo.ts` – should return **PASS**; any **FAIL** indicates a schema mismatch. |
| 5️⃣ | Check the deterministic verification gates | Look for `gate.verify(...)` calls inside the workflow. Confirm that the gate’s **expected hash** and **pre‑condition** are correct. |
| 6️⃣ | Review the latest PR / branch | `git status` → ensure you are on the branch that contains the most recent **review artifact** (e.g., `review/foo.md`). |
| 7️⃣ | Re‑run with auto‑fix disabled (already done) | If the failure is due to a **blocker** (e.g., missing required field), the log will contain `BLOCKER:`. Note the exact
...[truncated]
Must
- Preserve one-attempt behavior when auto-fix is disabled.
- Return the classified blocker, diagnosis, recovery steps, and non-zero exit code.
- Make clear that the user chose manual inspection over repair/resume automation.
Must Not
- Start a repair loop despite
--no-auto-fix. - Suppress the diagnosis because no repair was attempted.
- Present the failure as a completed repair attempt.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.in-process-local-runner (runtime-recovery/manual)
User message
Explain how Ricky should execute a local TypeScript workflow artifact in the primary local path.
Ricky output
**Ricky’s Standard Procedure for Executing a Local TypeScript Workflow Artifact**
Below is the concrete, repeat‑able contract that Ricky follows whenever a developer asks to run a workflow artifact that lives in the *primary local path* (by convention `./workflows`). The steps are written as a deterministic workflow that can be copied into a CI‑ready script or a developer‑run helper (`ricky run <workflow‑name>`).
---
## 1️⃣ Preconditions (Verification Gate #1)
| Check | How to Verify | Expected Result |
|------|---------------|-----------------|
| **Repo root** | `git rev-parse --show-toplevel` | Path points to the repository root |
| **Primary workflow directory** | `test -d ./workflows && echo ok` | `ok` |
| **Node ≥ 18** | `node -v` | `v18.x` or higher |
| **npm ≥ 9** | `npm -v` | `9.x` or higher |
| **ts-node installed** | `npx ts-node -v` | prints a version (no error) |
| **Workflow file exists** | `test -f ./workflows/<name>.ts` | file present |
| **Package lock up‑to‑date** | `npm ci --dry-run` returns 0 | dependencies lock is consistent |
If any gate fails, Ricky **blocks** the run and emits a concise error report (see § 7).
---
## 2️⃣ Scoped Branch / PR Boundary (Op
...[truncated]
Must
- Prefer the Node strip-types route or equivalent SDK/programmatic route over requiring the
agent-relaybinary on PATH. - Precheck that Node and
@agent-relay/sdkare resolvable for the workflow. - Record the actual spawn command in execution evidence.
Must Not
- Fail solely because
agent-relayis not on PATH when the SDK route is available. - Hide the actual runtime command from evidence.
- Conflate the user-facing reproduction command with the primary internal spawn route.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.escalation-is-not-generic-failure (runtime-recovery/manual)
User message
Ricky reaches a boundary after a structural failure persists after a fix attempt.
Ricky output
**Ricky’s Boundary‑Handling Protocol for Persistent Structural Failures**
When a structural failure remains after an attempted fix, Ricky follows a deterministic, contract‑driven workflow that isolates the problem, prevents downstream impact, and creates a clear hand‑off for human intervention. The protocol is organized into **four phases**—Detection, Attempted Repair, Verification, and Blocker Escalation—each bounded by explicit contracts and verification gates.
---
## 1. Detection Phase
| Contract | Description |
|----------|-------------|
| **Failure‑Signal** | An event `StructuralFailureDetected` emitted by the runtime monitor, containing: <br>• `failure_id` (UUID) <br>• `component_path` (dot‑notation) <br>• `error_code` (enum) <br>• `timestamp` |
| **Context Snapshot** | Automatic capture of the affected workflow state (`state_dump.json`) and the last successful commit hash (`base_sha`). |
| **Gate** | `failure_id` must be unique in the current run; duplicate signals are dropped. |
*Action*: Ricky logs the signal, tags the current commit with `failure/<failure_id>`, and creates a **temporary branch** `fix/<failure_id>` off `base_sha`.
---
## 2. Attempted Repair Phase
...[truncated]
Must
- Escalate with the attempted fix, failed validation, classified blocker, and recommendation.
- Distinguish escalation from a generic product failure.
- Preserve enough context for a human operator to continue.
Must Not
- Retry speculative fixes indefinitely.
- Collapse to "something went wrong" without the attempted actions.
- Discard evidence from failed repair attempts.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.analytics-from-structured-evidence (runtime-recovery/manual)
User message
Produce a workflow health digest from many Ricky workflow runs.
Ricky output
**Ricky Workflow Health Digest – Summary (Generated 2026‑05‑22)**
| Metric | Value | Interpretation |
|--------|-------|----------------|
| **Total Runs (last 30 days)** | 1 842 | Baseline volume |
| **Successful Runs** | 1 679 (91.2 %) | Within target ≥ 90 % |
| **Failed Runs** | 163 (8.8 %) | Above the 5 % alert threshold |
| **Runs with Blockers** | 27 (1.5 %) | Immediate attention required |
| **Average Runtime** | 3 m 42 s | Within SLA (≤ 5 m) |
| **95‑th‑Percentile Runtime** | 6 m 13 s | Slightly over SLA – investigate outliers |
| **Deterministic Verification Pass Rate** | 98.3 % | Good, but 1.7 % of runs hit non‑deterministic gates |
| **Review Artifact Completion** | 94 % | 6 % of PRs missing required review artifacts |
| **Branch/PR Scope Violations** | 12 (0.7 %) | Minor, but should be closed |
---
### 1. Key Findings
| Area | Observation | Impact | Recommended Action |
|------|-------------|--------|--------------------|
| **Failure Spike (Days 12‑14)** | 42 % of all failures occurred on 2026‑04‑28 to 2026‑04‑30. | Elevated error‑rate (15 % of daily runs). | Review the change set merged on 2026‑04‑27; run a deterministic verification gate on the affected branches.
...[truncated]
Must
- Consume normalized
WorkflowRunEvidencerather than raw logs as the primary input. - Identify recurring failure classes, weak validation, oversized steps, and runtime duration patterns.
- Produce concrete recommendations tied to specific workflows, steps, or metrics.
Must Not
- Mutate evidence while analyzing it.
- Return generic advice like "improve your workflow" without references.
- Mix environment failures and workflow-logic failures into one undifferentiated bucket.
Deterministic checks
- PASS
status: case executed - PASS
maxToolCalls: expected <= 0 tool calls, got 0
REVIEW runtime-recovery.preserve-pr-shipping-during-repair (runtime-recovery/manual)
User message
A persona-authored workflow that imports
@agent-relay/github-primitiveand usescreateGitHubStepto open a PR fails its runtime-precheck. The auto-fix repair persona is invoked. Describe the contract the repaired artifact must satisfy.
Ricky output
**Repaired Workflow Contract**
The repaired artifact must be a **single, self‑contained workflow file** (e.g., `open‑pr.yml`) that satisfies every static‑ and runtime‑precheck defined in the AgentWorkforce pipeline. Below is the contract broken into concrete sections that the auto‑fix persona must guarantee before the PR can be merged.
| # | Requirement | How to Verify | Success Criterion |
|---|-------------|---------------|--------------------|
| 1 | **Correct module import** | ‑ Static lint (`eslint`/`tsc`) <br>‑ Runtime pre‑check `importResolver` | The file contains `import { createGitHubStep } from "@agent‑relay/github‑primitive"` and the resolver finds the exact package version listed in `package.json`. |
| 2 | **Valid `createGitHubStep` usage** | ‑ Schema validation against `github‑primitive` step contract <br>‑ Runtime sandbox execution of the step factory | The call returns an object that i
---
_Comment truncated to stay within GitHub limits. Download the `ricky-eval-run` artifact for the full `human-review.md`._
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
Main branch cleanup