Skip to content

Commit a9363b4

Browse files
os-zhuangclaude
andauthored
docs(skill): add the first automation eval — ADR-0044 revise loop (#2300)
Adds skills/objectstack-automation/evals/approvals/test-revise-loop.md, the first concrete eval under the skills' (previously placeholder) evals/ format — Scenario / Expected Output / Common Mistakes / Validation Criteria. It pins the expected send-back-for-revision shape (revise branch + signal wait + type:'back' resubmit + maxRevisions), the three footguns the compile-time lint flags, and registerFlow- based pass criteria. This is the authorable (a) half of #2274's eval slice. The executable eval RUNNER (b) remains a separate platform project — the skills' evals/ harness is still "not yet implemented" per each dir's README. Refs #2274, #1770 (ADR-0044). Builds on the lint (#2279) and skill (#2294). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 61d441f commit a9363b4

1 file changed

Lines changed: 76 additions & 0 deletions

File tree

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
# Eval: approval send-back-for-revision loop (ADR-0044)
2+
3+
Validates that an AI assistant authoring an approval flow with a *send back for
4+
revision* step emits the full ADR-0044 shape — a `revise` branch, a signal
5+
`wait` node, and a resubmit edge typed `type: 'back'` — so the flow **registers**
6+
and the loop actually works at run time.
7+
8+
Skill rule referenced: `SKILL.md` → "Send-back for revision (ADR-0044)".
9+
10+
## Scenario
11+
12+
> Build an autolaunched flow `budget_approval` on the `project` object: when
13+
> `budget` increases past 100000, route to a manager for approval. The manager
14+
> can **approve**, **reject**, or **send the record back to the submitter for
15+
> revision**; after the submitter reworks and resubmits, it returns to the
16+
> manager for another round. Cap it at two send-backs.
17+
18+
## Expected Output
19+
20+
An approval node with **three** labelled out-edges, a signal `wait` node for the
21+
revision window, and a **declared back-edge** closing the loop:
22+
23+
```typescript
24+
{
25+
name: 'budget_approval',
26+
type: 'autolaunched',
27+
nodes: [
28+
{ id: 'start', type: 'start',
29+
config: { objectName: 'project', triggerType: 'record-after-update',
30+
condition: 'budget > 100000 && budget != previous.budget' } },
31+
{ id: 'manager_review', type: 'approval',
32+
config: { approvers: [{ type: 'role', value: 'manager' }], lockRecord: true,
33+
maxRevisions: 2 } }, // send-back budget
34+
{ id: 'wait_revision', type: 'wait', label: 'Awaiting Revision',
35+
config: { eventType: 'signal', signalName: 'budget_revision' } },
36+
{ id: 'approved', type: 'end', label: 'Approved' },
37+
{ id: 'rejected', type: 'end', label: 'Rejected' },
38+
],
39+
edges: [
40+
{ id: 'e1', source: 'start', target: 'manager_review' },
41+
{ id: 'e2', source: 'manager_review', target: 'approved', label: 'approve' },
42+
{ id: 'e3', source: 'manager_review', target: 'rejected', label: 'reject' },
43+
{ id: 'e4', source: 'manager_review', target: 'wait_revision', label: 'revise' }, // send-back
44+
{ id: 'e5', source: 'wait_revision', target: 'manager_review', label: 'resubmit',
45+
type: 'back' }, // declared back-edge
46+
],
47+
}
48+
```
49+
50+
Mirrors `examples/app-showcase` -> `showcase_budget_approval`.
51+
52+
## Common Mistakes
53+
54+
| Mistake | Why it is wrong | Caught by |
55+
|---|---|---|
56+
| Resubmit edge **without** `type: 'back'` | `registerFlow` validates the graph-minus-back-edges as a DAG, so it rejects the cycle as un-declared | `registerFlow`; lint `flow-approval-revise-unmarked-backedge` |
57+
| `revise` edge to a wait node that **never loops back** | A valid DAG (registerFlow accepts it), but the submitter has nowhere to resubmit — the branch dead-ends | lint `flow-approval-revise-dead-end` |
58+
| `maxRevisions: 0` together with a `revise` edge | Send-back is disabled, so every revise auto-rejects and the branch never runs | lint `flow-approval-revise-disabled` |
59+
| Re-suspending the approval node in a "revise mode" (no wait node, no edge) | Hides a state machine inside one node — invisible to the canvas/run log; not the ADR-0044 model | design review |
60+
| Reusing `reject` for send-back | `reject` terminates; send-back is a *movement* that returns the record for rework (status `returned`, not `rejected`) | semantics |
61+
62+
## Validation Criteria
63+
64+
Score the generated flow:
65+
66+
1. **Registers**`registerFlow` accepts it (no un-declared-cycle error). *(required)*
67+
2. **Revise branch** — the approval node has an out-edge labelled `revise`. *(required)*
68+
3. **Back-edge** — exactly one edge closes the loop into the approval node, typed `type: 'back'`. *(required)*
69+
4. **Wait window** — the `revise` edge targets a `wait` node (signal flavour). *(required)*
70+
5. **Guard**`maxRevisions >= 1` on the approval config (the default `3` is fine; `0` fails). *(required)*
71+
6. **No lint findings**`lint-flow-patterns` emits none of the three `flow-approval-revise-*` warnings. *(required)*
72+
7. **Approve / reject intact** — the approval still has `approve` and `reject` out-edges. *(preferred)*
73+
74+
Pass = criteria 1–6 all hold. The canonical failure this eval guards against is a
75+
run that builds the loop but omits the back-edge (criterion 3) — accepted by a
76+
naive author, rejected by `registerFlow`.

0 commit comments

Comments
 (0)