|
| 1 | +# ADR-0037: Token / scope-tree execution — durable pause inside parallel branches and loop iterations |
| 2 | + |
| 3 | +**Status**: Proposed (2026-06-11) |
| 4 | +**Deciders**: ObjectStack Protocol Architects |
| 5 | +**Builds on**: [ADR-0019](./0019-approval-as-flow-node.md) (durable-pause node via suspend/resume — *between*-flow chaining added in its 2026-06-10 addendum), [ADR-0031](./0031-advanced-flow-node-executors-and-dag.md) (structured `loop` / `parallel` / `try_catch` constructs, DAG invariant), [ADR-0018](./0018-unified-node-action-registry.md) (open node/executor registry) |
| 6 | +**Consumers**: `@objectstack/services/service-automation` (engine core — `executeNode` / `traverseNext` / `resume`, `SuspendedRun`, `sys_automation_run`), `@objectstack/spec` (`automation/execution.zod.ts`), `../objectui` (Runs panel, flow runner) |
| 7 | + |
| 8 | +--- |
| 9 | + |
| 10 | +## TL;DR |
| 11 | + |
| 12 | +The engine tracks a paused run with a **single program counter** — `SuspendedRun.nodeId`, |
| 13 | +one position. That is enough for a linear pause (`approval` / `screen` / `wait` |
| 14 | +on the main path) but cannot represent **two pauses at once**. So the engine |
| 15 | +**forbids pausing inside a `parallel` branch or a `loop` iteration** (ADR-0019 M1 |
| 16 | +scope note, `engine.ts`: "durable pause across parallel gateways is out of |
| 17 | +scope"). That blocks the most-requested real shapes: *parallel approvals* |
| 18 | +("finance AND legal sign off concurrently") and *batch approvals* ("route each |
| 19 | +line item over $10k"). |
| 20 | + |
| 21 | +This ADR adopts a **token / scope-tree** runtime model — the established |
| 22 | +BPMN-engine representation (Camunda / Flowable). A run's live state becomes a |
| 23 | +**set of tokens** (execution positions) organized in a **scope tree** (the root |
| 24 | +flow, each parallel branch, each loop iteration, each try region is a scope). |
| 25 | +Any token can pause independently; a scope's join is a barrier that completes |
| 26 | +when its child tokens arrive. The flow **authoring model is unchanged** — token |
| 27 | +tree is a pure runtime representation, invisible to the flow JSON, so ADR-0031's |
| 28 | +AI-authored structured constructs and the DAG invariant both stand. |
| 29 | + |
| 30 | +This is a **core-engine rewrite** of traversal + suspend/resume + persistence — |
| 31 | +deliberately phased, with the single-program-counter case becoming the |
| 32 | +degenerate *one-token* tree so today's flows are bit-for-bit unchanged. |
| 33 | + |
| 34 | +## Context — current state (verified 2026-06-11) |
| 35 | + |
| 36 | +- **One position per run.** `SuspendedRun = { runId, flowName, nodeId, |
| 37 | + variables, steps, context, … }` — a single `nodeId`. Resume restores that one |
| 38 | + position and calls `traverseNext` from it. |
| 39 | +- **Suspend unwinds the stack.** A pausing node throws `FlowSuspendSignal`, |
| 40 | + caught at `execute()` / `resume()`, which snapshots the one position. There is |
| 41 | + no place to record a *second* live position. |
| 42 | +- **Structured constructs run their bodies synchronously.** `loop` / `parallel` |
| 43 | + / `try_catch` execute their region(s) to completion in-line (ADR-0031). A |
| 44 | + pausing node inside a region throws the suspend signal up through the |
| 45 | + container, discarding the other branches / iterations — hence the hard ban. |
| 46 | +- **`parallel` is concurrent but not pausable.** `traverseNext` already runs |
| 47 | + unconditional out-edges via `Promise.all`, and the `parallel` block joins at |
| 48 | + block end — but a suspend inside one branch unwinds that branch and the |
| 49 | + siblings already in flight are **not** cancelled or persisted. Correctness |
| 50 | + holds only because pause-in-branch is forbidden. |
| 51 | +- **Between-flow pause already works.** ADR-0019's addendum (subflow linked |
| 52 | + runs, #1693) chains *separate* runs across the subflow boundary. That is the |
| 53 | + *inter*-flow half and is orthogonal to this ADR — it keeps working unchanged. |
| 54 | + |
| 55 | +The gap is strictly **intra-flow concurrency + pause**: one run, several live |
| 56 | +positions. |
| 57 | + |
| 58 | +## The reframing — why a token tree, and why not the alternatives |
| 59 | + |
| 60 | +A flow run is a **token game** on a graph (BPMN's own mental model): a token |
| 61 | +sits on a node; executing the node moves the token along its out-edges; a split |
| 62 | +turns one token into several; a join consumes several and emits one. The |
| 63 | +engine's "single `nodeId`" is the special case of *exactly one token that never |
| 64 | +splits*. Generalizing to *a set of tokens* is the minimal change that makes |
| 65 | +concurrent pause representable. |
| 66 | + |
| 67 | +- **Why not "serialize the interpreter stack" (Salesforce-Flow style).** |
| 68 | + Snapshotting the call stack inlines child state into the parent and destroys |
| 69 | + per-branch run identity; it also does not naturally express *N* independent |
| 70 | + paused positions. Rejected. |
| 71 | +- **Why not "event-sourced deterministic replay" (Temporal style).** Replay |
| 72 | + requires every node to be deterministic / idempotent. ADR-0018's **open node |
| 73 | + registry** lets third-party executors run arbitrary side effects — the replay |
| 74 | + precondition does not hold for this platform. Rejected. (This is a |
| 75 | + *generative-ecosystem* constraint, not a taste call: low-code + open plugins ⇒ |
| 76 | + the Camunda branch, not the Temporal branch.) |
| 77 | +- **Why the token/scope tree.** It is the runtime dual of ADR-0031's structured |
| 78 | + regions: a region instance *is* a scope, a scope's tokens *are* its live |
| 79 | + positions, and the join *is* the scope barrier. It is locally composable, |
| 80 | + statically bounded (no back-edges — ADR-0031's DAG invariant is preserved), |
| 81 | + and is the proven model in every BPMN engine. We are not inventing a runtime; |
| 82 | + we are adopting the standard one. |
| 83 | + |
| 84 | +## Decision |
| 85 | + |
| 86 | +### D1 — A run's live state is a set of **tokens** in a **scope tree** |
| 87 | + |
| 88 | +- A **token** is one execution position: `{ tokenId, scopeId, nodeId, status }` |
| 89 | + where `status ∈ { running, paused, completed, cancelled }`. |
| 90 | +- A **scope** is a region instance: the **root** flow, or an instance of a |
| 91 | + `parallel` branch / `loop` iteration / `try` or `catch` region. Scopes nest by |
| 92 | + containment → a tree. Each scope records `{ scopeId, parentScopeId, kind, |
| 93 | + iteration?, joinState }`. |
| 94 | +- A linear flow with no concurrency is a **one-token, one-scope** tree — |
| 95 | + identical behavior to today (the back-compat anchor). |
| 96 | + |
| 97 | +### D2 — Split / join are scope operations |
| 98 | + |
| 99 | +- Entering a `parallel` block creates one **child scope per branch**, each |
| 100 | + seeded with a token at its region entry; the block's join barrier records how |
| 101 | + many branch tokens must arrive. |
| 102 | +- Entering a `loop` creates one child scope per iteration (sequential by default; |
| 103 | + the model permits concurrent iterations behind a flag — out of scope for v1). |
| 104 | +- A scope **completes** when all its tokens reach its single exit (ADR-0031 |
| 105 | + single-entry/single-exit regions make this well-defined); completion emits one |
| 106 | + token into the parent scope at the container's ordinary out-edge — the |
| 107 | + existing "after-block / after-loop continuation." |
| 108 | + |
| 109 | +### D3 — Any token may pause; the scope persists partial progress |
| 110 | + |
| 111 | +- A pausing node (`approval` / `screen` / `wait`) sets **its token** to `paused` |
| 112 | + and snapshots the **whole tree** (all tokens, all scope join states, the |
| 113 | + variable scoping per D5). Sibling tokens keep running; the run is `paused` |
| 114 | + while **any** token is paused or running-then-pausable. |
| 115 | +- The run is `completed` only when the root scope completes with no paused |
| 116 | + tokens; `failed` per D6. |
| 117 | + |
| 118 | +### D4 — Resume targets a token |
| 119 | + |
| 120 | +- `resume(runId, signal)` gains an optional `tokenId`. With exactly one paused |
| 121 | + token (today's universal case) it resolves unambiguously — **the existing |
| 122 | + single-argument resume is unchanged**. Approval/wait/screen already carry a |
| 123 | + correlation key; the engine maps `correlation → tokenId`. |
| 124 | +- Resuming a token continues traversal **within its scope**; reaching the |
| 125 | + scope's exit decrements the parent join barrier. When the last branch token |
| 126 | + arrives, the join fires and the parent continues — possibly itself pausing |
| 127 | + again elsewhere. |
| 128 | + |
| 129 | +### D5 — Variable scoping is copy-on-write per scope |
| 130 | + |
| 131 | +- ADR-0031 keeps region bodies in the **enclosing variable scope**. With |
| 132 | + concurrent *paused* branches that share an enclosing map, a naive shared map |
| 133 | + lets one paused branch's later resume clobber another's reads. The model makes |
| 134 | + per-scope variable writes **copy-on-write**: a scope sees the enclosing values |
| 135 | + but its writes are isolated to its scope frame until it joins, at which point a |
| 136 | + defined **merge policy** folds them back (last-writer-wins by default; |
| 137 | + loop accumulation via explicit output variables, as today). The merge policy |
| 138 | + is a named decision point, not left implicit. |
| 139 | + |
| 140 | +### D6 — Failure and cancellation are scope-scoped |
| 141 | + |
| 142 | +- A token failing terminally **fails its scope**; by default the scope's failure |
| 143 | + **cancels its sibling tokens** in the same parent (interrupt semantics) and |
| 144 | + propagates up — matching the intuition "if the parallel block can't finish, |
| 145 | + the block fails." `try_catch` is the structured opt-out: a `try`-scope failure |
| 146 | + routes to the `catch` scope instead of propagating (ADR-0031, unchanged). |
| 147 | +- Cancellation of a *running* token is cooperative (checked at node boundaries); |
| 148 | + cancellation of a *paused* token consumes its continuation and records it |
| 149 | + cancelled. This cancellation primitive is what later unlocks **boundary |
| 150 | + events / timers** (a separate follow-up ADR builds on it — see Non-goals). |
| 151 | + |
| 152 | +### D7 — Authoring model and DAG invariant unchanged |
| 153 | + |
| 154 | +- The flow JSON does **not** change. Tokens/scopes are runtime-only; |
| 155 | + `flow.zod.ts` and the designer are untouched. ADR-0031's structured constructs |
| 156 | + remain the authoring surface and the AI design center (ADR-0010/0011). |
| 157 | +- No back-edges are introduced. Scopes are acyclic single-entry/single-exit |
| 158 | + regions; iteration stays the loop container's job. The DAG invariant holds. |
| 159 | + |
| 160 | +## Representation — persistence evolution (additive) |
| 161 | + |
| 162 | +`SuspendedRun` / `sys_automation_run` evolve **additively**: |
| 163 | + |
| 164 | +- Keep `nodeId` as the **primary token's** position (the first/only paused token) |
| 165 | + so existing readers, the Runs panel, and one-pause flows keep working with no |
| 166 | + change. |
| 167 | +- Add `tokens_json` (and the scope tree) as a new JSON column / field carrying |
| 168 | + the full set when there is more than one. A row with no `tokens_json` is a |
| 169 | + one-token run — rehydrated as today. This mirrors the ADR-0019 discipline of |
| 170 | + not breaking the suspended-run table; the single new additive column is the |
| 171 | + deliberate exception this feature requires. |
| 172 | +- `resume` continuation, correlation→token mapping, and the cold-boot |
| 173 | + wait-timer re-arm (#1687) all extend to address a token instead of the run. |
| 174 | + |
| 175 | +`execution.zod.ts` gains a `tokens[]` / `scopes[]` shape on the run log; |
| 176 | +`ExecutionStepLogSchema` already tags steps with `parentNodeId` / `iteration` / |
| 177 | +`regionKind` (ADR-0031 #1505), which the token model formalizes as scope ids. |
| 178 | + |
| 179 | +## Consequences |
| 180 | + |
| 181 | +- **Unlocks** parallel approvals, batch (per-iteration) approvals, concurrent |
| 182 | + waits, and lays the cancellation primitive for boundary timers/events. |
| 183 | +- **Core risk.** Traversal, suspend/resume, and persistence are the engine's |
| 184 | + heart; this is the largest change to it since ADR-0019. Mitigated by phasing |
| 185 | + (below) with the one-token degenerate case as a behavior-preserving anchor and |
| 186 | + the full existing suite as the regression gate at every phase. |
| 187 | +- **Observability improves**: the Runs panel can show a tree of live positions |
| 188 | + ("branch ① paused at approval, branch ② done") instead of one node. |
| 189 | +- **No authoring or migration cost** for existing flows — they are one-token |
| 190 | + trees; the JSON, the designer, and stored runs are untouched. |
| 191 | +- **Subflow linked-runs (ADR-0019 addendum) composes**: a subflow token whose |
| 192 | + child run pauses stays `paused` in its scope exactly like any other pausing |
| 193 | + node — inter-flow chaining and intra-flow tokens stack cleanly. |
| 194 | + |
| 195 | +## Sequencing (roadmap) |
| 196 | + |
| 197 | +Each phase ships behind tests; the suite stays green throughout. |
| 198 | + |
| 199 | +1. **2a — Internal token model, zero behavior change.** Represent today's |
| 200 | + single program counter as a one-token / one-scope tree inside the engine. |
| 201 | + `executeNode` / `traverseNext` / `resume` operate on tokens; structured |
| 202 | + containers create child scopes but still run synchronously. No new capability; |
| 203 | + pure refactor that de-risks the rewrite. *Gate: full suite unchanged.* |
| 204 | +2. **2b — Pause inside `parallel` branches.** The most-requested case (parallel |
| 205 | + approvals). Join barrier persists partial completion; branch tokens pause and |
| 206 | + resume independently; D5 copy-on-write + merge lands here. |
| 207 | +3. **2c — Pause inside `loop` iterations.** Batch approvals. Sequential |
| 208 | + iterations first; the per-iteration scope is the unit of pause. |
| 209 | +4. **2d — Cancellation / interrupt (D6).** Sibling cancellation on scope |
| 210 | + failure; cooperative running-token cancellation. Unblocks a follow-up ADR for |
| 211 | + **boundary events / timers** (BPMN interrupting boundaries map onto this). |
| 212 | + |
| 213 | +## Non-goals / deferred |
| 214 | + |
| 215 | +- **Distributed token execution** across workers/nodes. v1 keeps one claimer per |
| 216 | + run (today's model); tokens are concurrent *within* a process, not sharded |
| 217 | + across machines. |
| 218 | +- **Parallel I/O speedup as a goal.** Concurrency here is about independent |
| 219 | + *pause*, not throughput; any wall-clock win is incidental. |
| 220 | +- **Full BPMN boundary-event / event-subprocess semantics.** The cancellation |
| 221 | + primitive (2d) is the foundation; the node-type surface is a separate ADR. |
| 222 | +- **Concurrent loop iterations** (fan-out map). The model permits it behind a |
| 223 | + flag; v1 ships sequential iteration only. |
| 224 | +- **Changing the authoring model.** Out of scope by D7 — tokens are runtime-only. |
| 225 | + |
| 226 | +## Relationship to prior ADRs |
| 227 | + |
| 228 | +- **ADR-0019** gave durable pause for a *single* position and (in its addendum) |
| 229 | + *between*-flow chaining. This ADR generalizes the *within*-flow position from |
| 230 | + one to a tree. The resume contract and `sys_automation_run` extend additively. |
| 231 | +- **ADR-0031** defined the structured regions; this ADR is their **runtime |
| 232 | + dual** — a region instance is a scope. The DAG invariant and AI-authoring |
| 233 | + center are explicitly preserved. |
| 234 | +- **ADR-0018**'s open registry is the reason replay-based models are rejected |
| 235 | + (D-reframing) and why the token/scope model is the right fit. |
0 commit comments