Skip to content

0039-token-scope-tree-execution.md

Latest commit

 

History

History
252 lines (215 loc) · 15 KB

0039-token-scope-tree-execution.md

File metadata and controls

252 lines (215 loc) · 15 KB
Raw

ADR-0039: Concurrent durable pause — multi-instance nodes now, token/scope-tree later

Status: Proposed (2026-06-11) — revised after a code + industry self-review Deciders: ObjectStack Protocol Architects Builds on: ADR-0019 (durable-pause node via suspend/resume — between-flow chaining added in its 2026-06-10 addendum), ADR-0031 (structured loop / parallel / try_catch constructs, DAG invariant), ADR-0018 (open node/executor registry) Consumers: @objectstack/services/service-automation (engine core — executeNode / traverseNext / runRegion / resume, SuspendedRun, sys_automation_run), @objectstack/spec (automation/execution.zod.ts), ../objectui (Runs panel, flow runner)

TL;DR

The engine tracks a paused run with a single program counter — SuspendedRun.nodeId, one position — and suspend is implemented as a thrown exception that unwinds the call stack (FlowSuspendSignal). That cannot represent two pauses at once, so the engine forbids pausing inside a parallel branch or loop iteration (runRegion converts a suspend inside a region into a hard error). This blocks parallel approvals ("finance AND legal sign off concurrently") and batch approvals ("route each line item over $10k").

The tempting answer — adopt a BPMN-style token / scope tree (Camunda) — is the right long-term runtime model but is a full engine-core rewrite: it is not just a data structure, it forces replacing three coupled things the current engine relies on (see Why the token tree is expensive). A code review (below) shows the cost is much larger than the data-structure change implies.

Decision: two tracks. Ship Track A — multi-instance / aggregating nodes first: model the demand as single nodes that wait for N decisions, the way Camunda multi-instance and AWS Step Functions Map do. Track A splits into a free tier and a bounded tier — a distinction worth stating up front: A1 (parallel approval — one approval node aggregating N decisions) needs no engine change and is shipped (#1708); A2 (a map / multi-instance node for batch approval) is NOT free — because each item can pause, it needs a bounded extension of the engine's resume path (N:1 aggregation or node re-entry), so it is a separately-justified increment, not a free rider on A1. Defer Track B — the general token/scope tree until demand exceeds what multi-instance covers; this ADR records its design so Track A is built toward it, not away from it.

Context — current state (verified 2026-06-11, against the code)

  • One position per run. SuspendedRun.nodeId is a single node id. resume restores that one position and calls traverseNext from it.
  • Suspend is a thrown exception. A pausing node throws FlowSuspendSignal; executeNode unwinds, execute() / resume() catch it and snapshot the one position. The JS call stack is the continuation while running; on resume the engine re-derives traversal from the single nodeId (it does not restore a stack).
  • runRegion bans pause structurally. parallel / loop / try_catch run their region(s) through runRegion, which catches a FlowSuspendSignal and rethrows it as Error("durable pause inside a structured region … is not supported"). That is where the ban lives.
  • Two concurrency sources, not one. Besides the structured parallel node, traverseNext already runs a node's multiple unconditional out-edges concurrently via Promise.all — raw graph fan-out. A suspend in either path unwinds and the siblings are not cancelled; correctness holds only because pause-in-branch is banned.
  • Variables are one flat shared Map. Map<string, unknown> is shared by the whole run and every region/branch/iteration — there is no scoping. Loop iterations overwrite the iterator var in place; node output is written as variables.set('${nodeId}.${key}', …). ADR-0031 deliberately runs regions "in the enclosing variable scope," i.e. on this same flat map.
  • Between-flow pause already works. ADR-0019's addendum (subflow linked runs, #1693) chains separate runs across the subflow boundary — orthogonal to this ADR and unchanged by either track.

The gap is strictly intra-flow concurrency + pause: one run, several live positions.

Why the token tree is expensive (the real cost)

A self-review against Camunda/Zeebe/Flowable and the actual code found that the token/scope tree is a data structure whose value only appears when paired with three execution-model changes the current engine does not have. Adopting the tree without these (as a first draft of this ADR did) is adopting the noun without the verb.

  1. Recursion + throw → an explicit token scheduler. Today execution is recursive executeNode and suspend is a thrown unwind. You cannot simultaneously "pause branch A" and "keep branch B running" with a thrown exception — Promise.all rejects on A's throw while B keeps mutating the shared map after the snapshot. Camunda/Zeebe instead run a command/job queue: pop a runnable token, advance it one step, persist; a token that hits a wait state simply stops being runnable (no exception). Concurrent pause requires this scheduler — it is the core rewrite, not a refactor.

  2. Flat shared map → hierarchical scope variables. Camunda resolves a variable by walking up the execution tree (token scope → parent → … → process instance); a write defaults to the current scope and is discarded when the scope ends unless promoted. (The first draft of this ADR invented a "copy-on-write + merge-on-join" scheme — no major engine does that; it is both harder and semantically surprising.) Moving from one flat Map to scope-chained resolution touches every variables.get/set, every template interpolation, and every CEL evaluation in the engine.

  3. Per-run serialization. Two sibling tokens (e.g. two parallel approvals decided at the same instant) would resume concurrently and race on shared run state and the join barrier. Camunda serializes commands per process instance (optimistic locking). v1 of Track B would likewise need to serialize token advances within a run — which means the concurrency is logical (independent pause points), not parallel execution. That is a real, honest limitation to state up front.

The token tree is correct long-term, but its cost is "rebuild the engine's execution model," not "add a tree to SuspendedRun."

Decision

Track A (now) — multi-instance / aggregating nodes

Model the concrete demand as single nodes that internally fan out and aggregate, leaving the engine's one-program-counter model intact:

Track A has two tiers of cost — a distinction the first revision of this ADR got wrong by lumping them together. They are not equal.

A1 — aggregating approval node (truly free; shipped #1708). One approval node with behavior: 'unanimous' over N approver groups opens one sys_approval_request whose pending_approvers lists all groups (notified in parallel) and stays suspended until every group approves, then resumes down approve / reject. "Finance AND legal" is exactly this — one node, one program counter, paused once. This needed no engine change: the unanimous-over-N aggregation already exists in the approvals service and is unit-tested; A1 added a showcase (showcase_invoice_signoff) and docs, browser- verified. The aggregation state lives in the plugin's own sys_approval_request row, not the engine.

A2 — map / multi-instance node (NOT free — engine-adjacent). A correction: a map node that serves batch approval (each item can pause) cannot be "no engine change," contrary to this ADR's first revision. Examined against the code, every flavor needs a bounded extension of the engine's resume/bubble path:

  • concurrent map (N items pause at once) needs durable N:1 aggregation + per-parent serialization + completion-ordering handling — i.e. part of Track B's hard concurrency, just confined to one node;
  • sequential map (one item at a time) needs resume-into-the-node (process the next item) instead of the engine's resume-past-the-node default — the DAG has no back-edge to loop the node;
  • only a synchronous, non-pausing map is engine-free, and that does not serve batch approval (which pauses). The map node reuses ADR-0019's linked-runs (#1693) for the 1:1 bubble but extends it to N:1 / re-entry. It is a real, bounded engine task — smaller than the full Track B scheduler, but not the zero-cost item A1 was. It should be built only against concrete batch-approval demand, with the aggregation / re-entry semantics designed first.

So Track A as shipped (A1) covers parallel approvals at zero engine cost. Batch approvals (A2) are a deliberate, separately-justified increment, not a free rider on A1.

Track B (deferred) — the general token / scope tree

When a flow genuinely needs to pause at arbitrary, independent positions that multi-instance cannot express (e.g. two unrelated long-running waits on different branches that each continue into different downstream logic), adopt the full model:

  • Token = { tokenId, scopeId, nodeId, status }, status ∈ { running | paused | completed | cancelled }.
  • Scope = a region instance (root flow, parallel branch, loop iteration, try region), nested by containment into a tree. A linear flow is a one-token / one-scope tree — the back-compat anchor (today's behavior unchanged).
  • Execution is the scheduler of §1 above, not recursion. Variables are scope-hierarchical (§2). Resume targets a tokenId (defaulting to the sole paused token for back-compat) and is serialized per run (§3). Split/join are scope operations; a scope's join is a barrier that fires when its child tokens reach its single exit (ADR-0031 single-entry/single-exit makes this well-defined). Failure fails the scope and cancels siblings (interrupt) unless caught by a try_catch scope; this cancellation primitive is what later unlocks boundary events/timers.
  • Persistence is additive: keep nodeId as the primary token's position so existing readers and one-pause flows are unchanged; add tokens_json for the full tree when there is more than one.
  • Authoring and DAG unchanged (D7 below): tokens are runtime-only; the flow JSON, the designer, and the AI design center (ADR-0010/0011) are untouched, and no back-edges are introduced.

D7 — invariants that hold on both tracks

  • The flow JSON, the structured-construct authoring surface (ADR-0031), the AI design center, and the DAG invariant are unchanged. Concurrency is a runtime concern, never an authoring one.
  • The single-position / single-token case stays bit-for-bit today's behavior.
  • Subflow linked-runs (ADR-0019 addendum) composes with either track.

Why not the other models

  • Serialize the interpreter stack (Salesforce-Flow style): inlines child state into the parent, destroys per-branch run identity, and still cannot express N independent pauses. Rejected.
  • Event-sourced deterministic replay (Temporal/Zeebe-internals style): requires every node to be deterministic/idempotent. ADR-0018's open node registry lets third-party executors run arbitrary side effects — the replay precondition does not hold here. This is a generative-ecosystem constraint, not a taste call. Rejected as the engine model.
  • Jump straight to the general token tree (first draft of this ADR): correct long-term but over-built for the near-term demand, and its true cost (the three execution-model changes above) is not yet justified. Deferred to Track B.

Consequences

  • Track A unblocks the real demand now (parallel + batch approvals) with no engine-core rewrite, no persistence change, and no new concurrency hazards.
  • Track B is recorded, not started. The team avoids a premature core rewrite while keeping a coherent target; Track A's multi-instance node is designed so its per-unit state could later be re-expressed as scoped tokens.
  • Honest limitation of Track A: it does not allow pausing at a free point inside a hand-drawn parallel/loop region — only the structured aggregating node pauses. If a flow needs that, it is the signal to start Track B.
  • Observability: Track A shows N per-unit rows under one node (e.g. the approvals list); Track B would show a tree of live positions. The Runs panel extends additively either way.

Sequencing

  1. A1 — aggregating approval node. ✅ Shipped (#1708). The unanimous-over-N-approver-groups aggregation already existed and was unit-tested; #1708 added the showcase_invoice_signoff worked example (finance AND legal, browser-verified) and docs. No engine change. Threshold / quorum (M-of-N) stays enterprise-tier per approval.zod.ts.
  2. A2 — map / multi-instance node (design-first; not started). Collection in, per-item child unit, aggregation, single suspend at the node. Cost correction: because items can pause, this needs a bounded engine resume-path extension (durable N:1 aggregation for concurrent, or resume-into-node for sequential) — it is not the zero-engine-change item A1 was, so it is gated on concrete batch-approval demand and a design note that nails the aggregation / re-entry + serialization semantics first.
  3. B-gate — only if a concrete flow needs arbitrary-position concurrent pause that a multi-instance node cannot express: open a follow-up ADR to start Track B at the scheduler, with the one-token refactor as the first, behavior-preserving step.

Non-goals / deferred

  • The general token/scope tree and its scheduler (Track B) — recorded, not scheduled.
  • Distributed token execution across workers/nodes (one claimer per run stands).
  • Concurrent loop iterations / true parallel I/O speedup (logical concurrency only; not a throughput feature).
  • Full BPMN boundary-event / event-subprocess semantics (built on Track B's cancellation primitive; separate node-type ADR).
  • Any change to the authoring model (D7).

Relationship to prior ADRs

  • ADR-0019 gave durable pause for a single position and (addendum) between-flow chaining. Track A reuses that pause as-is (the aggregating node pauses once); Track B would generalize the within-flow position to a tree.
  • ADR-0031 defined the structured regions. Track A's multi-instance node is a new structured construct alongside them; Track B's scopes are their runtime dual. The DAG invariant and AI-authoring center are preserved on both.
  • ADR-0018's open registry is why replay models are rejected and why, when Track B comes, the Camunda-style scheduler (not Temporal replay) is the fit.